Taking a Proctored Exam

Data sources

When I was tasked with this redesign, there was no process for recruiting test takers for research. Therefore, I had to figure out alternative information sources to base my design on. Here's what I did:

Support data

To reduce support load, I had to identify the biggest technical issues and how support helps users with them. To understand this, I used the available analytics, interviewed Proctorio's support staff, and even did a bit of support myself to get first-hand experience.

Literature review

I don't have to recreate the wheel. Academic literature and articles around the web reveal important design principles for designing for privacy. Articles on remote proctoring also reveal ethical considerations and misconceptions that we should address.

Reviews & social media

A final piece of data was to directly see what users were saying on any publicly available channels. I analyzed all user reviews and as many posts about Proctorio I could find on social media and identified common themes on reasons for frustration.

Findings

• Privacy distrust comes from not knowing what information is being accessed and when.
• Anxiety can also be driven by technical uncertainty. ”How will my computer behave during the exam?”
• Users weren't always aware of support.
• Many issues can be quickly solved with simple step-by-step instructions.

To improve the efficiency of the flow, I looked for the biggest blockers. One of the first steps in the setup process is a system check screen, where the user waits for things like their connection speed to be evaluated. This can take from a few seconds up to a couple of minutes.

There's no reason why this process can't happen in the background. So in the redesigned setup flow, this check happens in the background while the user completes the rest of the prechecks.

This new flow also presents other benefits discussed in the rest of this case study.

Intentional timing & duration for permissions

In the current flow, permissions are requested at the moment the extension is installed. This means users see Chrome's predefined permissions messages, which do not reflect how Proctorio uses them.

In the new flow, the permission request is the second step in the setup. This gives us the opportunity to present an explanation of why we need each permission and how we'll use them.

More importantly, a common misconception is that Proctorio would be running in the background outside of the exam. To address that, the extension will proactively give up the extension permissions as soon as the exam is done.

Clearly state when recording starts and ends

If the extension requests permission to access the camera and then the camera light comes on, the test taker will think that they are being recorded already when that is not the case. So on the new permissions screen, we now emphasize that recording has not started, and that it will only start after it's clearly indicated.

Be clear on what is NOT being recorded

It is just as important to clearly what you are NOT recording as it is to say what you are. If you don't, the user will probably fill in that blank for you, and usually not in a positive way (understandably).

So we added a section in the information screen that explains what IS and what ISN'T recorded.

Keep data in a familiar context

When using internet services, we often don't know where our data is going and who will have access to it. To counter that, we gave special emphasis to the following points:

• The location of the data, usually at the nearest data center.
• The duration of the data, usually one year.
• Who can access the data. Since Proctorio uses zero-knowledge encryption, no one but the institution can access it. Not Proctorio, Google, Facebook, parents, etc.

Emphasize that support is available at all times

Part of the anxiety derives from the uncertainty of what to do if something goes wrong. To address this concern, I made two changes:

1. Add a support button that is visible at the top of the set up at all times.
2. Highlight the availablility of immediate support in the information page.

The final step was to address any steps that have to be done by the user, usually as a way of addressing some issue that was detected during the system checks. While researching support issues, I was able to categorize them into three types:

1. Mandatory steps that can be automatically executed e.g. Closing open tabs.
2. Mandatory steps that must be manually executed e.g. Fixing camera issues.
3. Recommended steps e.g. Hiding the screenshare dialog to avoid accidentally stopping it during the exam.

In the old flow, troubleshooting was scattered across the setup flow, leading to a possibly tedious process of dealing with one error only to have to deal with another in the next step.

In the new flow, troubleshoot is consolidated into a single step at the end. This brings some advantages:

1. More time for the system checks to determine something is NOT an issue.
2. Less time between the system checks and the start of the exam, decreasing the chance that some change in the system may affect the exam.

Knowing the technical constraints is fundamental

Much of the initial work in this project went towards understanding the current technical constraints. This was key to allow me to optimize it. For example, by looking at the Google Chrome Extension API Docs, I learned that we could both request permissions on demand AND give them up when needed. Knowing this, I was able to suggest reordering the flow.

Don't leave room to the imagination

In a context where users don't trust the software, you cannot afford to let them fill in the blanks—even if you think something is obvious. Clearly state both what you do and what you don't.

Get creative to find the data

I had to move fast and didn't have immediate access to users. Getting creative to find secondary data was key to overcome this limitation. This data may not be perfect, but if you understand limitations and biases it can still give you important information on how to design.