Setting Up Accommodations

At the most basic level, this project's only requirement was to allow instructors to opt test takers out of some settings. While this is technically enough, there is a big limitation:

Instructors may be unfamiliar with both the Proctorio settings and with different disabilities or situations which may require accommodations.

So giving this feature to exam administrators without any form of guidance would be unfair to them, as it would put the burden of thoroughly understanding how each setting may interefere with different contexts. So it was imperative for us to provide instructors with the tools and information they need to make the right choices.

Mapping settings and disabilities

To start, I created a matrix mapping between settings and disabilities. Each setting was laid out in a row, and different types of disabilities were laid out in columns. I then added issues I knew existed for some settings (based on past calls with partner institutions) and some I speculated might exist.

Validating with a subject-matter expert

I then set up a call with a subject-matter expert (SME) from an accessibility consulting company Proctorio recently partnered with. I structured the call around the matrix I described above. We went over each setting and brainstormed accessibility issues.

To avoid influencing the SME, I did not mention any of the issues I had already put in the matrix before the discussion. I only brought them up if he didn't, which allowed me to validate the issue with him.

Getting feedback from clients

While the previous two activities were helpful, they were exploratory exercises. We also needed concrete information. So I turned to our partner institutions and partner success managers (PSM).

Proctorio's PSMs occasionally get accessibility-related emails and reports from partner institutions. I requested those and went over them to identify accessibility issues that cannot be fixed and would need accommodations.

I also conducted meetings with accessibility teams from different partner institutions. They were able to provide detailed descriptions of accessibility limitations and workarounds they have experienced in the past. I also got documents they sent to faculty describing accessibility considerations with Proctorio.

Putting it all together

With the information I got, I determined that there were three types of accommodations:

Setting the exemptions

From the exam settings, users can access the accommodations overview page. This page mainly features a table that lists all the accommodations currently in place, breaking it down by settings exemptions and analysis considerations, as described in the previous section.

To add an accommodations, exam administrators can search by the test taker’s name.

After selecting a test taker, exam administrators are taken to the individual accommodation screen:

This screen’s organization follows the three types of accommodations uncovered during research: settings exemptions, analysis considerations, and non-proctoring accommodations.

Organizing the settings exemptions

I organized the setings exemptions into four options:

1. No exemptions. This option is necessary because instructors may not want to set any exemptions but still set some analysis considerations (see the next section).
2. Motor & vision exemptions. This will exempt the test taker from all settings that may cause issues to test takers with motor or vision disabilities.
3. Custom exemptions. With this option, exam administrators are able to choose which settings to opt the test taker out from. Note that they are NOT able to use this option to add new settings for this test taker.
4. Exempt from proctoring. The test taker will do the exam without remote proctoring. Usually they will have to go to a testing center to take it instead.

These are presented as a list of radio inputs. When an option is chosen, the panel on the right shows which options will be disabled, why, and any extra actions related to that option.

Dealing with analysis considerations

Before the exam settings redesign, exam admins could set the analysis metrics during the exam set up. This was done through a set of dials that operated on a numerical scale. But we removed it in favor of delaying that choice until after the exam. So it wouldn't make sense to bring them back for the accommodations feature. It would also clash with the idea that we want to guide instructors instead of just letting them change settings.

The first decision was that it would be enough to treat these metrics as binary: you can either leave it on or turn it off entirely. The second was to focus primarily on the context instead of the metric.

The result was a series of toggle options based on different contexts elicited during research. For example, if the options stating that the test taker may be helped during the exam is enabled, Proctorio will not flag multiple faces.

Non-proctoring accommodations

The last item is simply a paragraph explaining that test takers may still need accommodations outside of the ones in this page. It directs them to look for the test platform’s accommodations settings.

On the test taker's side

In line with what we learned in the test taker setup project, we wanted to explicitly reinforce to the test taker that the accommodations are in place to reduce anxiety. So we decided to add a screen during the setup to let them know that the accommodations are in place.

To ensure privacy, we don't disclose which accommodations are set.

Making it a top-level navigation item

I added the accommodations page as top-level navigation item in the exam settings. This menu is always visible regardless of the scroll position, so the link to the accommodations is always visible.

Updating the settings summary

To both inform and raise awareness, I also added a section for accommodations in the exam summary. This features a yellow badge counter, a button that takes the user to accommodations page, and a list of settings and the reasons why they may need accommodations. This list reflects the active settings, so only items that apply to the current settings will be shown.

Different data sources paint a clear picture

For this project, it was critical to use multiple data sources. Each one with its own limitations and biases. I was very familiar with the software, but still have much to learn on accessibility. The experted we consulted was the opposite, but even with great knowledge of accessibility, his input was only exploratory. That's why it was key to get real-life usage data from the PSMs and clients. Putting all of these together, I was able to paint a more complete picture of the scenarios in which accommodations might be needed.

Well-intentioned features can still be misused

Allowing exam administrators to customize the exam experience for individual test takers is obviously necessary. But if not designed right, could allow exam administrators to target individual test takers with more strict settings. So throughout this whole project, we designed with this in mind:

• Custom accommodations only allow instructors to make the exam less strict.
• You cannot set exemptions in bulk.
• A warning is shown if more than a given percentage of test takers has exemptions set for them.

Have a clear agenda when talking with SMEs

When talking with SMEs, you must have a very clear agenda. Otherwise, it's very easy for them to go very deep into a single point while you still needed to cover more territory. It's also important to do your homework before the call and do your own research on the subject. This can aid in communication and help you ensure you explored everything you want to.