Students at German universities are in the middle of the third examination phase during the pandemic – and many universities are still ill-prepared for it. In the following, Leonie Ackermann explains why online proctoring is (still) not a good idea and at the same time gives an insight into the technical functioning of common service providers such as Proctorio and Examity.

The following text is an abridged version of a lecture Leonie Ackermann gave at the event “Online Exams and Proctoring from a Legal Perspective”, where Leonie Ackermann explained the technical basics of online-proctoring, as a supplement to David Werdermann’s legal assessment. The recording of the event can be found here. A link to download the slides of both speakers can be found in the video description.

tl;dr

Online proctoring services combine technologies that are already problematic when considered individually, and their use must be critically reflected upon. Their use is accompanied by profound intrusions into the privacy rights of exam takers. This should be sufficient reason not to use these tools. If you need further justification, here you go: Online proctoring cannot exclude deception despite fundamental rights violations.

Online proctoring: a brief introduction

Proctoring is supervision during an examination to prevent or detect and sanction undesirable behavior. Proctoring software is used to monitor Online Proctored Exams in order to verify the identity of those taking the exam and to detect any use of unauthorized aids during such online exams. The software for taking an exam on one’s own computer (usually web-based and accessible using a web browser) is not part of proctoring software. It is integrated into existing learning management systems (LMS) or examination software, for example, with the help of plug-ins. In Germany, in contrast to proctoring software, this is mostly open source or free software.

Live proctoring is the monitoring by a human person sitting in front of a computer. Monitoring using algorithms or machine learning or artificial intelligence (AI) is called automatic or automated proctoring. Some service providers* use a mix of both forms or offer to select one of the two.

Online proctoring services refer to the combination of a deployment of this software, possibly human proctors, additionally deployed personnel, and deployed infrastructure. The use of these services is considered when closed book exams, i.e., exams where only a very limited and predetermined number of proctoring aids are allowed, cannot be conducted in presence or can only be conducted with great difficulty.

How does the monitoring work?

Before the exam begins, the identity of the graduates is established. In some circumstances, graduates may be asked to film the (private) room where the exam is being taken (e.g., using a smartphone camera). Throughout the exam, additional measures using various technologies are used to detect and/or prevent cheating attempts. These vary depending on the manufacturer. Common ones are:

• Filming using the webcam
• Using a built-in or external microphone
• Face recognition using AI
• Behavioral control by human proctors or machine behavior recognition by AI
• Voice recognition
• Analysis of typing behavior
• Monitoring and restriction of functions of test takers’ computers
• Automated plagiarism detection on content entered during the test

To keep the scope of this post in check, I will limit my discussion to identity determination, facial recognition, and behavioral recognition.

Identification

A commonly used technology in identity verification of graduates is face recognition by AI. The software used is often not developed by the online proctoring services themselves, but purchased from external companies. For example, the University of Erfurt uses WISEflow for its online exams. This exam software draws on Amazon’s Rekognition product to determine biometric scores of graduates. However, Rekognition, like numerous other facial recognition AIs, has racist and sexist tendencies, as most recently shown in a 2019 study (study; article contextualizing this).

A lecturer at the University of Denver, which uses the online proctoring service Proctorio, reports that a student with black skin color was not recognized, while fellow students* with white skin color were easily recognized. Thus, the use of facial recognition AI to collect biometric data in Germany contradicts a demand of the German Federal Commissioner for Data Protection and Freedom of Information (BfDI): collection, storage and processing of biometric data should only take place if procedures are used which largely exclude a disadvantage of certain groups of people.

Further possibilities to identify students with black skin color were reported.

Other ways to verify identity may include login credentials for the LMS and/or uploading a photo ID or comparable document. The online proctoring service Examity calculates a “Biometric key stroke signature” from the input of the first and last name. How useful this procedure is for very short names, e.g. Wu Li, remains an open question. Probably, the disadvantage of certain groups of persons cannot be excluded even with this procedure.

Behavioral control and detection

Proctorio, according to its own statements, does not collect biometric data from graduates, at least in Germany. Facial recognition AI is used by the service provider for behavioral recognitionduring testing. From the Proctorio “FAQ for students [sic!]” :

“With this facial recognition, the system can track eye movements. The system detects anomalies by increasing glances in one direction and marks these incidents as potentially suspicious. […] This doesn’t mean that you can’t look away during pauses in thinking. As long as you don’t use any tools, you have nothing to worry about.”

Examity flags sections of surveillance video recorded during exams with colors and timestamps (green = no cheating detected; yellow = possible cheating; red = almost certainly cheating; blue = technical error). An example of a section marked yellow is when a child enters the room during an online proctored exam. Both Examity and Proctorio provide proctored examiners with the surveillance videos. The graduates have no claim to the recordings.

The fact that sensitive recordings of students in their private environment during a stressful situation are made available exclusively to lecturers should set alarm bells ringing. This offers substantial potential for abuse, for example in the direction of sexual harassment. It is therefore reassuring that some federal states have already banned the recording of images and sound during online examinations.

There remains the problem, however, that automated proctoring can classify behaviors as suspicious that are relaxation measures. During face-to-face exams, for example, it would be normal to lean back in your chair, let your eyes wander around the room, or close your eyes for a few seconds. But Fresenius University of Applied Sciences requires students to stare at the screen during online proctored exams for the entire duration. One student affected by this reported that his concentration was massively affected by the fear of getting it “wrong” and thus failing the exam.

Online proctoring services thus create an unhealthy work climate and by no means the “flexibility” claimed in advertising slogans. Especially neurodiverse persons run the risk of being marked as “suspicious” by behavior recognition algorithms at a higher-than-average rate and thus being discriminated against. Especially when examiners* consider AI applications to be objective, they are very likely to adopt the software’s decisions unquestioningly.

Online proctoring can’t prevent cheating

For those who want to look more closely at strategies for tricking proctoring, I can recommend the following links:

• On Knuckle Scanners and Cheating – How to Bypass Proctoring, Examity, and the Rest – Jake Binstein
• Need to cheat on a proctored, online exam? Make a cheat sheet and put it on your laptop screen. – Posted by JamirBlumenfeldwitz on reddit

Generally speaking, any*sufficiently tech-savvy exam taker (or those who know the right people) can rebuild (have rebuilt) their own device at will and make any changes that the online proctoring service (software or human* proctor*in) may not recognize. To put it in the words of telekobold from the Hacker- & Makerspace Koblenz, whose article helped me a lot in compiling this article:

“This would put [online proctored exams] on the one hand at a disadvantage graduates who are less technically proficient or just want to honestly take an exam, on the other hand such changes would not be possible in a paper-based or computer-based face-to-face exam (since this uses third-party hardware).”

So please put the financial and human resources into a good e-learning consultancy and the development of alternative exam formats away from online-proctoring and mass examps. That’s what we wanted anyway, right?