ProctorU, one of the largest and oldest providers of remote test monitoring and security, announced last week that it was phasing out its AI-only monitoring service. Going forward, the company said, it would only use trained human test supervisors – a service it has always offered.

They eliminate the monitoring process that registers a student during their test, uses AI and other technologies to analyze the session, flagging or noting it for suspicious or abnormal events and behaviors such as regularly staring off-screen or talking to someone.

This is a fairly standard start to remote monitoring. But what made this process “AI-only” was that once the test records were flagged or graded, they were turned over to faculty or test providers. No human on the surveillance side ever examined it, leaving teachers or assistants on the school side to examine the flags and make decisions about what happened and what did not happen during the test.

By abandoning the AI-only process, ProctorU said it places too much of a burden on teachers to review and control test sessions – a big job they are often unprepared for and unable to handle given their already heavy commitments. In one of the news reports regarding the decision, the company said it would take more than nine hours for a professor to review only the flagged portions of a single one-hour exam in a class of 150 students.

Additionally, ProctorU executives have stated that the AI ​​systems used in surveillance are sensitive by design, which could flag innocent activity as suspicious. It’s logic. If the goal of a first screen is to catch things for people to look at, you want to catch almost anything that could be anything. In other words, it is better to over-report than under-report. The problem is, it creates more flags – more work when the flags are just passed to teachers.

As a result, ProctorU and independent audits show that very few reported test sessions were reviewed at school. A study from Iowa showed that 14% were supervised by teachers. ProctorU said 11% were. Little, anyway.

This path – responsive AI, lack of exam time and no exam – made the process almost pointless.

Worse, we know that some teachers relied on flags, deviation scores, as evidence of misconduct. Considering what AI does, that’s impossible. Without human examination, determination of any kind – cheating or trivial – is impossible.

It is possible that the mere fact of having an AI-only system deterred some misconduct, given the possibility of detection. But with review rates so low, the deterrence is unlikely to have been so weak. In order for deterrents to work, you have to catch someone once in a while.

This important market decision has some implications for instructors and administrators. Here are five things teachers and other schools should do:

Understand the difference

When testing or homework is done online, the choice is not to supervise or not to supervise. For the record, not watching is a very bad idea. Research shows that failing to protect a test invites academic misconduct.

If a remote assessment is to be supervised, professors and administrators should be aware that there are three different models: the AI ​​only option, scan and transmit, an option to record, scan and review by supervisors. company trained and exam with live human supervisors. during the exam. The first type is what ProctorU is abandoning and the type that has generated the most controversy.

Because they aren’t the same, they don’t cost the same. In response to ProctorU’s decision to use only human monitors, an executive from Respondus, a competitor to ProctorU, said, “A typical university using Respondus Monitor spends less than 25 cents per monitoring session. Live human surveillance often costs $ 25 per exam. Prices and services are totally different.

They are very different. Understanding how they are different is important.

Make sure you know what type of system your surveillance company is using and what that means to you

The names of surveillance companies can be confusing – ProctorU, Proctorio, Respondus, Honorlock. Their product names can be even more so.

If you are unsure of the type of supervision your chosen company uses for your exams, ask.

If the answer is the cheapest version, recorded, AI-scanned, and forwarded, understand that the burden of reviewing recorded and reported sessions is likely yours. The company might be making a recording and telling you what to watch, but you’ll have to watch it and make a decision on what’s going on.

Review your flags

Whichever method your monitoring company uses, session videos will be transmitted or made available – either flagged, flagged and reviewed, or flagged and reviewed in real time. You will still need to review these sessions and decide what to do.

The advantage of having humans review the session before they get to you is that you will have a lot less to review. One estimate was that human-supervised testing reduced the number of exams required by professors by 85% – reducing from nine exam hours to just one.

In all cases, an examination is necessary. The system only works if a teacher or similar school authority monitors, engages students when there are incidents, and elevates misconduct incidents as necessary.

Ask for help

If reviewing reported test sessions is not possible, and it may not be, tell someone. Ask for help. Whatever the cost, remote test monitoring loses almost all of its impact if no one on the school side reviews the sessions. A system in which only 11% or 14% of sessions are examined is a failed system.

In addition, if it is not possible or reasonable to review sessions, school administrators should be aware of the burden that is created.

Ask better

People are definitely better at discerning attempts to be dishonest from common behavior. Computers can report things that are different, things that may be misconduct. Only a human can decide what really is.

If your department or school uses a system that bypasses humans, ask for one that doesn’t. Ask especially if you can’t or shouldn’t review computer-reported test session times. The best options are probably more expensive. But getting it right – avoiding false accusations, freeing teachers to teach and mentoring, having two humans involved instead of none – is probably worth it.

Derek Newton, educational writer in New York


Source link