Ineffective human review

What it is

Human review is the control most often named when an organisation is asked how it manages an AI system. It can be well designed and still fail in operation. Volumes rise and time per case falls. The information reviewers need is no longer shown. Experienced reviewers leave and are replaced by people who have only ever worked with the tool. Overrides are questioned by managers and agreements are not. Bit by bit the review becomes confirmation. The design failure is covered in inadequate human oversight design; this entry is about oversight that looked adequate when approved and has decayed since.

How it happens

• Throughput targets make careful review impossible.
• Reviewers are not told the outcome of their decisions, so they cannot learn.
• The share of cases needing review was set once and volumes have since grown.
• Reviewers have no route to say that the system seems wrong.
• Nobody measures whether review catches anything.

Why it matters

In UK data protection law, human involvement that is not meaningful leaves a decision solely automated, with the safeguards that then apply. More broadly, the organisation is running an unchecked system while reporting a checked one, and the gap is discovered only after harm.

Warning signs

• Override rates are near zero or have fallen over time.
• Review takes a few seconds per case.
• No test has been done of whether reviewers spot seeded errors.
• Reviewers report pressure not to disagree.

Who owns it

The business owner of the process is accountable for review that works. Operations managers set staffing and targets. The second line tests effectiveness.

In practice

A benefits agency requires an officer to confirm each fraud referral made by a model. When the control was approved, officers handled thirty cases a day. Two years later they handle two hundred, and confirm all of them. The same pattern generalises to any review step whose conditions have changed and whose effectiveness nobody has measured since.