Automation bias and over-reliance

What it is

People tend to defer to automated systems. A recommendation on a screen is taken as a starting point and seldom moved far from. The tendency grows when the system is usually right, when the person is busy, when disagreeing takes effort or needs justification, and when the output is fluent and confident. Generative models strengthen it, because their answers read like the work of a competent colleague. This is a property of people and has to be designed around. Training reduces it and does not remove it. It is the mechanism behind ineffective human review, and it affects every user, not only formal reviewers.

How it happens

• The system's answer is shown before the person has formed a view.
• Agreeing takes one click and disagreeing requires a written reason.
• Performance targets reward speed.
• The system has been right so often that checking feels wasteful.
• Confidence is displayed in a way that implies more certainty than exists.

Why it matters

Controls that rely on a person to catch the system's errors stop working, with no visible sign. The organisation believes it has human judgement in the process and in practice has the model's output with a signature. The EU AI Act names awareness of automation bias as something human oversight of high-risk systems must provide for.

Warning signs

• Reviewers agree with the system in nearly every case.
• Time spent per review has fallen steadily.
• Errors later found were approved by a person.
• Staff say they would not know how to do the task without the tool.

Who owns it

The business owner of the process is accountable for the quality of human judgement in it. Designers shape the interface and workflow. Managers set the targets that help or hinder scrutiny.

In practice

Radiographers are given a tool that flags scans as normal or abnormal. Over a year their rate of disagreement with it falls close to zero, and scans flagged as normal are closed in seconds. A class of abnormality the tool misses goes unreported. The same pattern generalises to any process in which a person is placed after a system that is usually right.