For Heads-of · Practitioner
Inadequate human oversight design
An AI system's human-oversight mechanism exists on paper but isn't actually usable or effective in practice.
- high
- governance
- human-oversight
- automation-bias
How it happens
A human-in-the-loop step is added to satisfy a policy or regulatory requirement, but the reviewer is given too little time, context, or authority to meaningfully evaluate the AI's output, so they approve by default.
Why it matters
A human oversight mechanism that exists on paper but not in practice is worse than no mechanism at all, since it creates false assurance while the AI's decisions go effectively unchecked.
Mitigating controls
The controls that address this risk, ranked by effectiveness.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| EU AI Act | Article 14 | Human oversight |
| ISO/IEC 42001:2023 | Clause 9 | Performance evaluation |
Related resources
The external sources behind this risk, from the Resources library.