Control · C-089 · Security & Resilience
AI failure scenario testing
Severe but plausible AI failure scenarios are exercised against impact tolerances on a set cycle, and the lessons are acted on.
- Control type
- Detective
- Family
- Risks addressed
- 6
- Audience
- Board · Heads-of · Practitioner
- Frameworks
- Article 6 · Scenario testing
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-11
What it does
It tests whether the organisation could stay within tolerance if its AI failed. Scenarios particular to AI are added to the resilience testing programme and run as exercises or live tests: the model provider is down for days, the model degrades without alarm, an update changes behaviour, the model must be withdrawn immediately, a compromised component is discovered. Each exercise records whether the service stayed within tolerance and what must change. The resilience function runs the programme; service and system owners take part. It operates at least annually and after significant change.
How to implement it
- Build an AI scenario library from incidents, near misses and the dependencies on the map.
- Include silent failure, where nothing reports a fault.
- Test detection as well as recovery: how long before anyone would notice.
- Invoke the fallback at realistic volume.
- Involve the supplier where the scenario depends on it.
- Record findings, assign actions and report results to the board.
Evidence it operates
Design: a test programme that includes AI scenarios. Operation: exercise reports with outcome against tolerance; actions tracked to closure; scenarios added after incidents; board reporting of results.
Where it fits
In the resilience testing cycle. It is the primary control for untested AI failure scenarios, and supports no impact tolerance for AI-dependent services and no fallback for AI failure. It tests against the tolerances set through important business service impact-tolerance mapping.
Limits
An exercise shows how people respond to a scenario they were told about, in working hours. Desk-based tests reveal gaps in plans and not in systems. Live tests of AI failure carry real risk to customers and are often avoided for that reason. Scenarios reflect what their authors could imagine.
Risks this mitigates
Primary mappings
- Untested AI failure scenarios Medium
- Exercises severe but plausible AI failures against tolerances. Scenario testing does not include AI-specific disruption, so the organisation does not know whether it could stay within tolerance if its AI failed.
Supporting mappings
- AI-accelerated cyber threat High
- Exercises a rapid, AI-enabled attack. Advanced models lower the cost and skill needed to find and exploit vulnerabilities, so attacks become faster, more numerous and harder to distinguish from legitimate activity.
- Cascading failures in agent workflows High
- Exercises cascade scenarios. An error or compromise in one agent or step propagates through chained agents and automated workflows faster than people can notice or intervene.
- Model monoculture and herding High
- Exercises a common-mode failure. Many organisations rely on the same few models, providers and data, so they make the same errors, react in the same way and fail at the same time.
- No fallback for AI failure High
- Proves the fallback works under realistic conditions. When an AI component fails or has to be withdrawn, no manual process, simpler model or degraded mode is ready to take over.
- No impact tolerance for AI-dependent services High
- Tests whether the tolerance can be met. A service that depends on AI has no impact tolerance that covers AI-specific disruption, such as a model that is available and wrong.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| Digital Operational Resilience Act (DORA) | Article 6 | ICT risk management framework Testing within the ICT risk framework. |
| SS1/21: Operational Resilience | Scenario testing | Scenario testing Scenario testing. |
Related terms
How to cite
Axiom Verity (2026). AI failure scenario testing. Reference Library, C-089. https://axiom-verity.com/controls/ai-failure-scenario-testing. Reviewed 2026-11-11.
Found an error or a better source? Report a correction.