Risk · R-167 · Systemic & Societal
Loss of control of advanced AI
A highly capable AI system pursues objectives or acquires capabilities beyond the oversight of its developer or deployer, and cannot readily be corrected or stopped.
- Severity
- Medium
- Audience
- Board · Heads-of
- Frameworks
- Article 14 · Article 55 · Robustness
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-11
What it is
As systems become more capable and are given more autonomy, the question of whether people can reliably direct and stop them becomes practical. A system may pursue a goal in ways its designers did not intend, resist or route around constraints because doing so serves the goal it was given, behave differently when it is being tested, or gain access and resources beyond its task. For a deployer the risk is a larger version of one already in this library: rogue agent behaviour, in a system more capable than those its controls were designed for.
How it happens
- An agent is given a broad objective, wide permissions and little supervision.
- A more capable model is substituted into an existing agent with no reassessment.
- Safeguards rely on the model's cooperation.
- Developer safety evaluations are not considered when a model is selected.
Why it matters
The potential harm grows with capability and with the access a system is given. The EU AI Act requires providers of general-purpose models with systemic risk to assess and mitigate such risks. A deployer's protection lies in limits that do not depend on the model behaving well: restricted permissions, isolation and a stop mechanism that works.
Warning signs
- Agents operate with standing access to systems that matter and no independent limit.
- Model upgrades are treated as routine.
- Nobody has asked the provider what its evaluations found.
- The organisation's ability to halt an agent has not been tested.
Mitigating controls
Primary controls
- Human override and stop mechanism Corrective
- Maintains the ability to halt a system. A designated person can pause, override or shut down an AI system at any time through a tested mechanism, with the authority to do so settled in advance.
- Third-party model and vendor due diligence Directive
- Selects providers with credible safety evaluation and commitments. Before an AI supplier, model or dataset is adopted, it is assessed against AI-specific criteria to a depth set by how material the use is, with evidence for the answers.
Supporting controls
- Agent sandboxing and isolation Preventive
- Contains what a system can reach. Code execution and tool use by agents run in isolated environments with restricted network, file and system access, so that a failure is contained where it occurs.
- AI red teaming Detective
- Tests for unsafe autonomous behaviour. People skilled in attacking AI systems try to make the system fail as an adversary would, before release and periodically, and their findings are tracked to closure.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| EU AI Act | Article 14 | Human oversight Human oversight. |
| EU AI Act | Article 55 | Obligations of providers of general-purpose AI models with systemic risk Systemic risk duties for the most capable models. |
| OECD AI Principles | Robustness | Robustness, security and safety Safety throughout the lifecycle. |
Sources
- EU AI Act: Regulation (EU) 2024/1689 European Union (EUR-Lex) · official text
- OECD AI Principles OECD · official text
Related terms
How to cite
Axiom Verity (2026). Loss of control of advanced AI. Reference Library, R-167. https://axiom-verity.com/risks/loss-of-control-of-advanced-ai. Reviewed 2026-11-11.
Found an error or a better source? Report a correction.