Axiom Verity Independent AI governance research
Menu

Control · C-032 · Security & Resilience

AI red teaming

People skilled in attacking AI systems try to make the system fail as an adversary would, before release and periodically, and their findings are tracked to closure.

Control type
Detective
Risks addressed
13
Audience
Heads-of · Practitioner
Frameworks
Principle 3 · Principle 9 · Article 15 · Article 55
Reviewed
2026-11-10

What it does

It finds weaknesses that predefined tests do not contain. Red teamers work without a script, chain techniques together, and pursue whatever the system's purpose makes valuable to an attacker. The system owner commissions it; a team independent of the developers carries it out, before first release, after major change and on a cycle set by risk tier.

How to implement it

  • Define objectives from the system's real exposure: data it can reach, actions it can take, harm it could cause.
  • Give the team realistic access and enough time.
  • Cover the whole system, including retrieval sources, tools and human approval steps.
  • Record each finding with the method used so that it can be reproduced.
  • Convert findings into permanent cases in the adversarial test battery.
  • Report results to the system owner and to governance.

Evidence it operates

Design: a scope and agreed rules for each exercise, and evidence of the team's independence and skill. Operation: a dated report of findings; a remediation log with closure evidence; new regression tests traceable to findings.

Where it fits

Operates at validation and periodically in production. It complements adversarial robustness testing and is a primary control for jailbreaking and safeguard bypass.

Limits

An exercise is a sample in time, bounded by the team's skill and hours. Finding nothing proves little. Results lose value as the system changes, and findings that are not fixed turn the exercise into a record of known weaknesses. It is also costly, so depth should follow risk tier and not be applied uniformly.

Risks this mitigates

Primary

Adversarial testing coverage gap Medium
Extends testing to the attack classes structured tests miss. An AI system receives functional testing and no structured adversarial testing, before release or after, so the organisation has no evidence for the resilience it assumes.
AI-accelerated cyber threat High
Tests defences against AI-assisted attack techniques. Advanced models lower the cost and skill needed to find and exploit vulnerabilities, so attacks become faster, more numerous and harder to distinguish from legitimate activity.
Dangerous capability uplift Medium
Tests for uplift before release. A deployed model gives meaningful help towards chemical, biological, radiological or nuclear weapons, serious cyber attack or other mass-harm activity.
Jailbreaking and safeguard bypass High
Measures resistance to bypass techniques before release and after each change. Crafted prompts cause a model to disregard its safety training and usage restrictions, producing content or actions the organisation has prohibited.

Supporting

Cross-tenant and cross-session leakage High
Probes for leakage between sessions and tenants. Caches, memory, fine-tuned weights or shared context expose one user's or customer's data to another.
Harmful content generation High
Tests for harmful outputs before release. A system produces toxic, violent, hateful, sexual or otherwise harmful content, and that content reaches users or is published in the organisation's name.
Inadequate pre-deployment testing High
Adds adversarial testing for higher-risk systems. A system goes live without testing proportionate to its risk, including functional, safety, security and user testing in realistic conditions.
Loss of control of advanced AI Medium
Tests for unsafe autonomous behaviour. A highly capable AI system pursues objectives or acquires capabilities beyond the oversight of its developer or deployer, and cannot readily be corrected or stopped.
Membership and attribute inference Medium
Tests whether membership or attributes can in fact be inferred. Queries to a model reveal whether a particular person's record was in its training data, or reveal sensitive attributes about them.
Model inversion and training data extraction High
Attempts extraction before release. An attacker reconstructs training records, or close likenesses of them, from a model's outputs or its weights.
Prompt injection High
Finds injection routes before attackers do. Text supplied by a user, or contained in content the model retrieves or receives from a tool, is interpreted as an instruction and overrides the instructions the system was given.
Safeguard erosion through fine-tuning Medium
Checks adapted models for restored unsafe behaviour. Fine-tuning or other adaptation weakens a base model's safety behaviour, and the change goes untested because the base model had already been assessed.
System prompt leakage Medium
Tests how readily the prompt can be extracted. The instructions given to a model are extracted or inferred by a user, exposing configuration, restriction logic and anything sensitive that was placed in them.

Framework and clause references

FrameworkClauseTitle
Code of Practice for the Cyber Security of AIPrinciple 3Evaluate the threats and manage the risks to your AI system Evaluation of threats to the system.
Code of Practice for the Cyber Security of AIPrinciple 9Conduct appropriate testing and evaluation Testing.
EU AI ActArticle 15Accuracy, robustness and cybersecurity Cybersecurity.
EU AI ActArticle 55Obligations of providers of general-purpose AI models with systemic risk Adversarial testing of the most capable models.
NIST AI Risk Management Framework (AI RMF 1.0)MEASURE 2Systems are evaluated for trustworthy characteristics Evaluation.

Related terms

How to cite

Axiom Verity (2026). AI red teaming. Reference Library, C-032. https://axiom-verity.com/controls/ai-red-teaming. Reviewed 2026-11-10.

Found an error or a better source? Report a correction.