What it is

Red teaming comes from military and cyber security practice: a team plays the adversary to find weaknesses before a real one does. For AI systems the adversary's goals are wider than in a conventional penetration test. They include extracting data, taking control of an agent's actions or exhausting its budget, but also getting the system to produce output its owner has ruled out: dangerous instructions, discriminatory decisions, defamation, or advice it should not give.

Because the input space of a language model is effectively unlimited, red teaming is exploratory rather than exhaustive. A red team does not prove a system is safe. It finds the failures that matter most and estimates how hard they are to trigger.

Safety red teaming and security red teaming

The two share techniques and differ in almost everything else.

Safety red teaming asks whether the system can be induced to produce harmful content or behaviour. Its methods include jailbreaks, role-play, multi-turn escalation and probing for bias. It needs domain experts (in medicine, finance or chemistry, depending on the use) as much as security skills, and its findings are mostly owned by whoever controls the model's behaviour.

Security red teaming asks whether the system can be subverted: prompt injection through documents and web pages, data exfiltration, abuse of an agent's tools, model extraction. It needs application security skills, and its findings are mostly owned by the team that built the application around the model.

An engagement scoped as one and sold as the other leaves a gap. Ask which it was.

How it differs from penetration testing and evaluation

A penetration test of the infrastructure around an AI system is necessary and not sufficient: it will not test whether a document in the retrieval index can steer the model. Benchmark evaluation measures performance against fixed questions and is repeatable; red teaming is adversarial and creative, and its results depend on the testers. The three are complementary. Mature programmes use evaluation for regression and red teaming for discovery, and turn each red-team finding into a regression test so that it stays fixed.

Where it is required or expected

The EU AI Act requires providers of general-purpose AI models with systemic risk to carry out and document adversarial testing (Article 55), and expects high-risk systems to be resilient against attempts to alter their use or performance (Article 15). NIST's Generative AI Profile recommends red teaming across the lifecycle. The UK AI Security Institute and the US Center for AI Standards and Innovation test frontier models before release. And the OWASP Top 10 for LLM Applications and MITRE ATLAS give a deployer a structure for scoping its own tests. OWASP's AI Testing Guide, first released in November 2025, sets out a test method covering security and wider trustworthiness.

What good looks like

  • A written scope naming the system as deployed (model, prompts, retrieval, tools), the threats in scope, and which OWASP and ATLAS items are covered.
  • Testers with the right mix: application security for the security half, domain experts for the safety half.
  • Rules of engagement covering production data, live actions and stop conditions, agreed before testing starts.
  • Findings rated and owned, each with a severity, an owner and a fix date, recorded against the register.
  • Retesting at each material change, because a model or prompt update can reopen a closed finding. Automated red-teaming tools help here, but they find variations on known attacks rather than new ones.

The gap the register records is the adversarial testing coverage gap; the control is adversarial robustness testing.

Frequently asked questions

What is AI red teaming? Adversarial testing in which testers try to make an AI system produce harmful output or act against its owner, to find failures before real adversaries or users do.

How is AI red teaming different from penetration testing? A penetration test targets the infrastructure. AI red teaming also targets the model's behaviour and the application around it, including prompt injection through content the model reads.

Is AI red teaming required by law? For providers of general-purpose AI models with systemic risk, the EU AI Act requires adversarial testing. For most deployers it is not named as a legal duty, but it is the practical way to evidence robustness and security.

How often should an AI system be red-teamed? Before release and at every material change to the model, prompts, data sources or tools, with past findings kept as regression tests.