For Heads-of · Practitioner
Adversarial Attack
An attempt to manipulate or deceive an AI model by deliberately crafting inputs designed to cause misclassification or unintended behaviour.
- security
- attack
- adversarial
- robustness
Examples
Adding imperceptible noise to an image so a classifier mislabels it; planting false documents in the corpus a retrieval-augmented generation (RAG) system searches; writing instructions into content a language model will read, which is prompt injection.
Types
NIST's adversarial machine learning taxonomy (AI 100-2 E2025) groups them as evasion attacks (fooling a model at inference time), poisoning attacks (corrupting training data or the model), privacy attacks (extracting training data or the model itself) and, for generative AI, misuse and injection attacks. The register entry for the first group is adversarial evasion attacks.
Mitigation
Adversarial training, input validation, rate limiting, access controls and monitoring for unusual query patterns, checked before release and after every change through adversarial robustness testing.
Related Terms
Data Poisoning
An attack in which an adversary deliberately corrupts data an AI model learns from, to degrade it or plant hidden behaviour. It now reaches beyond training data to fine-tuning sets, retrieval corpora and agent memory, and a poisoned model can pass ordinary evaluation.
Jailbreak
A technique for inducing a language model to produce output that its safety training was intended to refuse. It targets the model's alignment rather than any particular application, and works substantially the same way wherever that model is deployed.
Membership Inference
An attack that works out whether a particular person's record was in a model's training data by examining how the model responds. Showing that someone's data was used can itself be a disclosure of personal data.
MITRE ATLAS
MITRE's knowledge base of the tactics and techniques adversaries use against AI systems, modelled on the MITRE ATT&CK framework for conventional cyber attacks and built from real-world case studies and red-team findings.
Model Extraction
Reconstructing a model's behaviour or parameters by querying it repeatedly and learning from the answers, to copy its intellectual property or to build a surrogate for planning further attacks.
Prompt Injection
An attack in which untrusted content reaching a language model is interpreted as instructions rather than data, causing the model to act on an attacker's intent instead of the operator's. It is a property of how the application assembles its context, not a flaw in any particular model.
Testing AI Systems
Systematic evaluation of AI models and systems to verify that they perform as intended, identify failure modes and assess risks.