For Heads-of · Practitioner
Adversarial Attack
An attempt to manipulate or deceive an AI model by deliberately crafting inputs designed to cause misclassification or unintended behaviour.
- security
- attack
- adversarial
- robustness
Examples
Adding imperceptible noise to an image to cause misclassification; injecting false documents into a retrieval-augmented generation (RAG) system; prompt injection to override model instructions.
Types
Evasion attacks (fool a model at inference time), poisoning attacks (corrupt training data), model theft (reverse-engineer/extract the model), backdoor attacks (plant hidden triggers).
Mitigation
Adversarial training, input validation, rate limiting, access controls, and monitoring for unusual query patterns.