For Heads-of · Practitioner
Model backdoors and trojans
A hidden, trigger-activated backdoor is embedded in a model via poisoned training data, weight tampering, or a payload injected into the model artifact.
- critical
- adversarial-ml
- backdoor
- mitre-atlas
How it happens
A model is trained, fine-tuned, or shipped with a hidden trigger, an unusual phrase, an image watermark, a specific input pattern, that causes it to produce an adversary-chosen output on demand, while behaving normally otherwise.
Why it matters
A backdoored model passes every ordinary evaluation, since the malicious behaviour only appears for the trigger the adversary knows and no one else is testing for.
Mitigating controls
The controls that address this risk, ranked by effectiveness.
Adversarial robustness testing
Structured red-teaming and adversarial testing of an AI system against known attack techniques before deployment and on a recurring cadence after.
Model artifact signing and integrity verification
Cryptographic signing of trusted model artifacts and verification of that signature before a model is loaded into production.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| MITRE ATLAS | AML.T0018 | Backdoor ML Model |
Related resources
The external sources behind this risk, from the Resources library.