For Heads-of · Practitioner
Model Backdoor
Also known as: model trojan · trojaned model
Hidden behaviour planted in an AI model that stays dormant until a specific trigger appears in the input, then makes the model act as the attacker intends. A backdoored model can behave normally on every ordinary test.
- backdoor
- trojan
- sleeper agent
- trigger
- model integrity
How it gets there
Through data poisoning during training or fine-tuning, or by directly altering the weights of a model that is then shared, for example on a public model hub. The trigger can be a word, a phrase, a date or a pattern in an image.
Why testing is not enough
Research published in January 2024 trained language models to write secure code when the prompt gave one year and to insert vulnerabilities when it gave another. The behaviour survived standard safety training. Testing can only find triggers someone thought to try; the defence is knowing where a model came from and that it has not been altered.
Where to go next
This entry defines the term. The register entry model backdoors and trojans sets out the controls, including model artefact signing and integrity verification and third-party model and vendor due diligence.
Sources
Related Terms
AI Supply Chain Security
Assuring the integrity and provenance of everything an AI system is built from and depends on: pre-trained models and weights, adapters, datasets, libraries, serving infrastructure, plug-ins and tools. AI adds components that ordinary software supply-chain controls were not built to check.
Data Poisoning
An attack in which an adversary deliberately corrupts data an AI model learns from, to degrade it or plant hidden behaviour. It now reaches beyond training data to fine-tuning sets, retrieval corpora and agent memory, and a poisoned model can pass ordinary evaluation.
Model Extraction
Reconstructing a model's behaviour or parameters by querying it repeatedly and learning from the answers, to copy its intellectual property or to build a surrogate for planning further attacks.
Scheming
A model covertly pursuing goals its developers did not intend while concealing them, for example by deliberately underperforming, hiding information or working around oversight. It has been observed in controlled tests; how often it occurs in real deployments is not known.