Risk · R-067 · Model & Application Security
Malicious model artefacts and unsafe loading
A model file or dependency obtained from a public source executes code or carries hidden behaviour when loaded.
- Severity
- High
- Audience
- Heads-of · Practitioner
- Frameworks
- AML.T0011 · AML.T0115.001 · LLM04
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-10
What it is
Model files are not inert data. Several common formats can contain executable code that runs when the file is opened, and a model can depend on custom code that is fetched and run alongside it. A file published under a trusted-looking name can therefore act as malware. This entry concerns the file as a delivery mechanism. A model whose learned behaviour is hostile is covered in model backdoors and trojans.
How it happens
- A model in a format that permits code execution is downloaded and loaded.
- A repository requires its own loading code, which is run without review.
- A popular model's name is imitated by a hostile upload.
- A legitimate publisher's account is taken over and a file replaced.
Why it matters
The code runs with the privileges of whoever loads the model, often a data scientist's workstation or a training server with access to data and credentials. It happens before any evaluation of the model begins.
Warning signs
- Models are downloaded directly from public hubs to working machines.
- No policy states which file formats may be loaded.
- Loading options that allow remote code are enabled by default.
- Artefacts are not scanned or checked against a published hash.
Who owns it
Security sets the standard for obtaining and loading artefacts. The platform owner enforces it. Each team is accountable for the artefacts it brings in.
In practice
An engineer evaluating open models for a summarisation task downloads one that tops a public leaderboard. Loading it starts a process that copies the workstation's cloud credentials to an external server. The same pattern generalises to any artefact loaded from outside before it has been checked.
Mitigating controls
Primary controls
- Model artifact signing and integrity verification Preventive
- Verifies the artefact is the one its publisher released. Approved model artefacts are cryptographically signed at validation, and the signature is verified before an artefact is loaded anywhere in production.
- Safe model loading and artefact scanning Preventive
- Scans model files and loads them only in formats that cannot execute code. Model files and their dependencies are obtained only from approved sources, scanned before use, and loaded in formats and with settings that cannot execute embedded code.
Supporting controls
- AI bill of materials Detective
- Records which third-party artefacts are in use. The models, datasets, libraries, tools and hosted components that make up each AI system are recorded and checked against known vulnerabilities, licence terms and supplier changes.
- Third-party model and vendor due diligence Directive
- Assesses the source before its artefacts are trusted. Before an AI supplier, model or dataset is adopted, it is assessed against AI-specific criteria to a depth set by how material the use is, with evidence for the answers.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| MITRE ATLAS | AML.T0011 | User Execution Execution of an unsafe artefact by the victim. |
| MITRE ATLAS | AML.T0115.001 | Publish Poisoned AI Artifacts: Models Publication of poisoned models. |
| OWASP Top 10 for LLM Applications | LLM04 | Supply Chain A supply chain entry point. |
Sources
- OWASP Top 10 for LLM Applications OWASP GenAI Security Project · framework
- MITRE ATLAS MITRE · framework
Related terms
How to cite
Axiom Verity (2026). Malicious model artefacts and unsafe loading. Reference Library, R-067. https://axiom-verity.com/risks/malicious-model-artefacts. Reviewed 2026-11-10.
Found an error or a better source? Report a correction.