Control · C-036 · Security & Resilience
Abuse monitoring
Usage of an AI system is analysed for patterns of misuse, extraction or attack, and accounts are restricted when such patterns are detected.
- Control type
- Detective
- Family
- Risks addressed
- 7
- Audience
- Heads-of · Practitioner
- Frameworks
- Principle 12 · AML.M0004 · AML.M0024
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-10
What it does
It detects what per-request controls cannot see: behaviour across many requests. Systematic querying, repeated attempts at bypass, automated use and off-purpose use show up as patterns. It also provides the evidence needed to act against an account, and early warning that a new attack technique is in circulation. The security operations function runs it continuously, with the system owner defining what counts as misuse.
How to implement it
- Define indicators: volume, repetition, systematic variation, blocked-content rate, off-topic use.
- Aggregate by account, tenant, device and network.
- Alert on thresholds and on departures from each account's baseline.
- Give analysts the means to review conversations within privacy rules.
- Apply graduated responses: warning, throttling, suspension.
- Feed confirmed cases back into filters and tests.
- Retain usage records long enough to investigate slow, distributed activity.
- Report trends in misuse to the system owner and to governance.
Evidence it operates
Design: a monitoring specification with indicators, thresholds and response actions. Operation: alert and case records with outcomes; accounts restricted; a simulated extraction pattern that raised an alert within the expected time.
Where it fits
Runs in production on usage logs. It is a primary control for model extraction and abuse of a deployed AI service, and relies on AI system monitoring and logging for its data.
Limits
Detection follows the event. Patient attackers stay under thresholds, and heavy legitimate users resemble abusers. Reviewing conversations involves personal data and needs a lawful basis and proportionate access. Monitoring that produces alerts nobody investigates is a control in name only, so staffing is part of the design.
Risks this mitigates
Primary mappings
- Abuse of a deployed AI service Medium
- Detects misuse patterns and restricts the accounts responsible. Users turn the organisation's AI system to purposes it was never offered for, including generating harmful material, at the organisation's cost and in its name.
- Generative AI used for disinformation Medium
- Detects coordinated or high-volume generation of misleading content. The organisation's generative systems, or its name and likeness, are used to produce and spread false information at scale.
- Model extraction and inference-API abuse High
- Recognises the systematic querying that extraction produces. Systematic querying of a model through its normal interface lets an attacker build a functional copy, taking the intellectual property and gaining a private replica to attack.
Supporting mappings
- Dangerous capability uplift Medium
- Identifies users probing for such help. A deployed model gives meaningful help towards chemical, biological, radiological or nuclear weapons, serious cyber attack or other mass-harm activity.
- Jailbreaking and safeguard bypass High
- Detects repeated bypass attempts and restricts the account. Crafted prompts cause a model to disregard its safety training and usage restrictions, producing content or actions the organisation has prohibited.
- Non-consensual intimate imagery and child sexual abuse material generation Critical
- Detects and restricts users attempting it. A system is offered or used to generate intimate depictions of identifiable real people without their consent, or child sexual abuse material.
- Unbounded resource consumption Medium
- Identifies the accounts driving the load. With no limit on the volume, size or cost of requests, a user or attacker drives inference far beyond what the system was sized or budgeted for, degrading service or running up charges.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| Code of Practice for the Cyber Security of AI | Principle 12 | Monitor your system's behaviour Monitoring. |
| MITRE ATLAS | AML.M0004 | Limit AI Service Query Volume and Rate Query restriction relies on detection. |
| MITRE ATLAS | AML.M0024 | AI Telemetry Logging and Monitoring Telemetry. |
Related terms
How to cite
Axiom Verity (2026). Abuse monitoring. Reference Library, C-036. https://axiom-verity.com/controls/abuse-monitoring. Reviewed 2026-11-10.
Found an error or a better source? Report a correction.