Risk · R-071 · Model & Application Security
Abuse of a deployed AI service
Users turn the organisation's AI system to purposes it was never offered for, including generating harmful material, at the organisation's cost and in its name.
- Severity
- Medium
- Audience
- Heads-of · Practitioner
- Frameworks
- Annex A.9 · Dangerous Content · LLM06
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-10
What it is
A system built on a general-purpose model can do far more than its operator intends. Abuse is use outside the intended purpose by people who are entitled to use the system: free content generation, material for fraud or harassment, automated account creation, or simply using a support assistant as a general chatbot. It overlaps with jailbreaking where restrictions have to be defeated first, but much abuse needs no bypass because no restriction was set.
How it happens
- The system answers any question, whatever its stated purpose.
- Accounts are created in bulk to use a free allowance.
- Outputs are used in spam, fraud or impersonation elsewhere.
- The interface is called by scripts instead of people.
Why it matters
The operator pays for the usage and may be associated with the result. Sustained abuse can breach the model supplier's terms and put the organisation's own access at risk.
Warning signs
- Usage unrelated to the system's purpose.
- A small number of accounts generating most of the volume.
- Conversation patterns that look automated.
- Notices from the model supplier about policy violations.
Mitigating controls
Primary controls
- Abuse monitoring Detective
- Detects misuse patterns and restricts the accounts responsible. Usage of an AI system is analysed for patterns of misuse, extraction or attack, and accounts are restricted when such patterns are detected.
- Content safety filtering Preventive
- Blocks the harmful output the misuse is seeking. Inputs and outputs are classified against defined harm categories by a mechanism independent of the main model, and are blocked, modified or escalated according to policy.
Supporting controls
- Inference usage quotas and rate limiting Preventive
- Limits how much any one user can extract. Limits on request rate, input size, output size and spend are enforced for each user, tenant and system, with alerts as limits are approached.
- Intended-use definition and use restrictions Directive
- States what the service is for, giving a basis to refuse other uses. The approved purpose, users, population and conditions of use of each AI system are stated precisely, and any use outside them requires reassessment before it begins.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| ISO/IEC 42001:2023 | Annex A.9 | Use of AI systems Use of AI systems within intended bounds. |
| NIST Generative AI Profile (NIST AI 600-1) | Dangerous Content | Dangerous, Violent, or Hateful Content The content such misuse seeks. |
| OWASP Top 10 for LLM Applications | LLM06 | Unbounded Consumption Misuse often presents as excessive consumption. |
Sources
- ISO/IEC 42001:2023 – AI management systems ISO/IEC · framework
- NIST Generative AI Profile NIST · framework
- OWASP Top 10 for LLM Applications OWASP GenAI Security Project · framework
How to cite
Axiom Verity (2026). Abuse of a deployed AI service. Reference Library, R-071. https://axiom-verity.com/risks/abuse-of-a-deployed-ai-service. Reviewed 2026-11-10.
Found an error or a better source? Report a correction.