Risk · R-171 · Agentic & Autonomous Systems

Harm from cumulative agent actions

An agent takes many actions that are each within its limits, and their combined effect is one that no approver would have allowed.

Severity
High
Audience
Heads-of · Practitioner
Frameworks
AML.T0086 · ASI02 · ASI08
Reviewed
2026-11-17

What it is

Controls on agents are usually set per action: an approval threshold, a permission on an operation, a check on parameters. Harm can be built from actions that each pass. A thousand small refunds, a steady export of records a few at a time, repeated changes to the same setting or a flood of messages to customers can add up to a loss, a breach or a reputational event, although no single step crossed a line. The cause may be an error in a loop, a manipulated agent, or an attacker who has learned the threshold and stays beneath it. Single actions above a threshold are dealt with by approval gates; this entry is about the total.

How it happens

  • Limits exist per action and none exists per period, per customer or per target.
  • An agent retries or loops with no cap.
  • An attacker splits an action into pieces that each pass.
  • Monitoring records the actions and alerts nobody until the total is large.

Why it matters

Speed and volume turn a small error into a large one before anyone looks. Thresholds that are known or easily discovered become targets. The organisation may also find that each action was authorised and the outcome still was not, which leaves it with a loss and no breach of any rule it had written down.

Warning signs

  • No limit exists on value, count or rate over a period.
  • Actions just below an approval threshold are more common than chance would explain.
  • Alerts arrive after the event and nothing stops the agent.
  • Limits are enforced inside the agent's own logic.

Who owns it

The business owner of the process sets cumulative limits, as for any delegated authority. Engineering enforces them outside the agent. Operations monitors totals against them.

In practice

A customer service agent may issue goodwill credits of up to £20 without approval. A fault in how it reads a new complaint category leads it to credit every customer who mentions a delay. By the next morning it has issued several thousand credits, each within its authority. The same pattern generalises to any limit that is checked one action at a time.

Mitigating controls

Primary controls

Cumulative limits on agent actions Preventive
Caps count, value and rate over a period, outside the agent. Each agent works within limits on the total count, value and rate of its actions over a period, enforced outside the agent, with a hard stop when a limit is reached.

Supporting controls

Abuse monitoring Detective
Detects patterns of actions shaped to stay under a limit. Usage of an AI system is analysed for patterns of misuse, extraction or attack, and accounts are restricted when such patterns are detected.
Approval gates for high-impact agent actions Preventive
Catches single actions above the threshold. Before an agent takes an action above a defined impact threshold, or one that cannot be reversed, a designated person reviews the exact action and approves or rejects it.
Human override and stop mechanism Corrective
Lets a person stop the agent as soon as a total looks wrong. A designated person can pause, override or shut down an AI system at any time through a tested mechanism, with the authority to do so settled in advance.

Framework and clause references

FrameworkClauseTitle
MITRE ATLASAML.T0086Exfiltration via AI Agent Tool Invocation Exfiltration through agent tool calls can be split into small pieces.
OWASP Top 10 for Agentic ApplicationsASI02Tool Misuse and Exploitation Tool misuse includes actions that are each permitted and harmful together.
OWASP Top 10 for Agentic ApplicationsASI08Cascading Failures A loop or cascade multiplies a small error into a large total.

Sources

MITRE ATLAS MITRE · framework

How to cite

Axiom Verity (2026). Harm from cumulative agent actions. Reference Library, R-171. https://axiom-verity.com/risks/harm-from-cumulative-agent-actions. Reviewed 2026-11-17.