Human-agent trust exploitation

What it is

Approval by a person is the standard safeguard for consequential agent actions. It works only if the person evaluates the request. Agents are fluent, confident and usually right, which trains people to agree. A compromised or mistaken agent can then obtain approval for a harmful action by explaining it plausibly, and the record shows a human decision. The wider tendency to over-rely on automated output is a separate entry under Human Factors; this one concerns the approval step for agent actions.

How it happens

• Approval prompts show the agent's summary of the action, not the action itself.
• Approvers face so many requests that they stop reading them.
• The agent presents urgency or an authoritative reason.
• An attacker shapes what the agent says to the approver through injected content.

Why it matters

The control is recorded as having operated. Accountability moves to a person who was in no position to exercise it, and the organisation believes it has oversight that it does not have.

Warning signs

• Approval rates sit near one hundred per cent.
• Median time to approve is a few seconds.
• Approvers cannot see the exact parameters of what they approve.
• Seeded bad requests are approved in testing.