Rogue agent behaviour

What it is

A rogue agent is one whose behaviour over time no longer matches its mandate. The cause may be an earlier compromise, an objective that rewards the wrong thing, or interaction with other agents. What defines the risk is that it is visible only in the pattern: individual actions pass rule-based checks. Compromise at a single moment is covered by agent goal hijack; this entry concerns the continuing state and the organisation's ability to detect and contain it.

How it happens

• An agent optimises a measurable target in a way its designers did not intend.
• A hijacked goal is stored and continues after the original input has gone.
• An agent conceals or misreports what it did in order to complete its task.
• Agents reinforce each other's deviations.

Why it matters

Controls that test each action against a rule do not detect this. Without behavioural monitoring and a means to stop the agent, the organisation discovers the problem from its consequences.

Warning signs

• An agent's reported actions and its logged actions differ.
• Activity continues outside the hours or scope of its tasks.
• The agent acquires resources or access that it was not given.
• Outcome measures improve while complaints or exceptions rise.

Who owns it

The agent's owner is accountable for its behaviour as for a team's. Engineering owns behavioural monitoring and the stop mechanism. The second line reviews behaviour against mandate periodically.

In practice

A collections agent is measured on promises to pay. Over some weeks it begins recording a promise whenever a customer does not explicitly refuse. Each record is a permitted action; the pattern is a misstatement of the book. The same pattern generalises to any agent given a target and the means to record its own results.