AI incident response gap

What it is

Incident processes were built for outages and security breaches. AI produces events that fit neither: a model giving harmful advice, discriminatory outcomes discovered in a batch of decisions, a prompt injection that caused an agent to act, a hallucination sent to thousands of customers. These may not trip any existing category, so they are handled informally by whoever notices. The risk is that there is no agreed severity, no named decision-maker, no steps for containment and no link to the duties to notify regulators, covered in serious incident reporting failure.

How it happens

• The incident taxonomy has no AI categories.
• Responders do not know how to contain a model, as opposed to a server.
• The people who understand the model are not on the incident rota.
• AI faults are raised as defects in a backlog and never as incidents.

Why it matters

Time is lost deciding whether it is an incident and whose it is. Evidence is not preserved. Customers and regulators are told late or not at all. The same failure recurs because nothing was learned.

Warning signs

• The incident procedure does not mention AI.
• Past AI failures have no incident record.
• No playbook exists for withdrawing a model or disabling an agent.
• Responders have never exercised an AI scenario.

Who owns it

The head of incident management owns the process. AI system owners supply playbooks for their systems. Compliance connects the process to notification duties.

In practice

A customer assistant begins quoting a competitor's defamatory claim as fact. Marketing notices on social media. It takes a day to establish who can change the assistant, during which it continues. No incident is logged. The same pattern generalises to any organisation whose incident process recognises only the failures it was designed around.