No fallback for AI failure

What it is

A fallback is whatever does the job when the AI cannot: staff working a manual process, an earlier rules-based system, a simpler model, or the service running with reduced function. Fallbacks disappear without anyone deciding to remove them. The manual process is retired to realise the savings, the staff who knew it are redeployed, the old system is decommissioned. After a year the organisation can no longer do the work without the model. The risk matters most when the model has to be switched off deliberately, because it is producing harm, and there is pressure to leave it running for want of an alternative.

How it happens

• The business case depended on removing the manual capacity.
• The earlier system was switched off at go-live.
• Skills needed for the manual process have lapsed.
• A fallback exists and has never been tested at volume.

Why it matters

Every other response to AI failure, including stopping the system, assumes there is something to fall back on. Without it, the organisation must choose between running a system it knows to be faulty and stopping a service.

Warning signs

• No document describes how the process runs without the model.
• Nobody still employed has performed the task manually.
• The fallback's capacity is a fraction of normal volume.
• Withdrawal of the model has never been rehearsed.

Who owns it

The owner of the business process is accountable for continuity. The system owner designs degraded modes. The resilience function tests them.

In practice

A lender automates affordability checks and reassigns its underwriters. Eighteen months later a fault is found in the model. Six people remain who can assess a case by hand, against a thousand applications a day. The model stays in use for a further month while a fix is prepared. The same pattern generalises to any automation whose savings came from removing the ability to do without it.