Uncontrolled AI change

What it is

Approval attaches to a system as it was when assessed. AI systems change in more ways than conventional software, and many of those ways do not look like a release: a prompt edited in a console, a retrained model, a refreshed dataset, a new document source, a different setting, or a supplier updating the model behind an interface. Each can alter behaviour as much as a code change. The risk is that these happen outside change control. Supplier-initiated change is treated further under Third-Party & Supply Chain.

How it happens

• Only code changes go through the change process.
• Prompts and settings can be edited directly in production.
• Retraining runs on a schedule with no review of the result.
• Nobody recognises a data refresh as a change.

Why it matters

Most of the risk in a system's life comes from what happens after its first approval. An unassessed change can undo validation, alter fairness, weaken safeguards or move the system into a different regulatory class, and without a record nobody can tell when behaviour changed or why.

Warning signs

• The production prompt differs from the version in the repository.
• No log shows who changed what and when.
• Behaviour shifts with no corresponding release.
• Retrained models are deployed automatically.

Who owns it

The system owner is accountable for the state of the system in production. Engineering operates change control. Governance defines what counts as a change and which changes need re-approval.

In practice

A product manager adjusts the instructions of a customer assistant to make it more helpful about fees. The edit is made in the supplier's console on a Friday. From Monday the assistant begins promising fee waivers that the firm does not grant. The same pattern generalises to any system whose behaviour can be altered without passing through a controlled process.