In one paragraph

The conventional account of model degradation says the model stays still while the world moves. Input distributions shift, user behaviour changes, performance decays until somebody retrains. That account is correct for a model you trained and control, and largely wrong for a system built on a hosted language model. There, the world may be perfectly stable while the model itself changes underneath you, silently, on the provider's schedule, sometimes without a version number attached. A 2026 study of the LLM supply chain documents exactly this: execution success rates on a code-generation task falling from 52% to 10% over three months with no version change, and category-specific regressions in structured output that aggregate performance scores did not reveal. The governance implication is a reversal. The control is a gate on the update and not a retraining trigger, and the gate has to be yours, because the provider's release notes will not tell you what broke in your application.

Three words, three different things

Model drift is a statement about the environment. The relationship between inputs and outputs has shifted: the population changed, behaviour changed, or the meaning of the target moved. The model is unchanged. The world it was fitted to is not. Classically it subdivides into data drift (input distribution shifts), label drift (target distribution shifts) and concept drift (the input-to-output relationship itself changes). Concept drift most reliably defeats naive monitoring, because inputs can look entirely normal while the correct answer has changed.

Performance regression is a statement about a measurement. Something you measure got worse. It is a symptom, and drift is only one of several possible causes. Others include a model version change, a prompt change, a corpus change, a data pipeline fault, or a change in how the metric itself is computed. Treat regression as synonymous with drift and you will reach for retraining when the actual cause was a deployment.

Model versioning is a statement about the artefact. Which model, trained how, on what, deployed when, and the ability to compare and roll back. It is the record that lets you tell the other two apart. Without it a regression cannot be attributed, and drift cannot be distinguished from a change somebody made.

The relationship dictates the order of investigation. A regression is what you observe. Drift and version change are two of the things that cause it. Versioning is what lets you work out which.

Where the classic picture still holds

None of this makes conventional drift monitoring obsolete. For a model an organisation trained and hosts, covering credit scoring, fraud detection, demand forecasting and most of the estate in a regulated firm, the classic account is exactly right. So are the classic controls: compare predictions against ground truth as labels arrive, monitor feature distributions against the training baseline, and alert on degradation past a threshold agreed in advance.

Two refinements are worth the effort in that setting.

Disaggregate. Aggregate accuracy is the metric most likely to look stable while something specific breaks. Performance by demographic group, by input type, by channel and by time period is where drift shows first, and where fairness moves, which is a different thing from accuracy moving. That is the gap disparate impact monitoring describes, and why bias testing and fairness monitoring is a distinct control and not a subset of performance monitoring.

Watch the label delay. Many models cannot be evaluated against ground truth for weeks or months, because a credit decision's outcome is not known at decision time. In that gap, proxy signals and input-distribution monitoring are all you have, and the honest position is that you are detecting the conditions for degradation and not degradation itself.

What actually changes in an LLM system

For systems built on hosted language models, four things can change behaviour. Only one of them is drift in the classical sense.

The provider updates the model. This is the dominant cause and the least governed. Hosted services get modified unilaterally, sometimes without a version identifier that would let you pin or compare. The study that documented the 52%-to-10% code-generation decline also recorded routing errors affecting up to 16% of requests on one model, and output corruption injecting non-ASCII characters on another. Those are step changes in a dependency, arriving without a change request, and not gradual decay.

Somebody changed the prompt. A behaviour change made outside release control, usually for a good product reason, and usually not recorded as a change to the system's specification.

The retrieval corpus moved. Documents added, removed or reindexed. The model is identical. What it knows is not.

Tool permissions changed. The system can now do something it could not, which changes outcomes without changing outputs.

Of these, only genuine environmental shift, meaning the users and their inputs actually changing, is drift as classically defined. An LLM programme importing a classical drift-monitoring playbook wholesale ends up watching the one variable that moves least.

The AI model drift and degradation risk covers both worlds. The distinction above determines which controls apply to a given system.

Why aggregate metrics hide it

The most useful single finding in the current literature is that regressions are frequently category-specific. A 2026 study of seven models from one major provider found regressions concentrated in particular capability categories, structured JSON output generation among them, that global performance scores did not surface.

The reason is arithmetic and not mysterious. If a system does ten things and one of them degrades badly, an aggregate score moves by a tenth of the damage, which is frequently inside normal variance. The organisation sees a stable number and a broken workflow.

So an evaluation suite organised around overall quality is close to useless for change detection. It has to be organised around the categories your application actually depends on: authentication flows, data validation, structured output, refusal behaviour, tone in regulated communications, whatever they happen to be. Scored per category, with thresholds per category. Untested AI failure scenarios is largely a description of the distance between the categories a team measures and the ones the business relies on.

Three controls that fit the actual problem

The current literature's framework maps neatly onto governance language, and it fits hosted LLM systems better than the retraining-centric model-risk playbook.

A production contract. Explicit behavioural rules with measurable thresholds, specific to the application's risks and not to general quality. "The assistant should be helpful" does not qualify. "Structured output must validate against the schema in 99.5% of cases on the reference set" does. This is the artefact that makes a regression detectable instead of arguable, and it is the same shape as a risk appetite tolerance: a metric, a population, a threshold, agreed in advance by somebody accountable.

Risk-category-based testing. Evaluation prompts grouped by deployment-relevant category, instead of pooled into one score, for the reason above.

A compatibility gate. A deployer-controlled checkpoint that blocks an update violating the contract. Most organisations do not have this, and it is the piece that shifts governance from reading provider release notes to verifying behaviour yourself. Where the provider allows version pinning, the gate is enforceable. Where they do not, it at least converts a silent change into a detected one, which is the difference between an incident and a mystery.

Underneath all three sits monitoring and logging, and behind them incident response and rollback. For a hosted model that means knowing in advance what you would actually do, given that "roll back the model" may not be an option you own.

Versioning what you do not control

Model versioning as conventionally described, covering architecture, training data version, hyperparameters, metrics and change metadata, assumes you trained the thing. For a hosted model you can record almost none of that.

What you can and must version is your side of the assembly. The model identifier and version string as reported by the provider at the time of each call. The prompt version. The retrieval corpus state. The tool permission set. The evaluation results that were current when the deployment was approved.

The test is the same one that applies to LLM governance generally. For any output produced last month, can you reconstruct what produced it? If the model version was not captured per call, the answer is no, and every subsequent investigation is inference.

Two practical notes. Record the version the provider reports, and not the one you configured, because aliases like "latest" resolve differently over time. And retain evaluation results with the assembly they were run against, since an evaluation score with no recorded assembly cannot establish that anything regressed.

What to monitor, and how often

A workable minimum for a hosted-LLM system, in descending order of value.

  • Category-level evaluation against the production contract, re-run at every model version change, every prompt change, and on a fixed cadence besides. A version change you were not told about will not trigger the first two.
  • The model version string per call, alerting on change. The cheapest control on this list, and the one that turns a silent update into a known one.
  • Refusal, escalation and error rates, which move before quality scores do.
  • Output shape conformance, meaning schema validation pass rates, which is where the category-specific regressions actually surfaced.
  • Input distribution, for genuine drift, which for most LLM applications is the slowest-moving of these and should not be the primary signal.

Model monitoring as a discipline is well developed for classical ML and immature for this. The gap is not tooling, since the platforms exist. It is that most organisations have not decided what they are monitoring for, which is a governance question and not an engineering one. The engineering section covers the production side.

What to evidence

Four artefacts carry the weight for a governance function.

  • The production contract per system, meaning the categories, thresholds and populations, with an owner and an approval date.
  • The version record, showing the model version in production over time, including changes you did not initiate.
  • Evaluation results tied to assemblies, and not floating scores.
  • The regression response, showing what happened when a threshold was breached.

Auditors ask for the last one. Most programmes cannot produce it, usually because no threshold was ever specific enough to be breached.

When something gets worse, check the version before assuming the world moved. In a hosted-model system it is more often the dependency than the environment, and the two have opposite remedies. One calls for retraining or recalibration. The other calls for a gate you should have had already.