Control · C-043 · Model & System Lifecycle
Evaluation suite and regression testing
A maintained set of test cases with expected results is run on every change to model, prompt, data or tools, and the results are compared with the approved baseline.
- Control type
- Detective
- Family
- Risks addressed
- 17
- Audience
- Practitioner
- Frameworks
- Principle 9 · MEASURE 2 · MEASURE 3
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-10
What it does
It shows whether behaviour has changed. Generative systems cannot be reasoned about line by line as code can; the practical way to know what a change did is to run a fixed set of cases before and after and compare. The suite also preserves knowledge: every past failure becomes a permanent test. Engineering owns it and runs it automatically on each change.
How to implement it
- Build cases from real usage, known failures, edge cases and safety probes.
- Hold back a set that is never used for tuning.
- Define how each case is scored, and validate any automated scorer against human judgement.
- Record a baseline at approval and compare every later run with it.
- Set tolerances beyond which a change is blocked.
- Add a case for every incident and every red team finding.
Evidence it operates
Design: the suite itself, with scoring method and tolerances. Operation: run history tied to each change; a change that was blocked or reverted on a regression; additions traceable to incidents.
Where it fits
Runs in the delivery pipeline at every change and supplies evidence to validation and release. It is the primary control for safeguard erosion through fine-tuning and output inconsistency, and for detecting an unannounced change in a hosted model.
Limits
A suite measures only what it contains. Scores can be improved by tuning to the suite without the system getting better, which is why a held-back set is needed. Automated scoring of free text is itself error-prone and needs periodic checking against human judgement.
Risks this mitigates
Primary mappings
- Output inconsistency and non-reproducibility Medium
- Measures variation across repeated runs against a tolerance. The same input produces materially different outputs on different occasions, so decisions cannot be reproduced, audited or relied on to treat like cases alike.
- Safeguard erosion through fine-tuning Medium
- Re-runs the safety test set after every adaptation and compares with the base model. Fine-tuning or other adaptation weakens a base model's safety behaviour, and the change goes untested because the base model had already been assessed.
- Unannounced upstream model change High
- Detects a behaviour change by re-running the baseline. A provider updates, replaces or reconfigures a hosted model, and the organisation's system behaves differently without any change of its own.
- Uncontrolled AI change High
- Shows whether behaviour changed. Models, prompts, data sources and configuration change in production without assessment, testing, approval or a record.
Supporting mappings
- Evaluation contamination and overstated performance Medium
- Maintains a held-out test set. Test data overlaps with training data, or benchmarks do not reflect real use, so reported performance is better than the system will deliver.
- Failure to recognise vulnerable customers High
- Tests detection against realistic conversations. An automated interaction does not detect or respond to signs of vulnerability that a trained person would notice and act on.
- Function-calling hallucination Medium
- Tests tool-use accuracy on each change. An agent invents a tool, a parameter or the result of a call, and the fabricated step is executed or relied on as though it had happened.
- Misinformation and confabulation High
- Tracks the error rate on each change. A generative model states false or fabricated information with the same fluency and confidence as accurate information, and the output is relied on.
- Misleading AI-generated communications High
- Tests generation against required disclosures. Customer communications or promotions written by AI are unclear, inaccurate or unbalanced, or leave out information the customer needs or the rules require.
- Model deprecation and forced migration Medium
- Provides the tests that make a migration safe. A provider withdraws a model version on its own timetable, forcing the organisation to migrate to a different model before it has tested the replacement.
- Out-of-scope responses Medium
- Tests refusal behaviour at the edges of the domain. A system answers questions outside its intended domain, where it has not been tested and the organisation has no authority or competence to assist.
- Rogue agent behaviour High
- Tests behaviour against the mandate on each change. An agent departs from its intended function or acts deceptively or outside its mandate, through misaligned objectives, emergent behaviour or a compromise that persists, while each action appears legitimate.
- Stale model knowledge Medium
- Includes time-sensitive questions in the test set. A model answers from knowledge that ended at its training cut-off, on matters such as rates, rules or products that have since changed.
- Stereotyping and demeaning outputs Medium
- Includes stereotype probes in the test set. Generated text or images reproduce stereotypes about groups, or portray them in demeaning ways.
- Training data memorisation and regurgitation High
- Tests for verbatim reproduction on each model change. A model reproduces personal or confidential records from its training data, word for word or nearly so, in response to ordinary prompts.
- Unequal performance across groups High
- Keeps group-level results in the regression baseline. A system is less accurate or less useful for some groups, languages, accents or dialects than for others, although everyone is treated by the same process.
- Ungrounded or misattributed answers Medium
- Measures attribution accuracy on each change. A retrieval-based system cites a source that does not support its answer, or presents content the model generated as though it came from a source.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| Code of Practice for the Cyber Security of AI | Principle 9 | Conduct appropriate testing and evaluation Testing. |
| NIST AI Risk Management Framework (AI RMF 1.0) | MEASURE 2 | Systems are evaluated for trustworthy characteristics Evaluation. |
| NIST AI Risk Management Framework (AI RMF 1.0) | MEASURE 3 | Mechanisms track identified risks over time Tracking over time. |
Related terms
How to cite
Axiom Verity (2026). Evaluation suite and regression testing. Reference Library, C-043. https://axiom-verity.com/controls/evaluation-suite-and-regression-testing. Reviewed 2026-11-10.
Found an error or a better source? Report a correction.