For Heads-of · Practitioner
Testing AI Systems
Systematic evaluation of AI models and systems to verify that they perform as intended, identify failure modes and assess risks.
- testing
- evaluation
- quality assurance
- verification
Test types
Unit tests (individual components), integration tests (components together), performance tests (speed and scale), adversarial tests (robustness to attack) and fairness tests (performance across groups), plus evaluation against a fixed set of cases with an agreed pass threshold.
Challenges
AI systems are probabilistic and their input space is vast, so comprehensive testing is impossible. Edge cases are common and hard to anticipate, conventional test approaches miss risks such as bias and manipulation, and results expire when the model, prompt or data changes.
Governance expectation
Test before deployment and keep testing afterwards, rerunning the suite on every version. Adversarial testing (red teaming) is increasingly expected for high-risk systems and is required for the most capable general-purpose models under the EU AI Act. Gaps in it are the adversarial testing coverage gap; see adversarial robustness testing.
Related Terms
A/B Testing
An experimental method that compares two versions of a system (A and B) by sending each to a random subset of users, to establish which performs better.
Adversarial Attack
An attempt to manipulate or deceive an AI model by deliberately crafting inputs designed to cause misclassification or unintended behaviour.
AI Assurance
The work of measuring, evaluating and communicating whether an AI system, or the organisation running it, meets the claims made for it, so that others can rely on those claims. It spans internal testing through to independent audit and certification, and the question it answers is not whether a system is good but whether a statement about it can be trusted.
AI Evaluation
Systematic measurement of what an AI model or system can do, how it tends to behave and how well its safeguards hold, using benchmarks, task-based tests, red teaming or studies with people. Evaluations are the evidence most AI claims rest on, and they have known weaknesses.
AI Red Teaming
Structured adversarial testing in which testers try to make an AI system fail, either by producing harmful or prohibited output (safety red teaming) or by being subverted to act against its owner (security red teaming). It finds failures an ordinary test suite cannot, because the tester's job is to imagine what the designers did not.
Confusion Matrix
A table comparing a classifier's predictions with the actual outcomes, from which precision, recall and false positive and false negative rates are calculated.
Performance Metric
A quantitative measure of how well an AI model or system performs a task, such as accuracy, precision, recall, F1 score or a domain-specific measure.