For Heads-of · Practitioner
AI Evaluation
Also known as: AI evals · model evaluation
Systematic measurement of what an AI model or system can do, how it tends to behave and how well its safeguards hold, using benchmarks, task-based tests, red teaming or studies with people. Evaluations are the evidence most AI claims rest on, and they have known weaknesses.
- evals
- benchmarks
- benchmark contamination
- capability evaluation
- testing
Kinds of evaluation
Capability evaluations measure what a model can do. Propensity evaluations measure what it tends to do when it could do otherwise. Safeguard evaluations measure whether protections hold under pressure, which is where AI red teaming comes in. For a deploying organisation the most useful is the one it runs itself: a fixed set of cases from its own use, with a pass threshold, rerun on every model or prompt change.
Why results mislead
Benchmark contamination: test questions, or close variants, leak into training data and inflate scores. Weak elicitation: a capability exists but was not drawn out, so risk is understated. Evaluation awareness: a model behaves differently when it recognises a test. And general benchmarks rarely match a particular use, so a vendor's leaderboard position says little about performance in yours.
What good looks like
Treat vendor benchmark claims as marketing until reproduced on your own cases; keep a private evaluation set the model cannot have seen; record which model version was evaluated; and rerun at every change. Gaps here are untested AI failure scenarios, and adversarial robustness testing covers the adversarial half.
Sources
Related Terms
AI Assurance
The work of measuring, evaluating and communicating whether an AI system, or the organisation running it, meets the claims made for it, so that others can rely on those claims. It spans internal testing through to independent audit and certification, and the question it answers is not whether a system is good but whether a statement about it can be trusted.
AI Red Teaming
Structured adversarial testing in which testers try to make an AI system fail, either by producing harmful or prohibited output (safety red teaming) or by being subverted to act against its owner (security red teaming). It finds failures an ordinary test suite cannot, because the tester's job is to imagine what the designers did not.
Performance Metric
A quantitative measure of how well an AI model or system performs a task, such as accuracy, precision, recall, F1 score or a domain-specific measure.
Sandbagging
Strategic underperformance on an evaluation, so that a model appears less capable than it is. It can be done by the model itself or arranged by a developer, and either way it undermines the evidence that safety decisions rest on.
Testing AI Systems
Systematic evaluation of AI models and systems to verify that they perform as intended, identify failure modes and assess risks.