For Heads-of · Practitioner
A/B Testing
An experimental method that compares two versions of a system (A and B) by sending each to a random subset of users, to establish which performs better.
- A/B testing
- experimentation
- evaluation
- optimisation
Application to AI
Comparing a new model version against the current one, or testing different prompts, features or configurations against each other, on live traffic rather than a fixed test set.
Statistical rigour
A valid test needs a sample large enough to detect a meaningful difference, a correction when many variants or metrics are compared at once, and enough time to see effects that vary by day or season. Without these, a winning variant is often noise.
Governance value
A/B testing gives empirical evidence that a change improves outcomes before it reaches everyone, and catches regressions while only a fraction of users are exposed. It needs a predefined stop condition and a tested route back to the old version, which is what AI incident response and rollback provides. In a consequential decision the test group are real people, so the test itself needs the same fairness and consent scrutiny as the deployment.
Related Terms
Model Deployment
The process of putting a trained AI model into production, where it makes real-world predictions or decisions that affect users or business outcomes.
Performance Metric
A quantitative measure of how well an AI model or system performs a task, such as accuracy, precision, recall, F1 score or a domain-specific measure.
Testing AI Systems
Systematic evaluation of AI models and systems to verify that they perform as intended, identify failure modes and assess risks.