For Heads-of · Practitioner
Confusion Matrix
A table comparing a classifier's predictions with the actual outcomes, from which precision, recall and false positive and false negative rates are calculated.
- confusion matrix
- metrics
- evaluation
- classification
Structure
Rows show actual labels and columns show predicted labels. The diagonal holds correct predictions; everything off the diagonal is an error, and where an error falls shows what kind it is.
Insights
It shows whether a model tends to over-predict or under-predict particular classes, which a single accuracy figure hides. In most applications the two kinds of error carry different costs (a missed fraud against a blocked legitimate payment), and the matrix is where that trade-off becomes visible.
Fairness analysis
Separate confusion matrices for different groups reveal whether error rates differ between them, the basis of the equalised-odds test. Not producing them is the disparate impact monitoring gap; bias testing and fairness monitoring is the control that does.
Related Terms
Fairness & Bias
The principle and practice of ensuring AI systems do not produce systematically different outcomes for individuals or groups because of protected characteristics or other vulnerable attributes. Algorithmic fairness is the same idea expressed as measurable criteria.
Fairness Metrics
Quantitative tests of whether a model's outcomes or errors differ across groups. The main families are demographic parity, equalised odds and calibration, and outside special cases they cannot all be satisfied at once, so choosing which to use is a governance decision rather than a technical default.
Performance Metric
A quantitative measure of how well an AI model or system performs a task, such as accuracy, precision, recall, F1 score or a domain-specific measure.
Testing AI Systems
Systematic evaluation of AI models and systems to verify that they perform as intended, identify failure modes and assess risks.