Evaluation contamination and overstated performance

What it is

Evaluation is meant to estimate how a model will perform on cases it has not seen. That estimate is inflated when the test cases were in fact seen during training, when the test set was used repeatedly to tune the model, or when the benchmark measures something easier than the real task. For large pre-trained models the problem is acute: public benchmarks are likely to be in the training data.

How it happens

• The same records appear in training and test sets.
• Information from the future, or from the outcome, leaks into the inputs.
• A model is tuned against the test set until the score improves.
• A supplier quotes results on public benchmarks the model may have memorised.

Why it matters

Approval, pricing of risk and the choice between models all rest on evaluation results. Overstated results lead to deploying a model that then underperforms, and the gap is discovered only in production.

Warning signs

• Test performance is markedly better than early production performance.
• No held-out dataset is reserved for final evaluation.
• Evaluation data was assembled after the model was built, by the same team.
• Benchmark scores are the only evidence offered.