Poor quality or unrepresentative training data

What it is

A model can be no better than its data. Quality has several dimensions: whether records are correct, whether fields are complete, whether the data is current, whether labels are right, and whether the data covers the people and situations the system will meet. This entry concerns defects with no deliberate cause. Deliberate manipulation is covered in data and model poisoning; data that under-represents particular groups is also a fairness risk, covered in training data bias.

How it happens

• Data is taken from an operational system built for another purpose.
• Labels were assigned inconsistently or by an earlier, flawed process.
• Historical data reflects products, customers or rules that no longer apply.
• Records with missing values are dropped, removing a whole segment.

Why it matters

Errors in data become errors in every output, and they are hard to see because the model fits the data it was given. Remedying the defect requires better data and retraining, which may not be possible quickly.

Warning signs

• No data quality assessment exists for the training set.
• The origin and meaning of fields are known only to one person.
• The training population differs visibly from current customers.
• Performance in production is below performance in testing.

Who owns it

The data owner is accountable for the quality of data supplied. The model owner is accountable for judging whether it is adequate for the purpose.

In practice

A claims triage model is trained on five years of handled claims. For two of those years a backlog meant that complex claims were coded with a default category. The model learns that the default category is routine and routes complex claims to the fast track. The same pattern generalises to any model trained on operational data whose history was not examined.