Training data bias and discriminatory outcomes

What it is

A model learns what its data shows. If past decisions favoured some groups, the model learns the preference as if it were a rule. If some groups are thinly represented, it learns them poorly. If labels were applied by people with their own biases, it inherits those too. The model then applies what it learned consistently and at scale, which can make the result worse than the history it was trained on. This entry is about bias that arrives through data. The way a model can discriminate without using a protected characteristic at all is covered in proxy discrimination via correlated features.

How it happens

• Historical decisions used as training labels reflect past discrimination.
• Some groups are under-represented, or absent, in the data.
• The outcome being predicted is itself a biased measure, such as arrests standing for offending.
• Labelling was done inconsistently between groups.
• A pre-trained model brings bias from data the organisation never saw.

Why it matters

In lending, recruitment, insurance, housing and public benefits, a biased outcome can be unlawful discrimination under the Equality Act 2010, and intention is no defence to the indirect form. The people affected are refused or disadvantaged without knowing why, and the pattern is invisible in any single case.

Warning signs

• Nobody has examined how the training labels were produced.
• The make-up of the training data differs from the population served.
• Outcomes differ by group and no one can explain why.
• The supplier cannot describe its training data.

Who owns it

The business owner of the decision is accountable for its fairness. The data owner answers for the data. Data science tests and reports. The second line challenges.

In practice

A recruitment model is trained on ten years of appointment decisions in a function where few women were appointed. It learns to mark down applications that mention women's clubs or career breaks. The same pattern generalises to any model taught to repeat past human decisions without asking whether those decisions were fair.