Mis-specified fairness measurement

What it is

There is no single measure of fairness. Equal approval rates, equal error rates and equal accuracy of scores are different things, and in general cannot all hold at once. Choosing among them is a judgement about which harm matters in this use. The risk is that the choice is made by default, by a tool's settings or by what is easiest to compute. Related errors are thresholds borrowed from another jurisdiction, groups defined too broadly, and failure to look at combinations of characteristics, where a system fair to women and to older people may still be unfair to older women.

How it happens

• A fairness library's default metric is used without discussion.
• A rule of thumb from another legal system is applied as a pass mark.
• Groups are merged to obtain large enough samples.
• Intersections are not examined.

Why it matters

A passed test closes the question inside the organisation. If the test measured the wrong thing, the assurance is false and harder to dislodge than no assurance at all. The legal test is not a statistic, and a tribunal will ask about disadvantage and justification, not about the chosen metric.

Warning signs

• No document explains why the metric was chosen.
• The same metric is used for every system.
• Legal and compliance had no part in setting the standard.
• Results are reported for single characteristics only.