Unequal performance across groups

What it is

Here nothing in the system distinguishes between people; it simply works less well for some of them. Speech recognition that mishears certain accents, face matching that fails more often on darker skin, a language model that writes worse in less common languages, document reading that struggles with non-Western name formats. The cause is usually thin representation in training and testing. The effect is that members of those groups meet more errors, more friction and more refusals, and may give up.

How it happens

• Training data under-represents some groups, languages or conditions.
• Testing was done on a population unlike the one served.
• Performance is reported as an average.
• The system was built for one market and extended to another.

Why it matters

A process that is the same for everyone and works worse for a protected group can amount to indirect discrimination, and for disabled people may breach the duty to make reasonable adjustments. Where the error is a false match or a wrongful refusal, the harm to the individual is direct.

Warning signs

• Error rates have never been measured by group.
• Complaints or abandonment cluster among particular communities.
• Staff routinely step in for certain customers.
• The supplier's test results describe a different population.

Who owns it

The business owner of the system is accountable for whom it serves well. Engineering measures and improves performance by group. Procurement obtains group-level evidence from suppliers.

In practice

A bank introduces voice authentication. It works reliably for most customers and fails far more often for those with a speech impairment and for some regional accents, who are sent to a slower manual check each time. The same pattern generalises to any system tested on the people who were easiest to collect data from.