Inadequate contestability and redress

What it is

Any system that decides about people will sometimes be wrong, so the ability to challenge is part of a fair process. Contestability needs four things: the person knows a decision was made and roughly why; there is a route to object; the review is done by someone who can see the evidence and is free to reach a different answer; and a wrong decision is put right, including its consequences. The risk is that one of these is missing. UK data protection law requires a means to contest significant solely automated decisions, and the EU AI Act gives affected people a right to an explanation of certain decisions made with high-risk systems.

How it happens

• The appeal route exists and customers are not told of it.
• Review consists of running the same model again.
• The reviewer cannot see how the system reached its result.
• A successful challenge corrects the decision and not the loss it caused.

Why it matters

Without effective challenge, individual errors persist and systematic ones are never found, because appeals are a main source of evidence that a system is wrong. For the person affected, the experience is of a decision nobody will answer for.

Warning signs

• Decision notices do not explain how to challenge.
• Appeals almost never succeed, or almost always do.
• Reviewers have no access to the inputs or reasons.
• No record links successful appeals back to model improvement.

Who owns it

The business owner of the decision is accountable for the review process. Complaints and operations run it. Compliance tests that it works. The model owner acts on what appeals reveal.

In practice

A tenant-screening tool marks an applicant as high risk. The letting agent tells the applicant the system said no, and that the supplier does not discuss individual results. The data behind the score belonged to someone with a similar name. The same pattern generalises to any decision passed between organisations, none of which regards the challenge as theirs to hear.