Wrongful automated refusal

What it is

Automated controls for fraud, financial crime, credit and claims are tuned to stop bad cases. Every such system also stops some good ones. A payment is blocked, an account frozen, a claim declined, an application refused, and the customer was legitimate. The error rate may be low, and at scale that is still many people. What turns an error into a conduct failure is what happens next: no reason given, no one to speak to, and no route by which the decision is looked at again. The legal safeguards for significant automated decisions are covered in automated decision-making without safeguards.

How it happens

• Thresholds are set to minimise loss, with no measure of the cost to legitimate customers.
• Staff cannot see why the system acted, or are told not to say.
• Staff have no authority to override.
• Reasons are withheld for fear of tipping off, in cases where that concern does not apply.

Why it matters

Losing access to an account or a payment can mean missed rent or wages. Refusals that fall more often on some groups raise discrimination questions as well. Regulators expect firms to weigh the harm of false positives, not only the harm of what gets through.

Warning signs

• False positive rates are not measured or reported.
• Overturn rates on review are high, or reviews never overturn.
• Complaints describe being unable to find out why.
• Blocked customers are concentrated in particular groups or areas.

Who owns it

The owner of the control, such as fraud or credit risk, is accountable for its errors as well as its catches. Operations handle review. Compliance monitors outcomes.

In practice

A bank's fraud model freezes the account of a customer who receives several payments from abroad after selling a car. The app says the account is under review. The telephone team can see the freeze and nothing else. It takes three weeks to lift. The same pattern generalises to any automated control whose designers were measured only on what it stopped.