Opaque model logic

What it is

Some models are understandable by inspection: a scorecard, a short decision tree. Most modern ones are not. A deep network or a large language model contains millions or billions of parameters, and no person can trace why a given input produced a given output. Techniques exist that approximate an explanation, and they have limits of their own. The risk is internal: the organisation relies on a system it cannot interrogate. It does not know what the model has learned, whether it is right for the right reasons, or how it will behave on a case unlike those tested. What the affected person is told is a separate matter, covered in inadequate explainability for affected individuals.

How it happens

• A complex model is chosen for a small gain in accuracy over an interpretable one.
• A supplier's model is used as a black box.
• Explanation tooling was not built, or was built and never checked.
• A general-purpose language model is used for a decision task.

Why it matters

Validation is reduced to testing inputs and outputs, which cannot show that the model's reasoning is sound. Faults such as reliance on a spurious feature go unseen until they cause harm. When a decision is challenged, the organisation cannot say why it was made. Supervisors of financial firms expect the degree of explainability to be weighed in model risk.

Warning signs

• Nobody can state the main drivers of the model's outputs.
• Model choice was made on accuracy alone.
• Validation contains no examination of what the model relies on.
• Unexpected outputs are explained after the event by guesswork.

Who owns it

The model owner is accountable for understanding the model to the degree its use requires. Data science provides explanation methods. Independent validation tests them. The business owner decides whether the remaining opacity is acceptable for the decision.

In practice

A hospital trust adopts an image model that performs very well at detecting a condition in testing. Later analysis finds it had learned to recognise the scanner type used in the specialist unit, where most positive cases were imaged. The same pattern generalises to any model trusted on its results by people who could not see what it had learned.