Unfaithful explanations

What it is

Explanation methods produce an account of a model's behaviour, and the account can be wrong. Feature-attribution techniques approximate a complex model with a simpler one and can disagree with each other on the same decision. A language model asked to explain its answer writes a plausible rationale, which is generated in the same way as any other text and need not correspond to what produced the answer. Reason codes may be assigned by a separate rule that only loosely tracks the model.

How it happens

• An explanation tool is adopted and its output is taken at face value.
• A model's own account of its reasoning is presented as the reason.
• Reason codes are mapped from scores by a table nobody has tested.
• Different methods give different answers and one is chosen for convenience.

Why it matters

People act on explanations: a customer changes what they were told mattered, a validator signs off a model, a reviewer accepts an output. If the explanation is unfaithful, each of those acts rests on a mistake. An organisation that gives a false reason for a decision may be in a worse position than one that gave none.

Warning signs

• Explanation methods have never been tested against known model behaviour.
• Two methods disagree on the main factors.
• Changing the factor named in an explanation does not change the outcome.
• The system's self-explanation is logged as the reason for a decision.