Explanation method validation

What it does

It checks that explanations are true. Before an explanation technique is used for validation, for reason codes or for explanations to customers, it is tested: does it identify the factors that actually drive the model, are its results stable, does it agree with other methods, and does changing the factor it names change the outcome. Where a model cannot be adequately explained, that is recorded and weighed in deciding where it may be used. Data science carries out the tests; independent validation reviews them; the model owner is accountable. It operates at validation and after material change.

How to implement it

• Choose explanation methods suited to the model type and the audience.
• Test them on cases where the true drivers are known, including synthetic ones.
• Check stability: similar cases should receive similar explanations.
• Compare methods, and investigate disagreement.
• Test counterfactually: alter the stated reason and confirm the result moves.
• Do not treat a language model's account of its own reasoning as an explanation without evidence.

Evidence it operates

Design: a standard for explanation methods by model tier. Operation: validation reports with faithfulness and stability results; a record of methods approved for each model; cases where a method was rejected; reassessment after model change.

Where it fits

Within model validation and ahead of any explanation given to others. It is the primary control for unfaithful explanations and, with independent model validation, for opaque model logic. It underpins explanations for affected individuals.

Limits

There is no ground truth for why a complex model produced an output, so faithfulness can be tested only indirectly. Methods that work for tabular models transfer poorly to language models. A validated method still gives an approximation. The more reliable answer, for decisions that must be explained, is often a simpler model.