An account, for someone

The word that does the work in any explainability requirement is whom. An explanation adequate for a data scientist debugging a model is not adequate for a customer contesting a decision, and neither is adequate for a regulator assessing whether the system is lawful. The same system needs different accounts for different audiences, and an organisation that has built one and calls the problem solved has usually built the developer-facing one.

For an affected person, an account is adequate when it lets them do something: understand what mattered, judge whether the inputs were correct, and know how to contest the outcome. That is a considerably higher bar than a list of feature attributions, and it is the bar regulation is drifting towards.

How it differs from interpretability

Interpretability is a property of the model's mechanism. Explainability is a property of the account given about it.

The relationship is asymmetric in a way worth internalising. An interpretable model makes explanation easier and does not supply it — someone still has to render the mechanism into terms the audience can use. An opaque model can be explained perfectly adequately for regulatory purposes, using post-hoc methods, provided the explanation pipeline is itself governed. Neither property implies the other.

Post-hoc explanation, and the faithfulness problem

The standard techniques — SHAP, LIME, counterfactual explanations, feature importance, attention visualisation — construct an account of a decision after the fact, usually by fitting a simpler surrogate around the model's behaviour near that decision.

They share a limitation that is easy to state and easy to forget: a post-hoc explanation reports what the surrogate attributes, not what the model did. It can be plausible and unfaithful at once, and it fails silently, because an unfaithful explanation looks exactly like a faithful one.

Three practical consequences:

  • Explanations need validating, at least by sampling — checking that the factors an explanation cites actually change the output when perturbed.
  • Counterfactual explanations are frequently the most useful for an affected person, because "your application would have succeeded if X had been Y" is directly actionable in a way an attribution score is not.
  • Disagreement between two explanation methods on the same decision is a finding, not a curiosity.

What the law actually requires

EU AI Act, Article 86 gives a person subject to a decision by a deployer, taken on the basis of output from an Annex III high-risk system (excluding point 2), that produces legal effects or significantly adversely affects their health, safety or fundamental rights, the right to obtain clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken. It does not apply where Union or national law provides an exception, or where the right already exists elsewhere in Union law.

Note what that wording asks for and what it does not. It asks for the role of the system and the main elements of the decision — not the model's internals, and not a technical attribution. An organisation that answers with SHAP values has answered a different question.

UK and EU GDPR, Article 22 gives rights in relation to solely automated decisions producing legal or similarly significant effects, including the right to obtain human intervention, express a point of view and contest the decision — and, through Articles 13–15, meaningful information about the logic involved. The ICO's position is that this means explaining the reasoning in a way the person can engage with, not disclosing the algorithm.

The two regimes overlap and are not identical, and a system can engage one without the other. See the compliance crosswalk.

What an explanation capability actually consists of

Not a library. Four things:

  • A defined audience and adequacy standard per system — what "understandable" means for this population, decided in advance rather than argued afterwards.
  • A generation path that produces the account reliably at the time of decision, not reconstructed months later from logs that may not contain what is needed.
  • A validation step that samples explanations for faithfulness.
  • A contest route — because Article 22 and Article 86 both assume the person can do something with the explanation, and an explanation with no route to challenge is a notification.

The fourth is the one most often missing, and it is not an explainability problem in the technical sense at all. It is a process one.

Where it sits in the register

Inadequate explainability for affected individuals is the register entry; automated decision-making safeguards is the control where the Article 22 machinery lives; and human-in-the-loop review is the usual compensating control where a model cannot be adequately explained but the decision must still be made.

One further link that is easy to miss: you cannot explain a decision to someone who does not know a machine made it. Explainability presupposes the disclosure obligation in undisclosed AI interaction, which since 2 August 2026 has been live under Article 50.