For Heads-of · Practitioner
Feature Attribution
Methods that estimate how much each input contributed to a model's prediction, such as SHAP and LIME. They are standard in model validation and explanation, but they approximate the model rather than describe it, and they can be manipulated.
- SHAP
- LIME
- explainability
- model validation
- post-hoc explanation
How they work
SHAP (Lundberg and Lee, 2017) gives each feature a share of a prediction using Shapley values from game theory, averaged over combinations of features. LIME (Ribeiro and others, 2016) fits a simple, interpretable model around a single prediction to see which inputs drive it locally. Both work on any model, which is why they are so widely used.
Their limits
They describe the model's behaviour near a decision, not its mechanism, and different methods can give different answers for the same prediction. Correlated features confuse them. Research has also shown that a model can be built to hide a discriminatory feature from these methods while still using it, so an attribution that shows no problem is not proof there is none.
Using them well
Use them for validation and investigation, check them with perturbation tests, and do not hand raw attribution scores to an affected person, who needs a plain account such as a counterfactual explanation. The register entry is inadequate explainability for affected individuals; bias testing and fairness monitoring uses them to look for hidden drivers.
Sources
- A Unified Approach to Interpreting Model Predictions (Lundberg and Lee)(opens in a new tab)
- Why Should I Trust You? Explaining the Predictions of Any Classifier (Ribeiro, Singh and Guestrin)(opens in a new tab)
- Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods (Slack et al.)(opens in a new tab)
Related Terms
Counterfactual Explanation
An explanation of an automated decision that states the smallest change to the inputs that would have changed the outcome, such as an application that would have been approved with a higher declared income. It tells the affected person what would have made the difference without exposing the model.
Explainability
The ability to give a person an account of why a system produced a particular output, in terms they can act on. Explainability is a property of the account and its audience, not of the model: an opaque model can be explainable, and an interpretable one can fail to be explained.
Fairness Metrics
Quantitative tests of whether a model's outcomes or errors differ across groups. The main families are demographic parity, equalised odds and calibration, and outside special cases they cannot all be satisfied at once, so choosing which to use is a governance decision rather than a technical default.
Mechanistic Interpretability
Research that tries to reverse-engineer what a neural network computes internally, identifying the features it represents and the circuits that connect them, in terms a person can understand. It is a research field, not yet an audit tool.
Model Interpretability
An intrinsic property of a model: the degree to which a person can understand the mechanism by which it turns inputs into outputs. A linear model or a shallow decision tree is interpretable by construction. A deep network is not, however good an explanation you can construct for it afterwards.