For Heads-of · Practitioner
Mechanistic Interpretability
Research that tries to reverse-engineer what a neural network computes internally, identifying the features it represents and the circuits that connect them, in terms a person can understand. It is a research field, not yet an audit tool.
- interpretability
- sparse autoencoders
- circuits
- features
- neural networks
What it does
Instead of explaining a model from outside, by observing which inputs change its outputs, mechanistic interpretability looks at the internal activations. Techniques such as sparse autoencoders decompose those activations into features that often correspond to recognisable concepts, and circuit analysis traces how features combine to produce behaviour. Researchers have used these methods to find, and in some cases adjust, features related to deception, bias and other safety-relevant concepts inside large models.
Why it matters
Testing can only find behaviour someone thought to look for. If a model's internal mechanisms could be inspected, hidden behaviour such as a model backdoor or an intention to deceive might be found directly. That is the long-term promise; today the methods cover only part of what large models do.
How it relates to explainability
It is the inside-out complement to feature attribution and other explainability methods, and the research frontier of model interpretability. For a deploying organisation it does not yet change what evidence to collect; the register entry for explanations that fall short is inadequate explainability for affected individuals.
Sources
Related Terms
Explainability
The ability to give a person an account of why a system produced a particular output, in terms they can act on. Explainability is a property of the account and its audience, not of the model: an opaque model can be explainable, and an interpretable one can fail to be explained.
Feature Attribution
Methods that estimate how much each input contributed to a model's prediction, such as SHAP and LIME. They are standard in model validation and explanation, but they approximate the model rather than describe it, and they can be manipulated.
Model Interpretability
An intrinsic property of a model: the degree to which a person can understand the mechanism by which it turns inputs into outputs. A linear model or a shallow decision tree is interpretable by construction. A deep network is not, however good an explanation you can construct for it afterwards.
Neural Network
A computational model loosely inspired by biological neurons, made of interconnected layers of artificial nodes that learn to recognise patterns in data.