A property of the model, not of the output

The distinction that makes the term useful is that interpretability is inherent. It is not something added to a model; it is a fact about the model's structure. A model is interpretable when a competent person can trace the path from input to output and describe the mechanism — the coefficients, the splits, the rules.

That framing has a consequence organisations often miss. Interpretability is decided at model selection, not afterwards. Once a deep network is chosen, no amount of downstream tooling makes the model interpretable; it makes explanations available, which is a different thing. The choice between an interpretable model and a more accurate opaque one is a governance decision with a real trade-off, and it is usually made by whoever picked the architecture, without the trade-off being surfaced.

How it differs from explainability

Interpretability is about the mechanism; explainability is about the account.

An interpretable model can be understood directly. An opaque model can be explained — using post-hoc techniques that construct a plausible account of a particular decision — without ever becoming interpretable. The account may be useful, and it is an approximation of the model rather than a description of it.

Two consequences follow, and both matter for governance:

  • An explanation can be wrong in ways an interpretation cannot. A post-hoc method that attributes a decision to the wrong factors produces confident, plausible, unfaithful output. There is no equivalent failure mode when reading a decision tree.
  • Regulatory obligations generally attach to the account, not the mechanism. Which means an opaque model with a well-governed explanation pipeline can be compliant, and an interpretable model whose reasoning nobody ever conveys to the affected person can be non-compliant. The two properties are not ranked.

What makes a model interpretable

Broadly: linear and logistic regression with a manageable number of features; shallow decision trees; rule lists and scorecards; generalised additive models. The common property is not simplicity for its own sake but that the model's structure maps onto a human account of it.

Two cautions worth stating. Interpretable does not mean small enough to read. A linear model with two thousand features is interpretable in principle and opaque in practice. And interpretable does not mean correct — a model can be transparently, comprehensibly wrong, and its interpretability makes that easier to discover, which is precisely the argument for it.

Techniques that improve understanding of opaque models

Feature importance analysis, partial dependence plots, SHAP values, attention visualisation. These are genuinely useful for debugging, for detecting spurious correlations, and for building the account an explainability obligation requires.

They do not confer interpretability. Describing them as "making the model interpretable" is the single commonest error in this area, and it matters because it encourages a belief that the mechanism is understood when only an approximation of it is.

Where it becomes a governance question

Where a model's output materially affects a person, the organisation has to be able to say something defensible about why. Interpretability is one route; a governed explanation pipeline is the other. What is not defensible is choosing an opaque model, building no explanation capability, and discovering the gap when someone asks.

That gap is what inadequate explainability for affected individuals describes, and it is why the decision belongs on the record at model selection rather than at deployment.