Explainability and Interpretability
Being able to say why a model produced an output is what lets humans check it, debug it, and decide whether to act on it.
Two Related Goals
Interpretability is the degree to which a human can understand a model's inner workings directly. Explainability is the ability to give an account of why a particular output was produced, even for a model too complex to grasp whole. Both serve the same end: letting people scrutinize a decision rather than accept it blindly.
Why It Matters
- Debugging: an explanation reveals when a model is right for the wrong reason.
- Trust: users can check whether the reasoning matches domain knowledge.
- Accountability: a decision that affects people should be explicable to them.
- Discovery: an interpretable model can teach us something about the phenomenon.
The Accuracy-Interpretability Tension
Simple models, linear relationships, short decision rules, are easy to interpret but may fit complex phenomena poorly. Highly flexible models often fit better but resist understanding. This tension is real but not absolute; sometimes an interpretable model performs just as well, and reaching for an opaque one is a choice that should be justified, not a default.
The Limits of Post-Hoc Explanation
Explanations generated after the fact for an opaque model are approximations of its behavior, not its true reasoning, and can be misleading, plausible stories that do not reflect what the model actually did. An explanation that sounds convincing is not thereby correct. Where the stakes are high, an inherently interpretable model can be worth more than an opaque one with an attached explanation.
In Scientific Modeling
A mechanistic physics model is interpretable by construction: its terms correspond to physical processes, so a result can be traced to the physics that produced it. When faster data-driven surrogates are used, keeping them tied to and checked against the interpretable physics is what preserves the ability to understand, and to trust, what they produce.