Interpretable machine learning

Overview

Interpretable machine learning builds the explanation into the model. The model is a short rule list, a small table of points, an additive sum of one-feature functions, or a network whose units carry named concepts. A person can read the model directly instead of trusting an explanation of a black box after training 2103.11251 Introduction.

The survey behind this page argues two things. First, for high-stakes decisions, an inherently interpretable model should be tried first, and post-hoc explanations of a black box can mislead and can justify keeping a model that was never needed 2103.11251 Introduction. Second, interpretable models need not cost accuracy; the survey supports this with examples and cited experiments, not a proof 2103.11251 Introduction.

Eight cards cite this slug. The survey gives the agenda and ten open problems. Two cards compare an additive model with flexible baselines on the same tasks. The neural additive model card uses four tabular datasets, with author-reported numbers 2004.13912 tab. 1. The additive Gaussian process card compares against a squared-exponential GP on five regression and six classification datasets 1112.4394 tab. 2 1112.4394 tab. 4. The SHAP and LIME card shows that post-hoc feature rankings on tabular data change with the classifier and with correlated features 2305.02012 §2.1. Three more are cited for narrower points: a search for adversarial inputs, which also finds that single hidden units are no more meaningful than random directions 1312.6199 §3 1312.6199 §4.2; a probe that tries to find the signal inside a trained black box 2107.10356 §C3; and a linear model over a few ranked features in electronic health records 2605.04666 tab. 1. The eighth card is a health-care survey. It defines explainability, sorts explanation methods, and gives a decision guide for choosing between an interpretable model and a post-hoc explanation. It runs no experiments, and its authors say the guide still needs testing 2007.15911 §2.1 2007.15911 §6 2007.15911 §7.1. This page reports the numbers above but does not rank the approaches in general, because no card tests them all on one task. For the broader post-hoc side, see Explainable AI.

Variants

The survey's variants each follow one of its ten challenges. The cards link no reusable reference implementation for the survey's variants: the survey presents them as research directions, and the one repository linked from the probe card has no license file. The neural additive model paper released code under Apache-2.0, and the SHAP and LIME card's repositories have no license file. The additive Gaussian process code is GPL-2.0, so that card cites it by path only.

The health-care survey offers a second way to split the design space. Its first split is between explainable modelling, where the task model is itself readable, and post-hoc explanation, where an explanation is added to any model. Post-hoc methods are then sorted by type (model, attribution or example) and by scope (global or local) 2007.15911 §4 2007.15911 tab. 1. The model-based subsection below covers surrogate explanations; the attribution and example methods it lists are covered on Feature attribution and Explanation evaluation.

Model-based surrogate explanations

A simpler model is fitted to a complex one, either for the whole model (global) or around one prediction (local). The health-care survey lists tree extraction, BETA and symbolic metamodelling as global examples, and LIME and Anchors as local ones 2007.15911 §4.1 2007.15911 tab. 1. Its evaluation treats the surrogate's completeness as true by construction, since it can always compute the output, and measures soundness as how often the surrogate and the model agree on predictions 2007.15911 §5.1 [arxiv:2007.15911 tab. 2]. The local variant has no metric for clarity in that survey 2007.15911 tab. 2. Card: Explainable AI covers the wider post-hoc family; the health-care survey card is Role of explainability in health care.

Sparse logical models

Decision trees, decision lists and decision sets with a penalty on size. The survey notes that optimal tree search is NP-complete, which is why greedy methods are standard 2103.11251 §1. Its open problems are scaling optimal search, handling continuous variables without losing optimality, and enforcing global constraints such as monotone probabilities [arxiv:2103.11251 §1]. A quoted example reports optimal search on medium datasets within 10 minutes for state-of-the-art methods with sparsity constraints [arxiv:2103.11251 §1]. Card: Interpretable Machine Learning survey.

Scoring systems

Linear models with small integer coefficients, so a user can add points by hand. The survey groups solution methods into exact mixed-integer programming, linear-programming approximations, rounding (Sequential Rounding and Discrete Coordinate Descent are described), and expert-in-the-loop tools 2103.11251 §2. The example coefficient range of -10 to 10 is the paper's illustration, not a recommendation 2103.11251 eq. 3. Open problems are scaling exact methods and making the elicitation of constraints from experts easier 2103.11251 §2. Card: Interpretable Machine Learning survey.

Generalized additive models

A prediction written as a sum of functions, each of one feature 2103.11251 eq. 4 2103.11251 §3. Each function can be inspected on its own. Open problems are controlling sparsity, smoothness and monotonicity, and using these models to find data errors and misleading patterns in complex records 2103.11251 §3. Card: Interpretable Machine Learning survey.

Neural additive models

A learned variant of the same form. Each input column gets its own small neural network, the outputs are summed with one bias, and a link function gives the prediction 2004.13912 §2 2004.13912 eq. 1. Each network sees only its own column, so its curve can be plotted against that input 2004.13912 §2. The paper's hidden unit, ExU, passes an exponentially scaled, shifted input through a ReLU clipped at 1. The authors introduce it because standard ReLU networks trained with mini-batches fail to fit sharp jumps in a one-dimensional target 2004.13912 §2 2004.13912 eq. 2 2004.13912 fig. 2. The paper also describes a multitask version, where each task takes a learned weighted sum of subnets per feature 2004.13912 §4.2. On synthetic data built to favour multitask learning, its mean MSE over six tasks is 0.712, against 1.083 for single-task models 2004.13912 tab. 2 2004.13912 §4.2.1. On COMPAS, the single-task and multitask intervals overlap 2004.13912 tab. 3. Card: Neural Additive Models.

Additive Gaussian processes

A Gaussian process whose kernel is a sum over interaction orders. Order n sums products of one-dimensional base kernels over all n-subsets of inputs, and each order has its own variance σ_n² 1112.4394 §3.1 1112.4394 eq. 1 1112.4394 eq. 3. The top order is one product over all inputs; with squared-exponential base kernels it is the usual SE kernel [arxiv:1112.4394 eq. 4] 1112.4394 eq. 5. Unlike a GAM, the kernel can express interactions between inputs 1112.4394 §1. All orders are summed in O(D²) with Newton–Girard identities 1112.4394 §3.3 [arxiv:1112.4394 eq. 6]. The Gram matrix costs O(N²DR) for maximum order R [arxiv:1112.4394 §3.4].

The learned order variances show which interaction orders carry the fit. On pumadyn-8nh, order 8 holds 99.5% of the variance; on concrete, order 1 holds 70.6% and order 2 holds 13.3% 1112.4394 tab. 1. The authors present these shares as a way to see which orders matter 1112.4394 §1 1112.4394 §3.2. Read them as properties of the fitted model, not of the data alone. The paper's figure of low-order functions recovered on concrete is the closest thing to a readable form 1112.4394 fig. 2.

Against a squared-exponential GP, the additive model has lower mean squared error on concrete (0.089 against 0.157) and on servo (0.110 against 0.126). It is level with it on bach and pumadyn-8nh, and worse on housing (0.102 against 0.092) 1112.4394 tab. 2. These are means over 10 train-test splits, with paired t-tests 1112.4394 §5.3. Its interpretability rests on the additive structure and the order variances, not on a rule list or a table of points. Card: Additive Gaussian Processes.

Case-based reasoning with prototypes

Nearest-neighbour and prototype methods. Deep prototype networks compare image patches with learned prototypes, so a prediction points to the prototypes behind it 2103.11251 fig. 6 2103.11251 fig. 7. Open problems are extending the approach to video, adding prior knowledge or expert supervision to learning prototypes, and repairing invalid prototypes 2103.11251 §4. Card: Interpretable Machine Learning survey.

Complete supervised disentanglement

Each human-defined concept is aligned with its own neuron, expressed as a constraint 2103.11251 eq. 5 2103.11251 §5. Open problems are making a whole layer interpretable, disentangling all layers at once, choosing which concepts to learn, and making the mapping from concepts to output readable 2103.11251 §5. Card: Interpretable Machine Learning survey.

Unsupervised disentanglement

Generative models and compositional architectures, such as capsules and slot attention, learn parts without given concept labels 2103.11251 fig. 10 2103.11251 fig. 11. The survey names the lack of quantitative evaluation as the first open problem, followed by transfer of compositional architectures to other domains and part-whole disentanglement on large datasets 2103.11251 §6. Card: Interpretable Machine Learning survey.

Dimension reduction for visualisation

Global methods (PCA, MDS) are contrasted with local methods (t-SNE, UMAP, PaCMAP) 2103.11251 fig. 12 2103.11251 fig. 13 [arxiv:2103.11251 fig. 14]. The survey treats capturing global structure without losing local detail as open 2103.11251 §7. It also says the hyperparameters are numerous and hard to tune without labels 2103.11251 §7, and that the mapping itself needs explaining 2103.11251 §7. Card: Interpretable Machine Learning survey.

Physics and causal constraints

Physics-informed neural networks minimise the residual of a PDE together with its initial and boundary conditions 2103.11251 fig. 15. Open problems are training, combining the approach with experimental design, and extending the constraints to non-neural models 2103.11251 §8. Card: Interpretable Machine Learning survey.

Rashomon sets

The Rashomon set is the collection of models whose loss is near the optimum. The survey lists characterising the set, choosing a measure of its size, computing that measure, visualising it, and choosing a model from it as open problems 2103.11251 §9. Two candidate size measures, the Rashomon ratio and the pattern Rashomon ratio, give different values on the toy decision-stump example (0.4 and 3/11) 2103.11251 fig. 17. Card: Interpretable Machine Learning survey.

Interpretable reinforcement learning

Policies written as decision trees or rule lists, over symbolic or relational state spaces 2103.11251 §10. Open problems are which constraints give an accurate interpretable policy, under which assumptions one exists, and how to simplify the state space 2103.11251 §10. Challenge 10.1 has a title but no body text in the card, so treat it as an open question 2103.11251 §10. Card: Interpretable Machine Learning survey.

Few-feature linear models

A linear SVM trained on the top 1, 3 or 30 temporal features of electronic health records. Features are ranked by their single-feature AUC on the training patients, and no joint search is run 2605.04666 p. 3. For 10 lab orders the held-out AUC is 74.15–87.46% with the top feature and 81.89–87.73% with the top 30 2605.04666 tab. 1. For 10 medication decisions it is 69.62–87.65% and 80.53–89.75% 2605.04666 tab. 2. The authors say the greedy ranking may not be the best choice with over 9,000 candidates 2605.04666 p. 5. The model is readable because it is linear over a few named features; the paper reports no comparison with a black-box model 2605.04666 p. 3. Card: Feature importance analysis for patient management decisions.

Post-hoc rankings on tabular data (SHAP and LIME)

Not an interpretable model. SHAP treats each feature as a player and attributes the model output over feature coalitions. The paper says a SHAP score does not measure a feature's weight; the ranking carries the meaning 2305.02012 §2.1. SHAP assumes features are independent, and correlated features can make its scores misleading 2305.02012 §2.1. LIME fits a local linear model around one instance, so its weights hold for that instance only 2305.02012 §2.2 2305.02012 §4.

On a UK Biobank cohort with four classifiers and ten features, the top three SHAP features agreed across models and the other seven varied 2305.02012 §2.1 2305.02012 fig. 2. On an airline passenger-satisfaction dataset with 22 features and 500 subjects, the paper reports a stability value, NMR, of 0.231 for LightGBM, 0.275 for logistic regression, 0.273 for SVC and 0.445 for a decision tree. Lower means more stable 2305.02012 §3. The classifiers used default parameters, and the paper says accuracy alone does not show that the most accurate model gives the better explanation 2305.02012 §3. Its MIP re-ordering moved "Ease of Online booking" from 15th in the SHAP list to 5th for LightGBM 2305.02012 tab. 3 2305.02012 §3.

NMR scores how far the ranking moves as the top features are removed in turn. MIP re-orders features by their rank across those rounds 2305.02012 §2.1. The paper gives no formula for either measure 2305.02012 §2.1. Each result comes from one dataset and one split, with no intervals 2305.02012 §2.1 2305.02012 §3. Card: A Perspective on Explainable AI Methods: SHAP and LIME.

Probing a black box after training

Not an interpretable model. The probe card trains standard CNNs to predict self-reported race from medical images, then probes them with frequency filters, resolution changes, Grad-CAM saliency maps, occlusion and patch removal 2107.10356 §C1–§C4. Occluding the most salient region lowers held-out AUC for Black and White patients from 0.94 to 0.79 2107.10356 §C3. Using only the lung region lowers it to 0.68, 0.74 and 0.73 for Asian, Black and White patients, against 0.93, 0.94 and 0.94 for the full image 2107.10356 §C3. No single frequency band, region or patch removes the signal 2107.10356 §C4, and the authors state that they could not isolate the features responsible 2107.10356 §Discussion. Five radiologists judged the saliency maps qualitatively, and the authors found no consistent anatomical location 2107.10356 §C3. Read this as a warning about post-hoc probes: a map is a hypothesis to test, and the paper does not show that a readable model would have avoided the problem. Card: Reading Race.

Single units against random directions

Not an interpretable model either. The paper ranks held-out images by their activation along one unit's basis vector and along random directions, then compares the top images by eye 1312.6199 §3. Its MNIST and AlexNet figures show that both kinds of direction share semantic properties 1312.6199 fig. 1–4. The check is qualitative: no metric is reported, and the random directions are not sampled in a stated way [arxiv:1312.6199 §3]. Use it as a reason to test a unit's meaning against random directions before naming it. Card: Intriguing properties of neural networks.

When to use which

The survey does not give a decision procedure. The rows below follow its constraints and state only the evidence the cards report. Rows marked open point at research problems, not at ready methods.

If a task needs a black-box model for accuracy and a post-hoc explanation for auditing, the cards for Explainable AI and Feature attribution cover that route. The SHAP and LIME card adds the stability warning for tabular rankings.

Implementation sketch

The survey's generic form balances a loss against an interpretability penalty, under a constraint. Each variant fixes the loss, the penalty and the model class. The paper leaves the penalty and constraint as placeholders 2103.11251 eq. 1.

# Generic interpretable supervised learning [arxiv:2103.11251 eq. 1]
# Input:  data z_i = (x_i, y_i); model class F; trade-off constant C
# Output: model f in F that meets the interpretability constraint

def fit_interpretable_model(data, model_class, C):
    # Average loss over the training data [arxiv:2103.11251 eq. 1]
    # Loss is chosen per task; the paper leaves it open
    # Penalty and constraint are domain-specific placeholders
    # Minimise loss + C * penalty over model_class, subject to constraint
    return best_model

Contradictions

No direct conflict between results on the same task. Several tensions:

• Accuracy cost. The survey argues that interpretable models need not cost accuracy 2103.11251 Introduction. The neural additive model tables are consistent with a small cost on these four tabular datasets, not zero: the model trails the best black box on each, by a little on Credit and more on the regression tasks 2004.13912 tab. 1. The additive Gaussian process is mixed against a flexible GP baseline: it beats the squared-exponential GP on concrete and servo, but not on housing 1112.4394 tab. 2. The survey's claim rests on examples and cited experiments 2103.11251 Introduction, so the two do not test each other head-to-head.
• Metric wording. The neural additive model text says the classification metric is area under the precision-recall curve, but its tables and code use ROC AUC 2004.13912 §3 2004.13912 tab. 1. Confirm the metric before comparing numbers.
• Post-hoc explanations. The survey warns that post-hoc explanations of a black box can mislead 2103.11251 Introduction. The SHAP and LIME study points the same way, since its tabular rankings change with the model 2305.02012 §2.1 2305.02012 §3. The health-care survey makes the same point: post-hoc explanations are approximations that can be plausible and still wrong 2007.15911 §6.1. None of these papers tests whether a black box was needed in the first place.
• Method labels. The health-care survey files LIME and Anchors under model-based (surrogate) explanations, while other surveys file them under attribution 2007.15911 §4.1. The SHAP and LIME study describes LIME as a local linear model around one instance 2305.02012 §2.2. The two papers agree on what LIME computes; they differ on the category, so a comparison that counts categories can differ between sources.
• Single neurons. The unit-analysis result 1312.6199 §3 sits uneasily with the survey's disentanglement variant, which aligns each human-defined concept with its own neuron 2103.11251 §5. The 1312.6199 analysis uses ordinary networks, not disentangled ones, and reports no measure of how aligned units are. So it does not contradict the survey's claim, but it shows that a neuron-aligned basis is not the default.
• Rashomon measures. Inside the survey, the Rashomon section says the two size measures disagree and it is unclear which to use 2103.11251 fig. 17 2103.11251 §9.

The probe card 2107.10356 §C and the EHR study 2605.04666 p. 3 report no comparison with other methods, so they add no ranking.

Papers

• Additive Gaussian Processes: a GP whose kernel sums products of one-dimensional kernels over all interaction orders, with one variance per order, evaluated in O(D²) with Newton–Girard identities.
• Intriguing properties of neural networks: finds small, optimised input changes that flip labels, and reports that single hidden units are no more meaningful than random directions.
• Neural Additive Models: Interpretable Machine Learning with Neural Nets: one small neural network per input feature, summed with a bias; compared with EBMs, XGBoost and a 10-layer DNN on four tabular datasets.
• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges: survey and position paper with five principles and ten open technical challenges.
• Reading Race: AI Recognises Patient's Racial Identity In Medical Images: trains CNNs to predict self-reported race from medical images and probes them with post-hoc tests that do not find a single cause.
• A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME: shows that SHAP and LIME feature rankings on tabular data change with the classifier and with collinear features, and proposes NMR and MIP to test and adjust them.
• Feature importance analysis for patient management decisions: ranks temporal EHR features by single-feature AUC and fits linear SVMs on the top 1, 3 and 30.
• The role of explainability in creating trustworthy artificial intelligence for health care: narrative survey that defines explainability as interpretability plus fidelity, sorts post-hoc methods by type and scope, lists which properties have metrics, and gives a decision guide; no experiments.