---
kind: concept
slug: interpretable-machine-learning
title: Interpretable machine learning
summary: >-
  Models whose structure a person can read: sparse trees, scoring systems, neural and additive Gaussian processes, prototype networks and few-feature linear models. A survey urges trying them first in high-stakes decisions. A tabular study finds neural additive models trail the best black box by a small margin, and SHAP and LIME rankings move with the model. Results are author-reported.
tags: [interpretability, survey, decision-trees, scoring-systems, generalized-additive-models, neural-additive-models, additive-gaussian-processes, rashomon-set, post-hoc-explanation, feature-selection]
papers: ["1112.4394", "1312.6199", "2004.13912", "2007.15911", "2103.11251", "2107.10356", "2305.02012", "2605.04666"]
related: [explainable-ai, explanation-evaluation, disentangled-representation, prototype-learning, feature-attribution, counterfactual-explanation, mechanistic-interpretability, feature-selection, support-vector-machine, gradient-boosted-decision-trees, multi-task-learning, algorithmic-fairness, adversarial-robustness, gaussian-process, additive-kernel]
status: draft
updated_at: 2026-11-01
---

## Overview

Interpretable machine learning builds the explanation into the model. The
model is a short rule list, a small table of points, an additive sum of
one-feature functions, or a network whose units carry named concepts. A
person can read the model directly instead of trusting an explanation of a
black box after training [arxiv:2103.11251 Introduction].

The survey behind this page argues two things. First, for high-stakes
decisions, an inherently interpretable model should be tried first, and
post-hoc explanations of a black box can mislead and can justify keeping a
model that was never needed [arxiv:2103.11251 Introduction]. Second,
interpretable models need not cost accuracy; the survey supports this with
examples and cited experiments, not a proof [arxiv:2103.11251 Introduction].

Eight cards cite this slug. The survey gives the agenda and ten open problems.
Two cards compare an additive model with flexible baselines on the same tasks.
The neural additive model card uses four tabular datasets, with author-reported
numbers [arxiv:2004.13912 tab. 1]. The additive Gaussian process card compares
against a squared-exponential GP on five regression and six classification
datasets [arxiv:1112.4394 tab. 2] [arxiv:1112.4394 tab. 4]. The SHAP and LIME card
shows that post-hoc feature rankings on tabular data change with the
classifier and with correlated features [arxiv:2305.02012 §2.1]. Three more
are cited for narrower points: a search for adversarial inputs, which
also finds that single hidden units are no more meaningful than random
directions [arxiv:1312.6199 §3] [arxiv:1312.6199 §4.2]; a probe that tries to
find the signal inside a trained black box [arxiv:2107.10356 §C3]; and a
linear model over a few ranked features in electronic health records
[arxiv:2605.04666 tab. 1]. The eighth card is a health-care survey. It
defines explainability, sorts explanation methods, and gives a decision guide
for choosing between an interpretable model and a post-hoc explanation. It
runs no experiments, and its authors say the guide still needs testing
[arxiv:2007.15911 §2.1] [arxiv:2007.15911 §6] [arxiv:2007.15911 §7.1]. This
page reports the numbers above but does not rank the approaches in general,
because no card tests them all on one task. For the broader post-hoc side, see
[Explainable AI](/concepts/explainable-ai).

## Variants

The survey's variants each follow one of its ten challenges. The cards link
no reusable reference implementation for the survey's variants: the survey
presents them as research directions, and the one repository linked from the
probe card has no license file. The neural additive model paper released
code under Apache-2.0, and the SHAP and LIME card's repositories have no
license file. The additive Gaussian process code is GPL-2.0, so that card cites
it by path only.

The health-care survey offers a second way to split the design space. Its
first split is between explainable modelling, where the task model is itself
readable, and post-hoc explanation, where an explanation is added to any model.
Post-hoc methods are then sorted by type (model, attribution or example) and
by scope (global or local) [arxiv:2007.15911 §4] [arxiv:2007.15911 tab. 1].
The model-based subsection below covers surrogate explanations; the attribution
and example methods it lists are covered on
[Feature attribution](/concepts/feature-attribution) and
[Explanation evaluation](/concepts/explanation-evaluation).

### Model-based surrogate explanations

A simpler model is fitted to a complex one, either for the whole model (global)
or around one prediction (local). The health-care survey lists tree extraction,
BETA and symbolic metamodelling as global examples, and LIME and Anchors as
local ones [arxiv:2007.15911 §4.1] [arxiv:2007.15911 tab. 1]. Its evaluation
treats the surrogate's completeness as true by construction, since it can
always compute the output, and measures soundness as how often the surrogate
and the model agree on predictions [arxiv:2007.15911 §5.1] [arxiv:2007.15911
tab. 2]. The local variant has no metric for clarity in that survey
[arxiv:2007.15911 tab. 2]. Card:
[Explainable AI](/concepts/explainable-ai) covers the wider post-hoc family;
the health-care survey card is [Role of explainability in health care](/papers/2007.15911).

### Sparse logical models

Decision trees, decision lists and decision sets with a penalty on size. The
survey notes that optimal tree search is NP-complete, which is why greedy
methods are standard [arxiv:2103.11251 §1]. Its open problems are scaling
optimal search, handling continuous variables without losing optimality, and
enforcing global constraints such as monotone probabilities [arxiv:2103.11251
§1]. A quoted example reports optimal search on medium datasets within 10
minutes for state-of-the-art methods with sparsity constraints [arxiv:2103.11251
§1]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Scoring systems

Linear models with small integer coefficients, so a user can add points by
hand. The survey groups solution methods into exact mixed-integer programming,
linear-programming approximations, rounding (Sequential Rounding and Discrete
Coordinate Descent are described), and expert-in-the-loop tools
[arxiv:2103.11251 §2]. The example coefficient range of -10 to 10 is the
paper's illustration, not a recommendation [arxiv:2103.11251 eq. 3]. Open
problems are scaling exact methods and making the elicitation of constraints
from experts easier [arxiv:2103.11251 §2]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Generalized additive models

A prediction written as a sum of functions, each of one feature
[arxiv:2103.11251 eq. 4] [arxiv:2103.11251 §3]. Each function can be
inspected on its own. Open problems are controlling sparsity, smoothness and
monotonicity, and using these models to find data errors and misleading
patterns in complex records [arxiv:2103.11251 §3]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

#### Neural additive models

A learned variant of the same form. Each input column gets its own small
neural network, the outputs are summed with one bias, and a link function
gives the prediction [arxiv:2004.13912 §2] [arxiv:2004.13912 eq. 1]. Each
network sees only its own column, so its curve can be plotted against that
input [arxiv:2004.13912 §2]. The paper's hidden unit, ExU, passes an
exponentially scaled, shifted input through a ReLU clipped at 1. The authors
introduce it because standard ReLU networks trained with mini-batches fail to
fit sharp jumps in a one-dimensional target [arxiv:2004.13912 §2]
[arxiv:2004.13912 eq. 2] [arxiv:2004.13912 fig. 2]. The paper also describes a
multitask version, where each task takes a learned weighted sum of subnets per
feature [arxiv:2004.13912 §4.2]. On synthetic data built to favour multitask
learning, its mean MSE over six tasks is 0.712, against 1.083 for single-task
models [arxiv:2004.13912 tab. 2] [arxiv:2004.13912 §4.2.1]. On COMPAS, the
single-task and multitask intervals overlap [arxiv:2004.13912 tab. 3]. Card:
[Neural Additive Models](/papers/2004.13912).

### Additive Gaussian processes

A Gaussian process whose kernel is a sum over interaction orders. Order n sums
products of one-dimensional base kernels over all n-subsets of inputs, and each
order has its own variance σ_n² [arxiv:1112.4394 §3.1] [arxiv:1112.4394 eq. 1]
[arxiv:1112.4394 eq. 3]. The top order is one product over all inputs; with
squared-exponential base kernels it is the usual SE kernel [arxiv:1112.4394
eq. 4] [arxiv:1112.4394 eq. 5]. Unlike a GAM, the kernel can express
interactions between inputs [arxiv:1112.4394 §1]. All orders are summed in
O(D²) with Newton–Girard identities [arxiv:1112.4394 §3.3] [arxiv:1112.4394
eq. 6]. The Gram matrix costs O(N²DR) for maximum order R [arxiv:1112.4394
§3.4].

The learned order variances show which interaction orders carry the fit. On
pumadyn-8nh, order 8 holds 99.5% of the variance; on concrete, order 1 holds
70.6% and order 2 holds 13.3% [arxiv:1112.4394 tab. 1]. The authors present
these shares as a way to see which orders matter [arxiv:1112.4394 §1]
[arxiv:1112.4394 §3.2]. Read them as properties of the fitted model, not of
the data alone. The paper's figure of low-order functions recovered on
concrete is the closest thing to a readable form [arxiv:1112.4394 fig. 2].

Against a squared-exponential GP, the additive model has lower mean squared
error on concrete (0.089 against 0.157) and on servo (0.110 against 0.126). It
is level with it on bach and pumadyn-8nh, and worse on housing (0.102 against
0.092) [arxiv:1112.4394 tab. 2]. These are means over 10 train-test splits, with
paired t-tests [arxiv:1112.4394 §5.3]. Its interpretability rests on the
additive structure and the order variances, not on a rule list or a table of
points. Card: [Additive Gaussian Processes](/papers/1112.4394).

### Case-based reasoning with prototypes

Nearest-neighbour and prototype methods. Deep prototype networks compare image
patches with learned prototypes, so a prediction points to the prototypes
behind it [arxiv:2103.11251 fig. 6] [arxiv:2103.11251 fig. 7]. Open problems
are extending the approach to video, adding prior knowledge or expert
supervision to learning prototypes, and repairing invalid prototypes
[arxiv:2103.11251 §4]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Complete supervised disentanglement

Each human-defined concept is aligned with its own neuron, expressed as a
constraint [arxiv:2103.11251 eq. 5] [arxiv:2103.11251 §5]. Open problems are
making a whole layer interpretable, disentangling all layers at once,
choosing which concepts to learn, and making the mapping from concepts to
output readable [arxiv:2103.11251 §5]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Unsupervised disentanglement

Generative models and compositional architectures, such as capsules and slot
attention, learn parts without given concept labels [arxiv:2103.11251 fig. 10]
[arxiv:2103.11251 fig. 11]. The survey names the lack of quantitative
evaluation as the first open problem, followed by transfer of compositional
architectures to other domains and part-whole disentanglement on large
datasets [arxiv:2103.11251 §6]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Dimension reduction for visualisation

Global methods (PCA, MDS) are contrasted with local methods (t-SNE, UMAP,
PaCMAP) [arxiv:2103.11251 fig. 12] [arxiv:2103.11251 fig. 13] [arxiv:2103.11251
fig. 14]. The survey treats capturing global structure without losing local
detail as open [arxiv:2103.11251 §7]. It also says the hyperparameters are
numerous and hard to tune without labels [arxiv:2103.11251 §7], and that the
mapping itself needs explaining [arxiv:2103.11251 §7]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Physics and causal constraints

Physics-informed neural networks minimise the residual of a PDE together with
its initial and boundary conditions [arxiv:2103.11251 fig. 15]. Open problems
are training, combining the approach with experimental design, and extending
the constraints to non-neural models [arxiv:2103.11251 §8]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Rashomon sets

The Rashomon set is the collection of models whose loss is near the optimum.
The survey lists characterising the set, choosing a measure of its size,
computing that measure, visualising it, and choosing a model from it as open
problems [arxiv:2103.11251 §9]. Two candidate size measures, the Rashomon
ratio and the pattern Rashomon ratio, give different values on the toy
decision-stump example (0.4 and 3/11) [arxiv:2103.11251 fig. 17]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Interpretable reinforcement learning

Policies written as decision trees or rule lists, over symbolic or relational
state spaces [arxiv:2103.11251 §10]. Open problems are which constraints give an
accurate interpretable policy, under which assumptions one exists, and how to
simplify the state space [arxiv:2103.11251 §10]. Challenge 10.1 has a title but
no body text in the card, so treat it as an open question
[arxiv:2103.11251 §10]. Card: [Interpretable Machine Learning survey](/papers/2103.11251).

### Few-feature linear models

A linear SVM trained on the top 1, 3 or 30 temporal features of electronic
health records. Features are ranked by their single-feature AUC on the
training patients, and no joint search is run [arxiv:2605.04666 p. 3]. For
10 lab orders the held-out AUC is 74.15–87.46% with the top feature and
81.89–87.73% with the top 30 [arxiv:2605.04666 tab. 1]. For 10 medication
decisions it is 69.62–87.65% and 80.53–89.75% [arxiv:2605.04666 tab. 2]. The
authors say the greedy ranking may not be the best choice with over 9,000
candidates [arxiv:2605.04666 p. 5]. The model is readable because it is
linear over a few named features; the paper reports no comparison with a
black-box model [arxiv:2605.04666 p. 3]. Card: [Feature importance analysis for patient management decisions](/papers/2605.04666).

### Post-hoc rankings on tabular data (SHAP and LIME)

Not an interpretable model. SHAP treats each feature as a player and
attributes the model output over feature coalitions. The paper says a SHAP
score does not measure a feature's weight; the ranking carries the meaning
[arxiv:2305.02012 §2.1]. SHAP assumes features are independent, and correlated
features can make its scores misleading [arxiv:2305.02012 §2.1]. LIME fits a
local linear model around one instance, so its weights hold for that instance
only [arxiv:2305.02012 §2.2] [arxiv:2305.02012 §4].

On a UK Biobank cohort with four classifiers and ten features, the top three
SHAP features agreed across models and the other seven varied
[arxiv:2305.02012 §2.1] [arxiv:2305.02012 fig. 2]. On an airline
passenger-satisfaction dataset with 22 features and 500 subjects, the paper
reports a stability value, NMR, of 0.231 for LightGBM, 0.275 for logistic
regression, 0.273 for SVC and 0.445 for a decision tree. Lower means more
stable [arxiv:2305.02012 §3]. The classifiers used default parameters, and the
paper says accuracy alone does not show that the most accurate model gives the
better explanation [arxiv:2305.02012 §3]. Its MIP re-ordering moved "Ease of
Online booking" from 15th in the SHAP list to 5th for LightGBM
[arxiv:2305.02012 tab. 3] [arxiv:2305.02012 §3].

NMR scores how far the ranking moves as the top features are removed in turn.
MIP re-orders features by their rank across those rounds
[arxiv:2305.02012 §2.1]. The paper gives no formula for either measure
[arxiv:2305.02012 §2.1]. Each result comes from one dataset and one split, with
no intervals [arxiv:2305.02012 §2.1] [arxiv:2305.02012 §3]. Card:
[A Perspective on Explainable AI Methods: SHAP and LIME](/papers/2305.02012).

### Probing a black box after training

Not an interpretable model. The probe card trains standard CNNs to predict
self-reported race from medical images, then probes them with frequency
filters, resolution changes, Grad-CAM saliency maps, occlusion and patch
removal [arxiv:2107.10356 §C1–§C4]. Occluding the most salient region lowers
held-out AUC for Black and White patients from 0.94 to 0.79
[arxiv:2107.10356 §C3]. Using only the lung region lowers it to 0.68, 0.74
and 0.73 for Asian, Black and White patients, against 0.93, 0.94 and 0.94 for
the full image [arxiv:2107.10356 §C3]. No single frequency band, region or
patch removes the signal [arxiv:2107.10356 §C4], and the authors state that
they could not isolate the features responsible [arxiv:2107.10356 §Discussion].
Five radiologists judged the saliency maps qualitatively, and the authors
found no consistent anatomical location [arxiv:2107.10356 §C3]. Read this as a
warning about post-hoc probes: a map is a hypothesis to test, and the paper
does not show that a readable model would have avoided the problem. Card:
[Reading Race](/papers/2107.10356).

### Single units against random directions

Not an interpretable model either. The paper ranks held-out images by their
activation along one unit's basis vector and along random directions, then
compares the top images by eye [arxiv:1312.6199 §3]. Its MNIST and AlexNet
figures show that both kinds of direction share semantic properties
[arxiv:1312.6199 fig. 1–4]. The check is qualitative: no metric is reported,
and the random directions are not sampled in a stated way [arxiv:1312.6199
§3]. Use it as a reason to test a unit's meaning against random directions
before naming it. Card: [Intriguing properties of neural networks](/papers/1312.6199).

## When to use which

The survey does not give a decision procedure. The rows below follow its
constraints and state only the evidence the cards report. Rows marked open
point at research problems, not at ready methods.

| Situation | Use | Why |
|---|---|---|
| High-stakes tabular decision that must be audited | Sparse trees, lists or sets (§1) | The survey argues for trying these first [arxiv:2103.11251 Introduction]. Its one worked comparison is a single benchmark dataset: an optimal tree with 9 leaves reached 73.15% test accuracy against 69.44% for CART with 16 leaves [arxiv:2103.11251 fig. 3]. |
| Tabular decision that needs per-feature curves and near-boosted-tree accuracy | Neural additive models (§GAM) | On MIMIC-II, AUC 0.830 against 0.844 for XGBoost and 0.835 for EBMs; logistic regression 0.791 [arxiv:2004.13912 tab. 1]. Behind the best black box on all four datasets: by 0.001 AUC on Credit and 0.145 RMSE on FICO [arxiv:2004.13912 tab. 1]. Author-reported. |
| Small or medium tabular regression where a GP fits and per-order structure matters | Additive Gaussian processes (§ above) | Lowest regression MSE of the five models on concrete and servo [arxiv:1112.4394 tab. 2]. Its order shares show where the fit's structure sits [arxiv:1112.4394 tab. 1]. Not best on housing [arxiv:1112.4394 tab. 2], and the paper's classification results are mixed [arxiv:1112.4394 tab. 4]. |
| A checklist a person can score by hand | Scoring systems (§2) | Integer coefficients [arxiv:2103.11251 §2]. Expert-in-the-loop tools exist for constraint elicitation [arxiv:2103.11251 §2]. |
| Effects of each feature should be inspectable | Generalized additive models (§3) | Each feature gets its own function [arxiv:2103.11251 §3]. Sparsity and monotonicity controls are open [arxiv:2103.11251 §3]. |
| Similar-example reasoning is acceptable for images | Prototype networks (§4) | Predictions point to learned prototypes [arxiv:2103.11251 fig. 6]. Prototype validity is an open problem [arxiv:2103.11251 §4]. |
| Named concept units inside a network | Supervised disentanglement (§5) | Requires concept labels [arxiv:2103.11251 §5]. Whole-network disentanglement is open [arxiv:2103.11251 §5]. |
| No concept labels available | Unsupervised disentanglement (§6) | Quantitative evaluation is an open problem [arxiv:2103.11251 §6]. |
| Looking at high-dimensional data by eye | Global (PCA, MDS) or local (t-SNE, UMAP, PaCMAP) (§7) | The choice is a trade-off the survey describes [arxiv:2103.11251 §7]. Tuning is hard without labels [arxiv:2103.11251 §7]. |
| A PDE-governed physical system | Physics constraints (§8) | Residuals of the PDE and its conditions are minimised [arxiv:2103.11251 fig. 15]. Training is open [arxiv:2103.11251 §8]. |
| Several models fit about equally well and one must be picked | Rashomon-set analysis (§9) | The survey notes the size measures can disagree [arxiv:2103.11251 fig. 17]. |
| A policy that must be read by a person | Interpretable RL (§10) | Trees or rule lists as policies [arxiv:2103.11251 §10]. Accuracy conditions are open [arxiv:2103.11251 §10]. |
| A linear model over a few ranked features for clinical prediction | Few-feature linear models | Top-30 linear SVM reached 80.5–89.8% AUC on 20 decisions, and a single top feature reached 69.6% for one drug [arxiv:2605.04666 tab. 1] [tab. 2]. Ranking is univariate and greedy [arxiv:2605.04666 p. 5]. Validate against a black box on your own data. |
| Explanation is very important to the decision, as in health care | Explainable modelling, per the health-care survey's guide | At this level of importance the guide returns an interpretable model [arxiv:2007.15911 §6.1] [arxiv:2007.15911 fig. 2]. The authors say the guide is untested [arxiv:2007.15911 §7.1]. |
| Explanation is needed, a complex model clearly beats an interpretable one, and fidelity matters most | Model-based surrogate, global or local (see the surrogate section above) | Surrogates are complete by construction, and their agreement with the model can be measured [arxiv:2007.15911 §5.1] [arxiv:2007.15911 tab. 2]. The survey notes that a surrogate is still an approximation [arxiv:2007.15911 §6.1]. |
| You must rank features of a black-box tabular model | Post-hoc SHAP or LIME, compared across several models | Rankings moved with the classifier, and only the top three features agreed in one case [arxiv:2305.02012 §2.1]. The paper recommends comparing several models and stating the dependence [arxiv:2305.02012 §4]. LIME describes one subject only [arxiv:2305.02012 §2.2]. |
| You must find what a trained black box relies on | Post-hoc probes, see [Explainable AI](/concepts/explainable-ai) | Occlusion and lung masking changed AUC, but no single region carried the signal [arxiv:2107.10356 §C3]. Treat a saliency map as a hypothesis to test. |
| You want to name a hidden unit as a concept | Compare the unit with random directions first | The unit's top images were judged alike to random-direction top images, by eye [arxiv:1312.6199 §3] [arxiv:1312.6199 fig. 1–4]. No metric exists in the paper. |
| The model must hold up under small input changes | Adversarial-robustness methods, see [Adversarial robustness](/concepts/adversarial-robustness) | Optimised small changes flipped labels for every MNIST and AlexNet sample tested, and many transferred between models [arxiv:1312.6199 §4.2]. None of the cards here tests a readable model for this. |

If a task needs a black-box model for accuracy and a post-hoc explanation
for auditing, the cards for [Explainable AI](/concepts/explainable-ai) and
[Feature attribution](/concepts/feature-attribution) cover that route. The
SHAP and LIME card adds the stability warning for tabular rankings.

## Implementation sketch

The survey's generic form balances a loss against an interpretability
penalty, under a constraint. Each variant fixes the loss, the penalty and the
model class. The paper leaves the penalty and constraint as placeholders
[arxiv:2103.11251 eq. 1].

```pseudocode
# Generic interpretable supervised learning [arxiv:2103.11251 eq. 1]
# Input:  data z_i = (x_i, y_i); model class F; trade-off constant C
# Output: model f in F that meets the interpretability constraint

def fit_interpretable_model(data, model_class, C):
    # Average loss over the training data [arxiv:2103.11251 eq. 1]
    # Loss is chosen per task; the paper leaves it open
    # Penalty and constraint are domain-specific placeholders
    # Minimise loss + C * penalty over model_class, subject to constraint
    return best_model
```

## Contradictions

No direct conflict between results on the same task. Several tensions:

- **Accuracy cost.** The survey argues that interpretable models need not cost
  accuracy [arxiv:2103.11251 Introduction]. The neural additive model tables
  are consistent with a small cost on these four tabular datasets, not zero:
  the model trails the best black box on each, by a little on Credit and more
  on the regression tasks [arxiv:2004.13912 tab. 1]. The additive Gaussian
  process is mixed against a flexible GP baseline: it beats the
  squared-exponential GP on concrete and servo, but not on housing
  [arxiv:1112.4394 tab. 2]. The survey's claim rests
  on examples and cited experiments [arxiv:2103.11251 Introduction], so the
  two do not test each other head-to-head.
- **Metric wording.** The neural additive model text says the classification
  metric is area under the precision-recall curve, but its tables and code use
  ROC AUC [arxiv:2004.13912 §3] [arxiv:2004.13912 tab. 1]. Confirm the metric
  before comparing numbers.
- **Post-hoc explanations.** The survey warns that post-hoc explanations of a
  black box can mislead [arxiv:2103.11251 Introduction]. The SHAP and LIME
  study points the same way, since its tabular rankings change with the model
  [arxiv:2305.02012 §2.1] [arxiv:2305.02012 §3]. The health-care survey makes
  the same point: post-hoc explanations are approximations that can be
  plausible and still wrong [arxiv:2007.15911 §6.1]. None of these papers
  tests whether a black box was needed in the first place.
- **Method labels.** The health-care survey files LIME and Anchors under
  model-based (surrogate) explanations, while other surveys file them under
  attribution [arxiv:2007.15911 §4.1]. The SHAP and
  LIME study describes LIME as a local linear model around one instance
  [arxiv:2305.02012 §2.2]. The two papers agree on what LIME computes; they
  differ on the category, so a comparison that counts categories can differ
  between sources.
- **Single neurons.** The unit-analysis result [arxiv:1312.6199 §3] sits
  uneasily with the survey's disentanglement variant, which aligns each
  human-defined concept with its own neuron [arxiv:2103.11251 §5]. The
  1312.6199 analysis uses ordinary networks, not disentangled ones, and reports
  no measure of how aligned units are. So it does not contradict the survey's
  claim, but it shows that a neuron-aligned basis is not the default.
- **Rashomon measures.** Inside the survey, the Rashomon section says the two
  size measures disagree and it is unclear which to use
  [arxiv:2103.11251 fig. 17] [arxiv:2103.11251 §9].

The probe card [arxiv:2107.10356 §C] and the EHR study
[arxiv:2605.04666 p. 3] report no comparison with other methods, so they add
no ranking.

## Papers

- [Additive Gaussian Processes](/papers/1112.4394): a GP whose kernel sums products of one-dimensional kernels over all interaction orders, with one variance per order, evaluated in O(D²) with Newton–Girard identities.
- [Intriguing properties of neural networks](/papers/1312.6199): finds small, optimised input changes that flip labels, and reports that single hidden units are no more meaningful than random directions.
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](/papers/2004.13912): one small neural network per input feature, summed with a bias; compared with EBMs, XGBoost and a 10-layer DNN on four tabular datasets.
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](/papers/2103.11251): survey and position paper with five principles and ten open technical challenges.
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](/papers/2107.10356): trains CNNs to predict self-reported race from medical images and probes them with post-hoc tests that do not find a single cause.
- [A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME](/papers/2305.02012): shows that SHAP and LIME feature rankings on tabular data change with the classifier and with collinear features, and proposes NMR and MIP to test and adjust them.
- [Feature importance analysis for patient management decisions](/papers/2605.04666): ranks temporal EHR features by single-feature AUC and fits linear SVMs on the top 1, 3 and 30.
- [The role of explainability in creating trustworthy artificial intelligence for health care](/papers/2007.15911): narrative survey that defines explainability as interpretability plus fidelity, sorts post-hoc methods by type and scope, lists which properties have metrics, and gives a decision guide; no experiments.