Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

In plain words

Some machine learning models are built so a person can read how they reach an answer, such as a short list of if-then rules or a small table of points to add up. The paper argues that these models should be the first choice for high-stakes decisions, and that trying to explain a complicated black box after the fact is often misleading. It also argues that the choice between accurate and understandable is usually a false one: a readable model often turns out just as accurate, and it is easier to debug.

The paper then lists ten open research problems in building such models. They cover decision trees and scoring systems that are both small and optimal, additive models that stay smooth and simple, neural networks whose individual neurons carry a clear meaning, and reinforcement learning policies a person can read. A reader who wants a starting point in this field can pick one problem and work on it.

TL;DR

A position and survey paper, not a method. It sets out five principles for interpretable ML and then describes ten technical challenges with their history and open questions. Use it as a map of the field and a list of unsolved problems. It reports no new experiments; the quantitative items are small illustrative examples.

Problem

Black-box models are hard to troubleshoot, can rely on the wrong signals, and can be unfair in ways that are hard to detect. Post-hoc explanations of such models can be wrong or misleading, and the paper argues that they are often used to justify keeping a black box that was never needed [Introduction]. The paper's question is which constrained model classes can be as accurate as black boxes while staying understandable [Introduction].

Method

The paper contains no algorithm of its own. It gives a generic formulation of interpretable learning, eq. 1 (see Key equations below), in which a loss term is balanced against an interpretability penalty and an interpretability constraint, and each challenge is a way to define and optimise such a term for one model class [Introduction].

# Generic interpretable supervised learning  [eq. 1] [Introduction]
# Input:  data z_i = (x_i, y_i); model class F; trade-off constant C
# Output: model f in F that meets the interpretability constraint

def fit_interpretable_model(data, model_class, C):
    # Average loss over the training data  [eq. 1]
    # The loss is chosen to match the domain (a placeholder in the paper)
    # Penalty and constraint are domain-specific (placeholders in the paper)
    # Minimise loss + C * penalty over model_class, subject to constraint
    return best_model

Challenges, with the sub-problems the paper names:

1. Sparse logical models (decision trees, decision lists, decision sets) §1. Optimal tree search is NP-complete, so greedy methods are standard; the tree-size objective is eq. 2 §1. Open problems: scale optimal search (1.1), handle continuous variables without losing optimality (1.2), and enforce global constraints such as falling probabilities (1.3) §1.
2. Scoring systems §2. Integer-coefficient linear models, formulated in eq. 3. Four solution families: exact mixed-integer programming, linear programming approximations, sophisticated rounding (Sequential Rounding and Discrete Coordinate Descent are described), and expert-in-the-loop tools §2. Open problems: scale optimal scoring systems (2.1) and make constraint elicitation easier (2.2) §2.
3. Generalized additive models §3. A sum of one-feature functions, with the indicator-threshold form in eq. 4 §3. Open problems: control sparsity, smoothness and monotonicity (3.1), and use GAMs to find data errors and misleading patterns in complex records (3.2) §3.
4. Modern case-based reasoning §4. Nearest-neighbour and prototype methods, including deep prototype networks that compare image patches with learned prototypes fig. 6 fig. 7. Open problems: extend to video (4.1), add prior knowledge or expert supervision to prototype learning (4.2), and repair invalid prototypes (4.3) §4.
5. Complete supervised disentanglement §5. Aligning each concept with its own neuron, expressed by the constraint in eq. 5. Open problems: make a whole layer interpretable (5.1), disentangle all layers at once (5.2), choose which concepts to learn (5.3), and make the mapping from concepts to output interpretable (5.4) §5.
6. Unsupervised disentanglement §6. Generative models and compositional architectures (capsules, slot attention) without given concepts fig. 10 fig. 11. Open problems: quantitative evaluation (6.1), transferring compositional architectures to other domains (6.2), and part-whole disentanglement on large datasets (6.3) §6.
7. Dimension reduction for visualisation §7. Global methods (PCA, MDS) versus local methods (t-SNE, UMAP, PaCMAP) fig. 12 fig. 13 fig. 14. Open problems: capture global structure without losing local detail (7.1), choose hyperparameters, which the paper says are numerous and hard to tune without labels (7.2), and explain the mapping itself (7.3) §7.
8. Physics and causal constraints §8. Physics-informed neural networks minimise the residuals of a PDE together with its initial and boundary conditions fig. 15. Open problems: training (8.1), combining with experimental design (8.2), and constraints for non-neural models (8.3) §8.
9. The Rashomon set §9. The set of models with near-optimal loss. Open problems: characterise it (9.1), choose a measure of its size (9.2), compute or approximate that measure (9.3), visualise it (9.4), and choose a model from it (9.5) §9. Two candidate size measures are the Rashomon ratio and the pattern Rashomon ratio, illustrated on a decision-stump example fig. 17.
10. Interpretable reinforcement learning §10. Policies as decision trees or rule lists, and symbolic or relational state spaces. Open problems: which constraints give an accurate interpretable policy (10.1), under what assumptions one exists (10.2), and how to simplify the state space (10.3) §10.

A final section lists further problems: preprocessing, uncertainty, splitting easy and hard cases, tabular neural networks, co-design with visualisation, interpretable causal matching, variable importance, deployment, legal requirements, and other model families §11.

Hyperparameters

This paper proposes no trained method, so no hyperparameters are reported. The only tunable constants in its formulas are:

Results

Survey: there are no benchmark results of the authors' own. Figures from the paper are the only quantitative items. All numbers are author-reported.

The paper states that no general accuracy-interpretability tradeoff has been shown [Introduction], and supports this with examples and cited experiments, not with a theorem.

Paper vs code

No official code. The paper is a survey and links no repository. A GitHub repository search for "interpretable machine learning grand challenges" found no results. Survey repositories found by such a search, such as awesome-lists, would not count as code.

Pitfalls

• The sleep-apnea scoring example is inconsistent. The text says points are given for diabetes §2, but the table's row 2 is hypertension tab. 2. The table's total row says "Add points from row 1-6", while only five rows are listed tab. 2. Treat the table as illustrative, not as a validated tool.
• The accuracy-equals-interpretability argument rests on examples and cited experiments [Introduction]. The paper calls its own position a principle, not a proven result.
• The monk-2 comparison fig. 3 is one dataset with one tree of each kind. Whether the reported test difference is significant is not stated.
• The NP-completeness claim for optimal decision trees is cited to a 1976 paper §1. It is not proved in this survey.
• Two Rashomon-set size measures can give different answers for the same model class fig. 17. The paper says it is not clear which one to choose fig. 17 §9.
• Challenge 10.1 has a title but no body text §10. Read it as an open question with no further detail.
• Equation 1 and equation 3 leave the loss and penalty as placeholders (for example InterpretabilityPenalty). They are not implementable without a choice of loss and penalty for each task eq. 1 eq. 3.
• Several figures in the source are commented out and do not appear in the published layout (e.g. a scoring-system figure and a policy-tree variant) §2 §10. Figure numbers in this card follow the rendered paper.

Lineage

• Builds on: the Rashomon effect of Breiman, and earlier work on sparse decision trees, scoring systems, and GAMs that the paper cites in each section. No single prior method is named as the base.
• Compared against: post-hoc explanation methods (SHAP, LIME, saliency maps), which the paper argues against [Introduction].
• Superseded by: not known from content already in this wiki.