Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead

In plain words

Many machine learning models are black boxes: even their builders cannot easily read how they reach an answer. A common response is to build a second model that explains the first one after the fact. This paper argues that for decisions with serious consequences, such as bail, parole, medical care or lending, that is the wrong fix. The explanation is only an approximation, so it can be wrong or misleading, and it can hide problems in the first model.

The alternative is to build models a person can read from the start: short rule lists, small point-based scoring sheets, or models made of parts that can each be inspected. On data with meaningful features, such models are often about as accurate as black boxes. The paper also lists the open technical problems in building them well, and suggests policy changes that would reward doing so.

TL;DR

A position paper, not a method. For high-stakes decisions, it argues for inherently interpretable models over black boxes with post-hoc explanations, which it says can be unfaithful, misleading or incomplete §1 §2. It names three algorithmic challenges: optimal logical models, optimal sparse scoring systems, and interpretable computer vision §5. It reports no new experiments §6.

Problem

• Post-hoc explanations are separate models that approximate a black box, so they are not what the original model computes. Even a fairly accurate surrogate can use different features, and a surrogate that agrees 90% of the time is wrong on the remaining 10% §2 (ii).
• Explanations can be misleading when shown only for the predicted class. Saliency maps show where a network looks, not what it does there, and they can look alike for every class §2 (iii) fig. 2.
• Black boxes are hard to combine with information outside the database, such as the seriousness of a current charge, which a fixed risk score may ignore §2 (iv).
• Complex scoring procedures with many manually entered inputs are prone to human error. The paper cites a recidivism instrument with more than 130 factors §2 (v).
• The paper claims the accuracy/interpretability trade-off is largely a myth for structured data with meaningful features, where simple and complex classifiers often perform about the same after preprocessing §2 (i) app. B.
• Scope: high-stakes decisions and troubleshooting of models, which the paper names as the main reasons for needing interpretability §1.

Method

This is an argument, not an algorithm. Its structure:

1. Why post-hoc explanations fall short §2: five issues, (i) to (v), as above.
2. Why interpretable models are not adopted, and the paper's answers §3: (i) a black box can be a source of profit, since a simple transparent model offers no licensing revenue §3 (i); (ii) interpretable models take domain expertise and constrained optimisation, which the paper considers worth the cost for high-stakes decisions §3 (ii); (iii) a transparent model that is flexible enough can find the same hidden patterns a black box finds §3 (iii).
3. Policy proposals §4:
   • a mandate that no black box is deployed where an interpretable model with the same performance exists, which would put the onus on the vendor;
   • a weaker version, requiring organisations that deploy black boxes to report the accuracy of interpretable methods;
   • paying developers of interpretable models a lump sum and releasing the model as a public good, rather than relying on licence revenue. The paper says these will not solve everything and could reduce participation where ML could help §4.
4. Challenge 1, optimal logical models §5. Minimise the training error plus a penalty on model size, over a family such as decision trees or rule lists eq. 1. The paper's example, a three-rule list from the CORELS algorithm, is shown in fig. 3. The paper says the problem is generally hard, and that CORELS solves its special case to optimality using bounds on the search space, a bit-vector library and specialised data structures [§5 Challenge 1]. Extensions to falling rule lists and binary-split trees are described as in progress §5 Challenge 1.
5. Challenge 2, optimal sparse scoring systems §5. A scoring system is a linear model with small integer coefficients (point scores). Minimise the average logistic loss plus a penalty on the number of non-zero scores, with scores restricted to the integers from −10 to 10. The paper names RiskSLIM as a cutting-plane method that adds cuts when the linear relaxation is integral and branches otherwise §5 Challenge 2 fig. 4.
6. Challenge 3, interpretability for vision §5. Define interpretability per domain, then build it into the model. The paper points to prototype networks, where a prototype layer learns prototypical parts from training images, and the class score is a weighted sum of similarities between parts of the input and those prototypes. The explanation is the computation itself, not a post-hoc summary §5 Challenge 3 fig. 5. The paper says these networks have been trained to be roughly as accurate as the black-box networks they were derived from §5 Challenge 3.
7. Rashomon argument §5, Rashomon subsection: if many models are about as accurate as the best one on a dataset, one of them is likely to be interpretable. The paper gives this as an argument and refers to a formal version by Semenova et al.
8. Instability as an advantage app. E: many equally accurate models mean a domain expert can choose the most interpretable one.
9. Counterfactual explanations app. C: the paper argues they do not suffice for high-stakes decisions, because the "lowest-cost" change depends on each person's cost, which is hard to elicit.
10. Smaller-than-global explanations app. D: a complicated global model, such as a disjunctive normal form model or a falling rule list, can still give short explanations for each individual decision.

Key equations:

# Objective of Challenge 1 [eq. 1]: penalised training error
# Input:  model f from a family F (trees, rule lists, ...); n labelled rows
# Output: scalar penalised error

def penalised_error(model, inputs, labels, lam):
    # Count training points the model misclassifies  [eq. 1]
    mistakes = 0
    for i in range(n):
        if model.predict(inputs[i]) != labels[i]:
            mistakes = mistakes + 1
    error = mistakes / n
    # Size: logical conditions (e.g. leaves, rules)  [eq. 1]
    size = count_conditions(model)
    return error + lam * size

\min_{f \in \mathcal{F}} \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\text{$f$ misclassifies observation } i] + \lambda \cdot \mathrm{size}(f) \right) eq. 1

# Scoring-system objective, Challenge 2 (unnumbered display)  [§5 Challenge 2]
# Input:  integer scores b_1..b_p in {-10, ..., 10}; covariates x[i][j]
# Output: logistic loss plus sparsity penalty

def scoring_objective(scores, covariates, lam):
    # Average logistic loss over the n observations  [§5 Challenge 2]
    total = 0
    for i in range(n):
        margin = sum(scores[j] * covariates[i][j] for j in range(p))
        total = total + log(1 + exp(-margin)) / n
    # Penalty: number of non-zero point scores  [§5 Challenge 2]
    nonzero = count(scores[j] != 0 for j in range(p))
    return total + lam * nonzero

\min_{b \in \{-10,\ldots,10\}^p}\; \frac{1}{n}\sum_{i=1}^{n} \log\!\Bigl(1 + \exp\bigl(-\textstyle\sum_{j} b_j x_{i,j}\bigr)\Bigr) + \lambda \sum_{j} \mathbf{1}[b_j \neq 0] §5 Challenge 2

Hyperparameters

The paper reports no experiments, so it fixes no training settings. The values below are the paper's illustrations or search domains.

Results

Author-reported. Setup: no new experiments. The results below are prior work or illustrative statements. fig. 1 is explicitly not generated from data.

Paper vs code

No official code.

Searched: the manuscript text and arXiv metadata contain no repository link for this paper. The CORELS link in fig. 3 and the RiskSLIM method are other papers' work, not this manuscript's code. One GitHub repository search by title keywords returned nothing relevant. No code was cloned.

Pitfalls

• No method to implement. There is no algorithm, training procedure, dataset or hyperparameter set to reproduce. Implement from the cited works (CORELS, RiskSLIM, prototype networks), not from this card.
• Units of λ in eq. 1. The error term is a fraction of n, so λ trades off fractions of training error against one unit of size. The 0.01 example is the paper's illustration, not a recommended value §5 Challenge 1.
• Size depends on the model class. Leaves for trees, rules for lists, non-zero coefficients for scoring systems. The paper gives no single definition §5 Challenge 1 §5 Challenge 2.
• Labels in the scoring objective. The logistic-loss display has no label y_i. How labels enter the covariates (for example, a sign convention) is not stated. Whether an intercept is penalised is not stated §5 Challenge 2.
• Integer search domain. The −10 to 10 bound appears only in the display, and the paper does not justify it §5 Challenge 2.
• Figure 1 is illustrative. The paper says it has no quantified axes and no data behind it. Do not cite it as evidence for the shape of the accuracy/interpretability trade-off §2 (i) fig. 1.
• Saliency argument is qualitative. The claim that saliency maps can look the same across classes is argued from an example, not measured §2 (iii) fig. 2.
• Arithmetic on an assumed error rate. The "more than half of forms have an error" claim assumes a 1% per-entry error rate over 130+ factors. The rate is not measured in the paper §2 (v).
• Third-party cases are the author's account. The COMPAS, ProPublica and BreezoMeter statements describe cited reporting and proprietary systems. The paper says the BreezoMeter failure cannot be explained from outside §3. Treat these as the paper's characterisation, not verified facts.
• Rashomon is informal. "Reasonably accurate" has no numeric threshold in the paper. The formal version is in Semenova et al. §5 Rashomon subsection.
• Prototype accuracy claim is about prior work. The statement that prototype networks roughly match their base networks rests on cited papers, and no numbers appear in this paper §5 Challenge 3.
• Counterfactual "minimal change" needs user costs. It cannot be computed from the model alone app. C.
• Time-bound statements. Statements about GDPR, California bail reform and the state of the field are as of the paper's writing (version 3, updated 2019-09-22) and may be out of date §3 (i) §4.
• Policy proposals are suggestions. The paper presents them as ideas, and it is not legal advice.
• Appendices are part of the source. The \end{document} before the appendices is commented out in the fetched LaTeX, so app. A to E are included in this card's citations.

Lineage

• Builds on: none; this is a commentary. It notes that a preliminary version appeared at a 2018 workshop §1.
• Cited methods: CORELS (Angelino et al.) §5 Challenge 1; RiskSLIM (Ustun and Rudin) §5 Challenge 2; prototype networks (Chen, Li and colleagues) [§5 Challenge 3]; the Rashomon argument (Semenova, Parr and Rudin) §5.
• Compared against: post-hoc explanation methods such as saliency maps and counterfactual explanations §2 (iii) app. C, and the linear surrogate used in the ProPublica analysis of COMPAS §2 (ii).
• Superseded by: not known from wiki content.