Explaining a Series of Models by Propagating Shapley Values

In plain words

Models are often chained, so that one model's output is the next model's input. A deep network may feed a decision tree, or several companies may each run one model in a stack. Explaining a single prediction from such a chain is hard. The usual explanation tools are slow, or they assume they can open up one model at a time.

This method passes the explanation backward through the chain. The last model says how much each of its inputs mattered. Each earlier model then turns that credit into credit for its own inputs, using a fast method suited to its type (neural, tree or linear). Every link is rescaled so that the credit still adds up to the change in the final output. The result is a list of scores for the original input features, which together account for the gap between this prediction and a typical one.

The "typical" comparison is an average over many reference examples, not a single arbitrary one. The reference examples can be chosen by clustering, so that the comparison is with a meaningful group, such as people of the same age and sex. The approach is aimed at debugging, and at pipelines whose parts cannot be shared. The speed and quality numbers come from the authors and have not been reproduced here.

TL;DR

DeepSHAP explains a series of models (deep, tree and linear, and their compositions) by propagating Shapley-value attributions backward through the chain. At each link the credit is rescaled so that the attributions still sum to the output change §2 §6.4. Each single-baseline attribution is averaged with equal weight over a baseline distribution, which the authors show is the interventional Shapley value §6.2. The authors report about an order of magnitude speed-up over model-agnostic methods on an image-feature pipeline, and explanations for a stack whose models are never shared §4.3 §4.4. All results are author-reported, the guarantees are approximate for mixed series §5, and the code repository has no LICENSE file.

Problem

Model-agnostic local explainers (IME, LIME, KernelSHAP) can explain a series of models. But their sampling-based estimates vary, and their cost may be intractable for large pipelines §1. Model-specific methods are fast: DeepLIFT and Integrated Gradients for deep networks, and TreeSHAP or Gain for trees. They generally cannot explain a series of mixed models §1.

A single-baseline attribution, which compares each input with one reference sample, is biased. The paper shows this on CIFAR-10, where an all-black reference gives little credit to dark pixels §3.1 fig. 2.

In a distributed series, each model belongs to a different institution. A model-agnostic method then needs every model, which the owners will not share §1 §4.4. Shapley values are taken with an interventional conditional expectation and a flat causal graph. This lift does not need the joint density of the features §6.1.

Method

Overview. Each model in the series receives an approximate Shapley attribution for its own inputs. These are chained from the last model back to the first. The single-baseline results are then averaged over a baseline set §2 §6.2 §6.4.

Step 1. Choose the explicand and the baseline distribution D. The default is the training samples, each with equal weight §6.3. Alternatively, cluster a reduced set of training features with k-means, and use the cluster whose mean is nearest the explicand §6.3 eq. 14 eq. 15. The authors use age and sex as the clustering features in their NHANES experiment app. A.2.2.

Step 2. For one baseline x^b, get each model's attribution φ̂. Use DeepLIFT with the Rescale rule for deep layers, which the paper describes as a 1-partition approximation to the interventional Shapley value §6.6. Use interventional TreeSHAP for tree models, and exact Shapley values for linear models §6.4 §6.6. A 2-partition variant (RevealCancel) is also described §6.6.

Step 3. Propagate backward through the series. Start with the last model's attribution (ψ^k) eq. 2. For each earlier model i, divide the incoming credit element-wise by that model's output change, f_i(x^e) − f_i(x^b). Set a quotient to 0 where the divisor is 0. Then apply φ̂ of that model to the result eq. 3. Every intermediate credit vector sums to the same total, f_k(x^e) − f_k(x^b) §6.5, Theorem 1.

Step 4. Average over baselines. With an empirical baseline distribution D, the interventional Shapley value is the equal-weight mean of the single-baseline attributions eq. 11 eq. 12. Each single-baseline result sums to f_k(x^e) − f_k(x^b) eq. 13.

Step 5 (optional). Group features. Sum the attributions of each group, then rescale so the group values add up to the total attribution §6.7 eq. 33 eq. 34. Features not in any group go into a residual group.

Pseudocode for steps 3 and 4 follows. Here attribute(model, a, b) returns the approximate attribution matrix of shape (inputs, outputs). The [eq. N] tags match the numbering in the compiled paper.

# Propagate one baseline's credit back through the series  [§2] [§6.4]
# Input:  models [h_1 .. h_k] (each with an attribution function),
#         explicand x_e, baseline x_b
# Output: one attribution per raw input feature, shape (m,)

def series_attribution(models, x_e, x_b):
    # Outputs of every model, for both samples  [§2]
    outputs_e = [x_e]
    outputs_b = [x_b]
    for model in models:
        outputs_e.append(model(outputs_e[-1]))
        outputs_b.append(model(outputs_b[-1]))
    # Last model: its attribution matrix gives psi^k  [eq. 2]
    credit = attribute(models[-1], outputs_e[-2], outputs_b[-2])[:, 0]
    # Earlier models: rescale the credit by each output change  [eq. 3]
    for index in reversed(range(len(models) - 1)):
        change = outputs_e[index + 1] - outputs_b[index + 1]
        # Division is 0 where the change is exactly 0  [§2]
        ratio = safe_divide(credit, change)
        matrix = attribute(models[index], outputs_e[index], outputs_b[index])
        credit = matrix @ ratio
    return credit

# Average single-baseline attributions over the baseline set  [§6.2]
# Input:  models, explicand x_e, baselines of shape (|D|, m)
# Output: interventional Shapley attributions, shape (m,)

def deepshap(models, x_e, baselines):
    # One attribution vector per baseline sample  [eq. 12]
    per_baseline = [
        series_attribution(models, x_e, x_b) for x_b in baselines
    ]
    # Equal-weight mean over the empirical baseline distribution  [eq. 11]
    return mean(per_baseline, axis=0)

# Group attributions with a residual group  [§6.7] [eq. 33] [eq. 34]
# Input:  feature attributions (m,), membership (groups, m) of 0 and 1
# Output: one attribution per group, plus the residual group

def group_attribution(attributions, membership):
    # Residual group: features that no listed group covers  [§6.7]
    uncovered = 1 - membership.max(axis=0)
    membership = vstack([membership, uncovered])
    # Sum the members of each group  [eq. 33]
    group_sums = membership @ attributions
    # Rescale so the groups add up to the total attribution  [eq. 34]
    total = attributions.sum()
    return group_sums * safe_divide(total, group_sums.sum())

Key equations, as typeset in the paper:

\phi(f,x^e)=\frac{1}{|D|}\sum_{x^b\in D}\phi(f,x^e,x^b) eq. 1

\psi^k=\hat{\phi}(h_k,x^e,x^b) eq. 2

\psi^i=\hat{\phi}(h_i,x^e,x^b)\big(\psi^{i+1}\oslash(f_i(x^e)-f_i(x^b))\big),\quad i=1,\dots,k-1 eq. 3

\mu^{int}_D(f,x^e,S)=\mathbb{E}_D[f(X)\mid do(X_S=x^e_S)] eq. 8

\sum_i\phi^{\mu_D}_i(f,x^e)=f(x^e)-\mathbb{E}_D[f(X)] eq. 9

\sum_i\phi_i(f,x^e,x^b)=f(x^e)-f(x^b) eq. 13

\phi^0_{G_j}(f,x^e)=\sum_{i\in G_j}\phi_i(f,x^e) eq. 33

\phi_{G_j}(f,x^e)=\phi^0_{G_j}(f,x^e)\times\frac{\sum \phi_{G_j}(f,x^e)}{\sum \phi_i(f,x^e)} eq. 34

In the Hadamard division, the paper sets a_i/b_i to 0 when b_i is 0 §2 §6.4.

Two further results sit in the appendix. If every model is linear, the propagation gives exact interventional Shapley values §6.4 app. A.5.3. A bagged or boosted ensemble is linear in its members, so it is explained directly app. A.5.4. The interventional baseline decomposition has a proof in the appendix app. A.5.2.

Hyperparameters

Values are as stated in the paper. Appendix citations use the section numbers of the compiled paper. "not stated" means the paper gives no value.

Results

Author-reported. The paper has no tables. Headline numbers are in the text and appendix. Hardware and wall-clock times are not stated in the text, and the runtime values are only in the plotted figures. The first seven rows give the accuracy of the models being explained, which sets up the experiment. They are not DeepSHAP's own performance. The paper measures DeepSHAP's quality with ablation curves, which are plotted and not tabulated.

Other claims, all from the authors: a distributed stack can be explained without sharing its models if all parties use the same baseline set §4.4; DeepSHAP's ablation scores are comparable to IME on MNIST, and better than KernelSHAP and LIME there §4.3; and the ablation of the HELOC stack is competitive with the best methods §4.4.

Paper vs code

The official repository is suinleelab/DeepSHAP, linked in the paper's Code availability section §10 (commit cae8ffb, cloned at depth 1). The owner is the authors' lab. The repository has no LICENSE file, so code_license is "not stated", and we do not quote its code.

• Stacking code needs an external fork of shap. deepshap/explanation.py imports a local shap from ../../shap explanation.py. It also calls shap_values(..., model_stack=True) [code: deepshap/explanation.py#L123] explanation.py. This repository does not define that argument. The paper does not mention a fork.
• Feature-extractor and gradient-boosting hyperparameters are partly set by library defaults. The paper's HELOC GBT values (learning rate 0.1, depth 3) app. A.2.6 do not appear in the notebook, which calls XGBClassifier() with no arguments [code: 5_model_stack/2 Lending example.ipynb#L105] 2 Lending example.ipynb. The HELOC credit MLP learning rate is 0.005 in the paper app. A.2.6, but the notebook calls fit_mlp(X_train_credit, y_train) without eta [code: 5_model_stack/2 Lending example.ipynb#L147], so the default 0.001 applies model.py. The NHANES stacking MLPs do use 0.005 1 NHANES Model Ensemble.ipynb.
• Undocumented output scaling. model_pred multiplies Keras MLP outputs by 0.999 model.py. The paper does not mention this. The NHANES ensemble notebook imports the helper [code: 5_model_stack/1 NHANES Model Ensemble.ipynb#L45].
• Stacking MLP epochs are not in the paper. fit_mlp trains for 10 epochs model.py. The paper gives no epoch count for this experiment.
• Gene-set size filter ignores its argument. filter_max_size takes max_size but then sets max_size = 200 [code: 2_gene_expression_pathway/gene_set.py#L31]. The paper does not describe a size filter, and its size-bias discussion is commented out in the source.
• Group rescale normaliser. Eq. 34 as typeset puts the group's own sum in the numerator. The code instead divides the total attribution over all features by the sum over groups, including the residual group [code: 2_gene_expression_pathway/gene_set.py#L53-L54] [code: 2_gene_expression_pathway/gene_set.py#L79]. This is the normalisation that makes the groups add up to the total, which is the paper's stated intent §6.7.
• Zero division. The code divides only where the denominator is not exactly zero, and sets the result to 0 otherwise [code: deepshap/explanation.py#L136-L138]. This matches the paper §2.
• Efficiency check. The code asserts the stacked sum matches the interventional TreeExplainer sum to an absolute tolerance of 1e-5 [code: deepshap/explanation.py#L167]. The paper states the identity but no tolerance.
• Validation set in NHANES training. The MLP is fitted with validation_data=(trainx, validy), which is the training set [code: 1_multiple_references/NHANES_background_distribution/model.py#L37-L38]. The paper does not mention this. It does not change the saved weights, since there is no early stopping or model selection, but the validation curve is not held out.
• Ablation imputation. The paper's Methods defines imputation with the baseline sample §6.8. The code takes the reference as an argument and defaults it to zeros evaluation.py. The appendix imputes with explicand means for NHANES loss app. A.2.4, and with the mean of the baseline distribution for MNIST app. A.2.5.
• Model-agnostic sample count. The paper uses 100,000 samples for model-agnostic methods on MNIST fig. 6 app. A.2.5, and the notebook does the same [code: 4_feature_extraction/2 Zero prediction CNN->XGB ablation.ipynb#L182]. The library wrapper default is 5000 [code: deepshap/explanation.py#L8], which the notebooks override.

Pitfalls

• Baseline count drives cost and memory. Each baseline needs its own pass through the series. The code keeps one rescale factor per baseline, hidden unit and explicand explanation.py. Memory therefore grows with the product of those three counts. The paper gives no cost model.
• Near-zero output changes. The rule only guards exact zeros. A tiny change in a model's output gives a very large quotient, and the attributions downstream of that link inherit it. The paper does not discuss this.
• Efficiency is approximate for mixed series. Efficiency holds for each intermediate attribution if each model's attributions are efficient §6.5. Exactness holds only when every model is linear §6.4 app. A.5.3. The paper says the method is not guaranteed to satisfy axioms such as implementation invariance §5.
• Tolerances with large outputs. The code's efficiency check uses an absolute tolerance of 1e-5 explanation.py. A model with large unnormalised outputs may fail it even when the method is correct. Check the tolerance against the output scale.
• Explain the transformed output on purpose. Log-odds, probability and loss are all series outputs. Attributions in different output spaces can rank features differently. The paper shows this for log-odds and probability app. A.4.2.
• Distributed use needs an agreed baseline set. The privacy benefit holds only if every institution uses the same baselines §4.4. The paper does not say how to agree on them in practice.
• Off-manifold baselines. The interventional lift can evaluate the model on unlikely combinations of features. The paper suggests clustering to keep baselines in a realistic subpopulation §3.2 §6.1. It does not measure how much this matters.
• Group normalisation depends on overlap. Overlapping groups make the sum over groups exceed the total, and the rescale then shrinks every group §6.7 eq. 33 eq. 34. This is the case the paper's typeset equation leaves unclear (see Paper vs code).
• Ablation curves need matched imputation. Comparing ablation curves is only valid when the same imputation values are used for every method §6.8. The appendix and the code use different reference values across experiments (see Paper vs code).
• Published hyperparameters do not fix the HELOC result. The HELOC GBT and credit-MLP settings in the paper are not what the notebook runs (see Paper vs code). Expect different numbers unless those settings are corrected.
• Runtime and quality numbers are only in figures. Speed claims such as "two orders of magnitude" come from plots fig. 6 fig. 7 §4.2. The text gives no seconds and no hardware.

Lineage

• Builds on: DeepLIFT's Rescale rule, applied per model and chained across a series §6.4 §6.6. DeepLIFT is cited without an arXiv id in the paper's bibliography. The interventional lift follows Janzing et al. (arXiv:1910.13413) §2. Interventional TreeSHAP follows arXiv:1905.04610 §4.2.
• Compared against: IME, LIME and KernelSHAP as model-agnostic explainers §1 §4.3. Integrated Gradients (arXiv:1703.01365) is named as another efficiency-satisfying method that could be chained the same way §4.4.
• Extends: the linear-model exactness result builds on Chen et al. (arXiv:2006.16234) §6.2 app. A.5.3.
• Superseded by: not known from this wiki.