The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective

In plain words

Tools like LIME, SHAP and gradient-based saliency explain one prediction of a model by scoring each input feature. Different tools often give different answers for the same prediction. This paper asks how often that happens and what people do about it.

The authors first asked data scientists what counts as two explanations disagreeing. Those answers became six numbers that compare two explanations: do they name the same top features, in the same order, with the same sign. Running these numbers on several models and datasets shows that the tools often disagree. A second study showed people pairs of conflicting explanations. Many people then picked a tool by personal preference, by how well it fit their intuition, or by rules of thumb, rather than by a principled test.

TL;DR

• Post hoc explanation methods often disagree on the same prediction, especially on text and on pixel-level image attributions.
• Six metrics compare two explanations: four on the top-k features, two on a set of features chosen by the user.
• The metrics come from interviews with practitioners, not from a theory of correctness, and the results are author-reported.
• Use the metrics to measure disagreement before you pick or trust one explanation method.

Problem

Post hoc explanations are used in high-stakes settings, and prior work checked their faithfulness against ground truth or by retraining, which is often not possible in real applications §1. The paper notes there is little to no research on whether explanations from different methods agree, or on how practitioners handle the cases where they do not §1.

Method

The paper has four parts. It does not propose a new explanation method.

1. Interviews. Semi-structured interviews with 25 data scientists identified four ways that explanations were judged to disagree: different top features, different order of the top features, different sign of a top feature, and different relative order of features the user cares about §3.1.
2. Metrics. Six metrics formalise these notions. For each pair of explanations on the same prediction, lower values mean stronger disagreement §3.2 app. A. Feature, rank, sign and signed-rank agreement use the top-k features by absolute attribution. Rank correlation and pairwise rank agreement use a feature set chosen by the user §3.2.
3. Empirical analysis. The six metrics are computed for six explanation methods on four datasets and several models §4.1 §4.2. Details are in app. B.
4. User study. An online study with 25 data scientists showed pairs of conflicting explanations for one prediction and asked which they would rely on and why §5.1.

The metrics are the reusable part. Pseudocode for the top-k metrics and the two feature-set metrics follows. The formulas are unnumbered in the source, so they are cited by appendix section.

# Top-k agreement between explanations a and b  [app. A.1]
# Inputs: attribution vectors attr_a, attr_b over features; k
# Output: feature, rank, sign and signed-rank agreement, each in [0, 1]

def topk_agreement(attr_a, attr_b, k):
    # Top-k features by absolute attribution, the set TF  [app. A.1]
    top_a = top_k_by_magnitude(attr_a, k)
    top_b = top_k_by_magnitude(attr_b, k)
    common = set(top_a) & set(top_b)
    # Position R(E, f) and sign S(E, f) of each feature  [app. A.1]
    rank_a = position_by_magnitude(attr_a)
    rank_b = position_by_magnitude(attr_b)
    sign_a = sign(attr_a)
    sign_b = sign(attr_b)
    # Count shared top-k features that also match on the property  [app. A.1]
    feature = len(common) / k
    rank = count(f in common if rank_a[f] == rank_b[f]) / k
    sign_agree = count(f in common if sign_a[f] == sign_b[f]) / k
    signed_rank = count(
        f in common
        if rank_a[f] == rank_b[f] and sign_a[f] == sign_b[f]
    ) / k
    return feature, rank, sign_agree, signed_rank

# Metrics over a user-chosen feature set F  [app. A.2]
# Inputs: attribution vectors attr_a, attr_b; feature indices feats
# Output: rank correlation in [-1, 1]; pairwise agreement in [0, 1]

def rank_correlation(attr_a, attr_b, feats):
    # Spearman correlation of the ranks each explanation gives F  [app. A.2]
    ranks_a = rank_within(attr_a, feats)
    ranks_b = rank_within(attr_b, feats)
    return spearman(ranks_a, ranks_b)

def pairwise_rank_agreement(attr_a, attr_b, feats):
    # Share of feature pairs ordered the same way by both  [app. A.2]
    agree = 0
    for feat_i, feat_j in combinations(feats, 2):
        # Is feat_i more important than feat_j in each explanation?
        a_order = abs(attr_a[feat_i]) > abs(attr_a[feat_j])
        b_order = abs(attr_b[feat_i]) > abs(attr_b[feat_j])
        if a_order == b_order:
            agree += 1
    return agree / comb(len(feats), 2)

The paper's weighted rank agreement and top-k variants are optional extensions that the authors did not use in the main results app. D.2.

Hyperparameters

Model training and explanation settings, from the paper unless marked.

Results

Author-reported. Setup: four datasets (COMPAS, German Credit, AG News, ImageNet images with PASCAL VOC superpixels), six explanation methods, one model per data type unless stated §4.1 §4.2.

The tabular and text figures in §4.3 are heatmaps and box plots with standard errors up to 0.01 (tabular) and 0.0025 (text) fig. 1 fig. 4. The paper reports no confidence intervals on the user-study percentages.

Paper vs code

Official code: https://github.com/AI4LIFE-GROUP/disagreement-problem, MIT licence, cloned at commit 002c081f8f0519ce8da172934a86a750362a12e0. The owner is the AI4LIFE group, the lab of the last author. Paths are relative to the repository root. The paper does not link the code, but its appendix says the full figures are in "the code repository accompanying this paper" app. C.1.

• Training data. The paper does not mention resampling. The code upsamples the minority class in the training split to balance the two classes a-process-data.py. The models are trained on that balanced split b-train-models.py.
• Model count. The abstract and app. B.1 list four tabular models plus the text and image models, which makes six. The code also trains a second logistic model wrapped as a network, and three more feed-forward networks (A, B, C) b-train-models.py. The repository README says eight models. The paper does not use the extra models, and the explanation script uses only logistic, random forest, gradient-boosted and the main network c-generate-explanations.py.
• Pairwise rank agreement is defined twice. metrics.py defines pairwise_rank_agreement at line 53 (top-k version) and again at line 145 (all-feature version). Python keeps the later one, so only the line-145 version runs metrics.py metrics.py.
• Ties. The code ranks features with dense ranks, so tied attributions share a rank metrics.py metrics.py. The paper's R(E, f) is a position in the order app. A.1. The two agree when there are no ties.
• Rank correlation. The code computes Pearson correlation of the dense ranks, not scipy.stats.spearmanr metrics.py. Without ties this is Spearman's coefficient. With ties it can differ from the paper's stated Spearman.
• Importance order. The paper says "more important" without defining it app. A.2. The code orders pairs by absolute attribution metrics.py. The pseudocode above follows the code.
• Feature set. The paper's rank correlation and pairwise agreement take a user-chosen feature set §3.2. For tabular and text the experiments use all features §4.2. The code's rank correlation and pairwise functions take all features in the matrix metrics.py.
• LIME settings. The code sets discretize_continuous=False, sample_around_instance=True, a fixed seed of 12345, and names the categorical columns c-generate-explanations.py. The paper states none of these.
• KernelSHAP background. The code uses the training split as the background set c-generate-explanations.py c-generate-explanations.py. The paper does not state the background set.
• Sample sizes. The code runs every sample size from 25 to 2,000 c-generate-explanations.py and the comparison script loads 2,000 by default e-compare-explanations.py. This matches the paper's tabular choice app. B.2.
• Gradient×Input. The code uses Captum's InputXGradient c-generate-explanations.py. The paper's name for the method matches.
• Image and text pipelines. Not reviewed in detail. The repository folder for images is named cifar, while the paper uses ImageNet images with PASCAL VOC superpixels app. B.2. Check the image pipeline before reusing it.

Pitfalls

• Metric direction. Each metric is an agreement score: higher means more agreement, so lower means more disagreement §3.2.
• Feature agreement is trivially 1 when k is the number of features app. C.1, so it says nothing at that k.
• Top-k uses magnitude. Top-k sets are chosen by absolute value, but sign agreement and signed-rank agreement then compare signs of those features §3.2 app. A.1.
• Ties. The code gives tied features a shared dense rank metrics.py. The paper does not say how ties should be ranked.
• Sample size drives the cost. Tabular explanations used 2,000 perturbations or steps per point, text used 500, and images used 100 to 400 app. B.2. The paper reports no compute time.
• Convergence check is informal. The stopping rule is "does more samples change the explanations", measured by L2 distance and the metrics app. B.2. No threshold is stated.
• Recommended values are exceeded on purpose. The paper uses 500 IG steps for text (recommended lower) and 200 SmoothGrad samples for images (recommended 50) app. B.2. A reimplementation that uses the recommended values may get different disagreement numbers.
• Feature count for Forest Cover is inconsistent. The appendix text gives 54 features app. D.3 and, in the same section, 45. The figure caption gives 54 app. D.3. Treat the count as unresolved.
• Small human study. The user study used one network on the seven-feature COMPAS data, five prompts, and 25 participants §5.1. Percentages are over small counts, and no interval is reported.
• One model per data type in the heatmaps. The heatmaps show a neural network on COMPAS, an LSTM on AG News and ResNet-18 on ImageNet fig. 1 fig. 4 §4.3. Do not generalise these pairs to other models without checking the tabular results for the other three models app. C.1.

Lineage

• Builds on: the explanation methods it compares, LIME, KernelSHAP, Vanilla Gradient, Gradient×Input, Integrated Gradients and SmoothGrad §1 §4.2. Prior faithfulness and fairwashing work is cited in the introduction §1.
• Compared against: L2X in an additional experiment, with the same conclusion of poor agreement app. D.1.
• Superseded by: not stated.