OpenXAI: Towards a Transparent Evaluation of Model Explanations

In plain words

Explanation tools such as LIME or SHAP say which inputs pushed a model toward its answer. Many of them exist, they often disagree, and each comes with its own test code, so comparing them is slow and easy to get wrong. OpenXAI is a shared test bench for this. It bundles datasets, trained models, the common explanation methods and eleven scoring checks in one Python package.

One of the datasets is generated on purpose, so the reasons behind its answers are known. The checks ask three things: whether an explanation matches those reasons, whether changing the inputs it marks as important changes the answer more than changing the rest, and whether it moves too much after a tiny nudge to the input.

TL;DR

OpenXAI packages datasets, pretrained logistic regression (LR) and neural network (ANN) models, six feature-attribution methods plus a random baseline, and eleven metrics for faithfulness and stability into one pipeline §2. The benchmark numbers are author-reported. On LR, Gradient, Integrated Gradients and SmoothGrad score (near-)perfectly on most ground-truth agreement metrics, and Integrated Gradients is the most stable §3. The MIT-licensed code implements the setup, but it differs from the paper in several details that change the numbers (see Paper vs code).

Problem

• Explanations are used in high-stakes settings, so their reliability matters, but there was no systematic and reproducible way to benchmark them. Metrics and implementations were scattered across papers and libraries §1.
• Earlier synthetic benchmarks take the ground truth from the data-generating process. A model trained on such data need not follow that ground truth, so a correct explanation of the model can be scored as wrong §2. The paper's generator is built so that an accurate model must follow the ground truth under its stated assumptions app. C.
• Small qualitative studies, such as gradient maps for MNIST, do not show whether their findings hold in other settings §1.
• Scope: local feature-attribution methods on tabular data. The first release supports only tabular datasets [abstract].

Method

OpenXAI has four parts §2. Code paths below are relative to the repository at code_commit.

1. Datasets and models. Eight datasets: one synthetic and seven real-world tabular sets tab. 1 app. D.1. Two pretrained model types per dataset: LR and an ANN with two hidden layers of 100 ReLU units and a softmax output §3 app. D.4. Model weights ship with the package app. D.4.
2. Explainers. Vanilla Gradient, SmoothGrad, Integrated Gradients (IG), Gradient x Input, LIME and KernelSHAP, plus a random baseline §2 tab. 6. The implementations wrap Captum and the authors' LIME package app. D.2.
3. Metrics. Eleven metrics, grouped as ground-truth faithfulness (six), predictive faithfulness (two) and stability (three) app. A. Fairness is defined as comparing the average of these metrics across two subgroups app. A §3.
4. Benchmark runner. Explanations are generated for the first 1000 test instances of each dataset and model §3. Each metric is then computed per instance and averaged app. A.

Ground-truth agreement app. A

The explanation's top-K features are compared with the ground truth. For LR, the ground truth is the model's own coefficients, namely the difference between the two class weight vectors model.py. Six metrics:

• FA (feature agreement): fraction of top-K features shared with the ground truth.
• RA (rank agreement): shared features that also sit at the same rank.
• SA (sign agreement): shared features with the same attribution sign.
• SRA (signed rank agreement): shared features with the same sign and rank.
• RC (rank correlation): correlation between the two feature rankings, over all features.
• PRA (pairwise rank agreement): fraction of feature pairs ordered the same way, over all features.

For FA, RA, SA and SRA, the reported value is the area under the curve of the metric against K, computed over all values of K app. A. The released config caps K at 25% of the features (see Paper vs code). RC and PRA use all features app. A.

Predictive faithfulness app. A

Following Petsiuk et al. 2018 app. A, PGI perturbs the top-K features and keeps the rest; PGU does the reverse. Both report the average change in model output. Higher PGI is better, lower PGU is better.

# Predictive faithfulness for one instance  [eq. 1] [eq. 2] [app. A]
# Input:  x (d,), explanation e (d,), model f returning class probabilities,
#         k (number of top features), invert (False: PGI, True: PGU)
# Output: mean absolute change in the model output (scalar)

def prediction_gap(x, e, f, k, invert, n_samples=100, sigma=0.1):
    # Indices of the k features with the largest absolute attribution  [app. A]
    top = argsort(-abs(e))[:k]
    # PGI fixes the non-top features; PGU fixes the top features  [app. A]
    if invert:
        fixed = top
    else:
        fixed = features_not_in(top)
    free = not_fixed(fixed)
    # Gaussian noise on free features  [app. A]
    # Discrete features flip instead, with probability sigma * sqrt(2/pi)
    # [code: openxai/explainers/perturbation_methods.py#L6-L11]
    noise = normal(0, sigma, (n_samples, d))
    x_prime = x + noise * free
    # Average the absolute change in the output  [eq. 1] [eq. 2]
    return mean(abs(f(x) - f(x_prime)))

Relative stability app. A

Stability asks whether the explanation changes much more than the input or the model's internal state does when the input is perturbed very slightly. The score is the largest ratio of relative changes over the sampled neighbours.

# Relative stability for one instance  [eq. 3] [eq. 4] [eq. 5] [app. A]
# Input:  x (d,), explanation e (d,), explainer, model f, metric name
# Output: largest ratio of relative changes (scalar), or undefined

def relative_stability(x, e, explainer, f, metric):
    # Perturb every feature by a tiny Gaussian (sigma = 1e-5)  [app. D.3]
    candidates = x + normal(0, 1e-5, (1000, d))
    # Keep up to 100 neighbours that get the same predicted label  [app. D.3]
    same = candidates[f.label(candidates) == f.label(x)]
    kept = same[:100]
    worst = undefined
    for x_prime in kept:
        # Relative change of the explanation, p-norm with p = 2  [eq. 3]
        e_change = norm(e - explainer(x_prime)) / norm(e)
        # Relative change of the chosen representation  [eq. 3] [eq. 4] [eq. 5]
        r = representation(x, metric, f)
        r_prime = representation(x_prime, metric, f)
        r_change = norm(r - r_prime) / norm(r)
        worst = max(worst, e_change / r_change)
    return worst

def representation(x, metric, f):
    # RIS: the input; RRS: first hidden layer; ROS: model output  [eq. 3-5]
    if metric == "RIS":
        return x
    if metric == "RRS":
        return f.first_hidden_layer(x)
    return f.output(x)  # probabilities per [eq. 5]; the code uses logits

Stability values are reported as the log of the mean and the log of the standard error, so lower values mean more stable evaluate_metrics.py tab. 4 tab. 5.

Synthetic dataset, SynthGauss app. C

Clusters are well separated, features are independent (identity covariance), and each cluster has its own sparse ground-truth explanation. Parameters are in app. D.3: n = 1000 per cluster, d = 20, K = 10 clusters, cluster-centre distance κ = 6, mask probability p = 0.25, weights drawn from U(-1, 1), 25% test split.

# SynthGauss  [alg. 1] [app. C]
# Input:  n instances, K clusters, centres mus (K, d), covariances sigmas
#         (K, d, d), weights ws (K, d), masks ms (K, d)
# Output: features X (n, d), binary labels y (n,)

def synthgauss(n, K, mus, sigmas, ws, ms):
    # Pick a cluster for each instance, at random  [alg. 1]
    clusters = random_choice(K, size=n)
    # Sample each instance from its cluster's Gaussian  [alg. 1]
    X = sample_normal(mus[clusters], sigmas[clusters])   # (n, d)
    # Class probability: sigmoid of the masked weights applied to x  [alg. 1]
    effective = ws[clusters] * ms[clusters]              # (n, d)
    pi = sigmoid(sum(effective * X, axis=1))             # (n,)
    # Threshold at the median so that the classes are balanced  [alg. 1]
    y = (pi > median(pi)).astype(int)
    return X, y

Theorem 1 states that under the three stated properties (independent features, well-separated clusters, one unique ground truth per cluster), the most accurate model adheres to each cluster's ground truth app. C.

Hyperparameters

Values are from the paper's appendix app. D.3 unless marked as code. The code config is the one the repository uses for its benchmark run experiment_config.json.

Per-dataset training settings. Model selection picks the best test-accuracy epoch after warmup, provided the mean predicted label stays within a bound app. D.4.

Source: experiment_config.json. The paper says these settings are dataset-specific app. D.4 but does not list them.

Results

Author-reported. Setup: 1000 test instances per dataset, LR and ANN models, six explainers plus a random baseline §3. Standard errors appear in the source tables and are omitted here.

Datasets tab. 1

The synthetic size conflicts with the appendix parameters; see Paper vs code.

Model accuracy tab. 7

Ground-truth faithfulness, LR tab. 2 tab. 3 app. D.6

HELOC tab. 2 (PRA, RC, FA, RA, SA, SRA, PGU down, PGI up):

Adult tab. 3:

Synthetic app. D.6 (PRA, RC, FA, RA, SA, SRA, PGU down, PGI up):

Text summary §3: Gradient, IG and SmoothGrad reach perfect scores on the six agreement metrics, and LIME approaches them in many cases. On predictive faithfulness, Gradient, SmoothGrad and LIME are the most faithful on PGI; the PGU trend is similar, with IG on par with the leaders. The ANN results and the other datasets are in app. D.5 to app. D.7 and were not transcribed here.

Stability, LR tab. 4 tab. 5

Log of the mean; lower is more stable. Values are as printed in the tables.

Across all datasets, the text reports log averages of 1.95 (RIS) and 1.86 (ROS) for IG, 3.16 and 3.37 (RIS) for Gradient and Gradient x Input, and 2.91 and 4.49 (ROS) for the same two; LIME, SHAP, SmoothGrad and the random baseline fall between about 10 and 15 §3.

Fairness fig. 1 fig. 2

Two bar charts show average PGU for the majority (male) and minority (female) subgroups on German and Adult with LR. Larger gaps between the bars mean larger disparities. The text says Gradient x Input and SHAP show the most gap on Adult §3. The repository contains no subgroup code, so these figures cannot be regenerated from the released code (see Paper vs code).

Paper vs code

The paper describes the setup; the code at code_commit is the reference. Code paths are relative to the repository.

• SmoothGrad samples. Paper: 500 app. D.3. Code: default 100 smoothgrad.py and config 100 experiment_config.json.
• IG baseline. Paper: not stated. Code: the training-set mean is the baseline experiment_utils.py; the same training data feeds LIME experiment_utils.py.
• LIME kernel width. Paper: 0.75 app. D.3. Code: 0.75 times the square root of d lime.py. Code also multiplies each LIME weight by +1 or -1 according to the predicted label lime.py.
• Explained class. Paper: not stated. Code explains the predicted class generate_explanations.py metrics.py.
• Ground-truth sign. Paper: not stated. Code flips the LR ground truth for instances predicted as class 0 metrics.py.
• Area under the curve over K. Paper: all values of K app. A. Code: K from 1 to ceil(0.25 * d) under the config experiment_config.json experiment_utils.py metrics.py. The same cap applies to the predictive metrics metrics.py.
• ROS output. Paper: prediction probabilities f(x) eq. 5. Code: logits for LR model.py and ANN model.py, used in metrics.py.
• RRS layer. Paper: output of the first layer app. A. Code: post-activation (after ReLU) of the first hidden layer by default model.py metrics.py.
• Rank correlation. Paper: Spearman app. A. Code: Pearson correlation of dense ranks metrics.py. These agree without ties.
• Undefined stability ratios. Paper: silent. Code: skips undefined ratios and drops NaN instances, printing the count metrics.py metrics.py.
• Stability neighbours. Paper: 1000 samples, subset of 100 app. D.3. Code: takes the first 100 same-label samples, which can be fewer metrics.py.
• Synthetic weights. Paper: a separate weight vector w_k per cluster app. C. Code: one weight vector shared by all clusters, masked per cluster dgp_synthetic.py dgp_synthetic.py.
• Synthetic labels. Paper: median threshold alg. 1. Code: a grid search over 1000 thresholds to get a 50/50 split dgp_synthetic.py dgp_synthetic.py.
• Synthetic cluster spacing for K > d. Paper: multipliers 2 kappa, then 3 kappa app. C. Code: 2 kappa, then 4 kappa dgp_synthetic.py.
• Synthetic defaults. Paper: n = 1000 per cluster, K = 10, kappa = 6, p = 0.25, seed 564 app. D.3. Code class defaults: n = 10,000 per cluster, K = 100, kappa = 5, p = 0.5, seed 567 dgp_synthetic.py. The released files were not inspected.
• Scaling. Paper: min-max scaling of train and test app. D.4. Code: a separate min-max scaler fitted for each file dataloader.py.
• Training loss. Paper: not stated. Code: cross-entropy applied to the softmax output, so softmax is applied twice model.py model.py model.py.
• Train/test split. Paper: random 80/20 when no split is given app. D.1. Code: loads one train and one test file per dataset from Dataverse dataloader.py. The origin of each split was not checked.
• Fairness. Paper: subgroup averages and disparities app. A §3. Code: no subgroup, fairness or minority code in the Python sources.
• Metric count. Paper: "twenty-two" metrics in §2 and eleven in the abstract. Code: eleven experiment_config.json.

Pitfalls

• Ground truth is the model, not the data. For LR, "ground truth" is the model's own coefficients model.py. Agreement measures faithfulness to the model. The paper's synthetic argument covers the data side only under its stated assumptions app. C.
• Gradient and IG match ground truth by construction (inferred). For a binary LR, the gradient of the predicted-class probability is a per-instance scalar times the coefficient difference model.py. IG with multiply_by_inputs off averages those gradients integrated_gradients.py. Ranks and signs therefore match the ground truth for every instance. This is an inference from the code, not a claim the paper makes. The text says gradient methods are perfect on all six agreement metrics for all datasets §3, but Table 3 shows VanillaGrad and SmoothGrad at 0.998 to 0.999 on FA, RA, SA and SRA for Adult tab. 3.
• Test-set model selection. The best epoch is chosen on the test split model.py. Reported model accuracy is therefore optimistic.
• Per-split scaling. Test features are scaled with the test set's own min and max dataloader.py. A model applied to new data must be scaled with training statistics instead.
• Cost of stability. For one instance, up to 100 perturbed explanations are computed metrics.py metrics.py. With LIME at 1000 samples per explanation lime.py, that is on the order of 10^5 model queries per instance. The config uses parallel jobs experiment_config.json experiment_config.json.
• Maxima are sensitive. Stability is a maximum over neighbours, so one outlier sets the score. Undefined cases are dropped; the paper does not say so, and the code only prints the count metrics.py.
• ROS depends on the output space. The paper uses probabilities; the code uses logits. Ratios of relative changes differ between the two eq. 5.
• Seeds depend on test order. Metric seeds equal the test index metrics.py. Re-ordering the test set changes results.
• Stochastic explainers. LIME and SmoothGrad draw noise. Seeds are set in the code lime.py. The card did not test whether runs reproduce across library versions.
• Pretrained weights are external. Models are fetched from a Dataverse record model.py, and the training script in the repository may not reproduce those weights exactly. Not tested.
• Synthetic size. Table 1 gives 5,000 rows, but the appendix parameters give 1000 per cluster across 10 clusters, i.e. 10,000 app. D.3 tab. 1.
• Metric count. The text says both eleven and twenty-two metrics §2; the code implements eleven.

Lineage

• Builds on: the agreement metrics of Krishna et al. 2022, which this paper uses for ground-truth faithfulness app. A; the relative stability metrics of Agarwal et al. 2022 §2; and the perturbation idea of Petsiuk et al. 2018 for PGI and PGU app. A.
• Implementations come from Captum and the LIME package app. D.2.
• Compared against: a random baseline §3.
• Superseded by: not known from content on this wiki.