---
kind: concept
slug: explanation-evaluation
title: Explanation evaluation (judging whether an XAI explanation is good)
summary: >-
  How to check whether an explanation of a model is any good. A 2022 review splits "good" into twelve quality properties and catalogues 29 automated checks. Other papers use qualitative criteria, ablation curves, explainer disagreement, attention baselines or chain-of-thought prompt-change tests. No single metric is endorsed; the right check depends on what you can do to the model and data.
tags: [interpretability, explainable-ai, evaluation]
papers: ["1803.07517", "1908.04626", "2007.15911", "2011.05429", "2012.05876", "2103.04244", "2103.15679", "2105.00108", "2108.04840", "2110.10790", "2201.08164", "2202.01602", "2305.04388", "2309.01029", "2504.10527"]
related: [attention-interpretability, attention-mechanism, counterfactual-explanation, causal-inference, explainable-ai, feature-attribution, mechanistic-interpretability, chain-of-thought-prompting, llm-evaluation]
status: draft
updated_at: 2026-10-31
---

## Overview

An explanation method (for example, a feature-importance or heatmap method)
produces an explanation for one prediction, or for the model as a whole. The
open question is how to tell whether that explanation is good. Many papers
judge it with a few hand-picked examples that look plausible. The review
treats this "face validity" check as insufficient, because a plausible
explanation can still misrepresent the model, and the paper's reviewed set
reports that 33% of its 312 introducing papers use only such evidence, by the
review's own count [arxiv:2201.08164 §1] [arxiv:2201.08164 §5].

The shared idea across the variants is to break "explanation quality" into
separate, measurable properties, then pick a check for each property you care
about. Notation: a predictive model is f, an explanation method is e. A local
explanation is e(f(x)) for one input; a global one is e(f); a self-explaining
model has e = f [arxiv:2201.08164 §1.1].

Earlier work framed the same goal with a few qualitative criteria rather than a
catalogue. Its main criterion is fidelity, meaning how well the explanation
agrees with the model's input-output mapping [arxiv:1803.07517 §5.1]. Later
work uses that criterion as a measurement, for example by removing features and
watching the output [arxiv:2103.15679 §5] [arxiv:2105.00108 §3.2].

A 2020 position paper on neurosymbolic AI also takes fidelity, meaning how
closely an explanation matches the model rather than the data, as the test to
apply. It argues that many local methods have low fidelity, citing an earlier
measurement that the paper does not reproduce, and that rules extracted from a
network should carry confidence values [arxiv:2012.05876 §5]. It runs no
experiments [arxiv:2012.05876 §1].

A 2021 survey of post-hoc methods for neural NLP sorts evaluation into three
kinds: application-grounded (tested in the real deployment), human-grounded
(tested with people or a stand-in for people) and functionally-grounded (tested
for whether the explanation reflects the model). Most NLP work uses only the
last two. The survey also argues that simulated users are often used in place
of real ones, and that this inflates results [arxiv:2108.04840 §4]. A 2023 study
takes a different route: it changes the prompt in a way that should not matter,
then checks whether the written reasoning mentions the change [arxiv:2305.04388 §2].
A 2019 paper on attention weights compares them with a uniform baseline, with
seed-to-seed variation and with an adversary that builds different attention
for a similar prediction [arxiv:1908.04626 §3.2] [arxiv:1908.04626 §4].

## Variants

The variants fall into several families. They differ in what they measure and in
how much evidence they need from you.

### Co-12 properties and the automated checks

The review defines twelve explanation-quality properties, in three groups
[arxiv:2201.08164 §3.3] [arxiv:2201.08164 tab. 2]:

- **Content:** Correctness (faithful to the model), Completeness (how much of
  the model's behaviour is described), Consistency (identical inputs give
  identical explanations), Continuity (small input changes give small
  explanation changes), Contrastivity (explanations differ across targets and
  can answer "why not?"), Covariate complexity (features are understandable and
  their interactions simple).
- **Presentation:** Compactness (the explanation is small), Composition (format
  and organisation), Confidence (certainty information is present and accurate).
- **User:** Context (relevant to the user's needs), Coherence (agrees with prior
  knowledge), Controllability (the user can influence the explanation).

Correctness (faithfulness) and Coherence (plausibility) are kept apart on
purpose [arxiv:2201.08164 §2.3]. The card for this paper is
[From Anecdotal Evidence to Quantitative Evaluation Methods](/papers/2201.08164).

The review catalogues 29 automated methods, each with a primary property, the
explanation types it applies to, and what it checks [arxiv:2201.08164 §6]
[arxiv:2201.08164 tab. 3]. Grouped by property:

| Property | Methods (number in the catalogue) |
|---|---|
| Correctness | Model parameter randomisation (1); explanation randomisation (2); white-box check (3); controlled synthetic data check (4); single deletion (5); incremental deletion or addition (6) |
| Completeness | Preservation check (7); deletion check (8); fidelity (9); predictive performance (10) |
| Consistency | Implementation invariance (11) |
| Continuity | Stability for slight variations (12); fidelity for slight variations (13); connectedness (14) |
| Contrastivity | Target sensitivity (15); target discriminativeness (16); data randomisation check (17) |
| Covariate complexity | Covariate homogeneity (18); covariate regularity (19) |
| Compactness | Size (20); redundancy (21); counterfactual compactness (22) |
| Composition | Perceptual realism (23) |
| Confidence | Confidence accuracy (24) |
| Context | Pragmatism (25); simulated user study (26) |
| Coherence | Alignment with domain knowledge (27); agreement between XAI methods (28) |
| Controllability | Human feedback impact (29) |

The review says the table in [arxiv:2201.08164 tab. 4] lists which properties a
method *can* measure. It does not say which were measured in each paper
[arxiv:2201.08164 §6].

### Qualitative criteria and taxonomy

Before the catalogue, one paper set out four criteria and left them
unmeasured: fidelity, interpretability (split into clarity and parsimony),
generalizability across architectures, and explanatory power, meaning how many
user questions a method can answer [arxiv:1803.07517 §5.1]. The paper calls
fidelity the most important property. It gives no measurement procedure, so the
ratings are labels, not scores [arxiv:1803.07517 §5.1] [arxiv:1803.07517 §5.2].
The same paper sorts methods into rule extraction, attribution and intrinsic
classes and rates each against those criteria [arxiv:1803.07517 §5.2]. The card
is [Explanation Methods in Deep Learning](/papers/1803.07517).

The neurosymbolic position paper gives a checklist of the same kind, without
ratings. Its explanation argument is that a rule set can show a proof trace,
which a raw activation pattern cannot, and that fidelity is the property to
judge explanations by [arxiv:2012.05876 §5]. It is a position, not a test, so
it adds no check to the catalogue. The card is
[Neurosymbolic AI: The 3rd Wave](/papers/2012.05876).

Two more sources give a checklist and a method for designing explanations, not
a test. A human-centred survey chapter sorts explanation techniques by more than
50 user questions in nine categories and pairs each category with candidate
methods. The authors present that pairing as a suggestion, not a validated result
[arxiv:2110.10790 §3] [arxiv:2110.10790 tab. 1]. It also lists pitfalls: an
explanation may not serve the user's real goal, and people often skim them rather
than use them [arxiv:2110.10790 §4]. The NLP survey's per-family notes say that
validation is often weak or missing, especially for human-centred and
faithfulness checks [arxiv:2108.04840 §4] [arxiv:2108.04840 §17]. The cards are
[Human-Centered Explainable AI (XAI)](/papers/2110.10790) and
[Post-hoc Interpretability for Neural NLP: A Survey](/papers/2108.04840).

A 2025 semi-systematic review of explainable AI in food quality reaches a
similar point from the application side. It counts how studies explain their
models, and finds that most use visual heatmap methods on pictures, with
Grad-CAM-family methods preferred over LRP, and it lists formal evaluation of
explanations among its open gaps [arxiv:2504.10527 §9] [arxiv:2504.10527 tab. 2]
[arxiv:2504.10527 §1]. Its counts describe what was used, not how well it
worked, so it gives no check to apply. The card is
[Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review](/papers/2504.10527).

### Which properties have a metric, by explanation type

A 2020 survey of explainable AI for health care maps each explanation type to
the properties it can be checked for. Each cell is one of three marks: satisfied
by definition, measurable with a generic metric, or no known metric
[arxiv:2007.15911 tab. 2] [arxiv:2007.15911 §5]. The map is the authors'
classification from their literature review. It shows where to look for a
metric, not what any method scores.

- **Surrogate fidelity.** For model-based post-hoc explanations, soundness has
  a metric: the share of predictions on which the surrogate and the model agree
  [arxiv:2007.15911 §5.1]. A surrogate's completeness is marked as satisfied by
  construction, since it can always compute the model's output, so no metric is
  applied to it [arxiv:2007.15911 §5.1] [arxiv:2007.15911 tab. 2].
- **Attribution soundness and completeness.** Attribution soundness has three
  kinds of test: artificial ground truth, perturbation tests, and axiomatic
  tests such as sensitivity and implementation invariance [arxiv:2007.15911 §5.2].
  Attribution completeness, meaning whether the scores account for the full
  change in output, has no metric in the map [arxiv:2007.15911 tab. 2].
- **Parsimony and clarity.** Parsimony of model-based explanations can be
  counted: features, rules or tree splits, tree depth, operation counts, and a
  feature-interaction complexity measure. Global clarity of rule-based
  explanations has an unambiguity measure built on rule overlap and coverage.
  Local clarity is marked as no metric for model-based explanations, but as a
  generic metric for attribution-based ones [arxiv:2007.15911 §5.1] [arxiv:2007.15911 tab. 2].
- **Example-based explanations.** The authors know of no evaluation method for
  them, for either scope [arxiv:2007.15911 §5.3] [arxiv:2007.15911 tab. 2]. A
  check for counterfactuals therefore has to come from the counterfactual
  family above, not from this map.

The health-care survey is the only source here that gives this full map. Its
numbers are not checked by experiment. The card is
[The role of explainability in creating trustworthy artificial intelligence for health care](/papers/2007.15911).

### Perturbation and ablation curves in method papers

Two method papers use their own tests rather than the catalogue.

- **Positive and negative perturbation AUC.** Remove image or text tokens in
  order of their attribution and track accuracy. A larger area is better for
  negative perturbation (removing the most relevant tokens hurts), and a smaller
  one is better for positive perturbation (keeping only the relevant tokens
  preserves accuracy). This is the same deletion idea as checks 5 and 6 in the
  catalogue. For ViT-Base on ImageNet, the card reports negative-perturbation
  AUC of 54.61 for the proposed method against 54.16 for the best baseline [arxiv:2103.15679 tab. 2]. The method is described in [Generic Attention-model Explainability](/papers/2103.15679).
- **Ablation curves for pipelines.** Replace the least important features with
  reference values and track the model's score. The DeepSHAP paper reports that
  its ablation scores are comparable to IME on MNIST and better than KernelSHAP
  and LIME there [arxiv:2105.00108 §4.3]. It also reports that a k-means
  reference set "substantially outperforms" a general-population reference, in a
  qualitative comparison without a table [arxiv:2105.00108 §3.2] [arxiv:2105.00108 fig. 3].
  The card is [Explaining a Series of Models by Propagating Shapley Values](/papers/2105.00108).

The 2103.15679 card also reports a DETR weakly supervised segmentation test on
COCO validation, with AP 13.1 against 7.2 for the best baseline [arxiv:2103.15679 tab. 1]. This
is an overlap-with-annotation test, which the catalogue files under alignment
with domain knowledge (27).

### Planted-bug tests with known ground truth

Instead of scoring an explanation against the model, this family plants a known
bug in the model or its data and checks whether the explanation shows it. The
bugs are a spurious background, mislabelled training examples, re-initialised
top layers and out-of-distribution inputs [arxiv:2011.05429 §1] [arxiv:2011.05429 §2.1].
The authors compare gradient-family, path, surrogate and modified
back-propagation methods on a birds-vs-dogs CNN and a VGG-16 model [arxiv:2011.05429 §2.2].

- **Spurious background.** Against a relaxed ground-truth mask, the attribution
  SSIM ranges from 0.78 (KernelSHAP) to 0.98 (VGrad) [arxiv:2011.05429 tab. 2].
  The authors read this as the attributions picking up the background
  shortcut [arxiv:2011.05429 §3.1].
- **Mislabelled training examples.** SSIM between attributions from the correct
  and the flipped-label model ranges from 0.73 to 0.99 across the methods. The
  authors conclude that these methods are unlikely to identify mislabelled
  examples [arxiv:2011.05429 §3.2] [arxiv:2011.05429 fig. 6].
- **Re-initialised top layers.** Guided Backprop, DConvNet, PatternNet,
  PatternAttribution and LRP-SPAF stay visually and quantitatively unchanged as
  higher layers are re-initialised, so they cannot show that kind of defect.
  The paper reports this qualitatively, with no numeric table
  [arxiv:2011.05429 §4] [arxiv:2011.05429 fig. 7]. This is the same randomisation
  idea as checks 1 and 2 in the catalogue, but on planted bugs.

The same paper runs a 54-person study. With labels present, participants
rejected the random-label and re-initialised models mostly on predicted labels,
and were unsure about the spurious-background model [arxiv:2011.05429 §6]
[arxiv:2011.05429 app. H]. Its authors say the 10-way breed task may make
people lean on labels more than on attributions [arxiv:2011.05429 §6]. The
card is [Debugging Tests for Model Explanations](/papers/2011.05429).

### Counterfactual properties

For counterfactual explanations, a review lists five properties of a good
counterfactual: proximity (close to the original input), plausibility (does not
change features that cannot change), sparsity (few changed features), diversity
and feasibility [arxiv:2103.04244 §4.2]. The paper defines each in words and
gives no formula or threshold for any of them [arxiv:2103.04244 §4.2]. Distance
is usually an L_p norm; the review notes that L1 favours sparse changes
[arxiv:2103.04244 §4.1]. The card is
[Counterfactuals and Causability in Explainable AI](/papers/2103.04244). For
counterfactuals, the properties are the design targets, and the numbers still
have to come from the original algorithm's paper.

### Disagreement between explanation methods

This family asks how often two explainers disagree on the same prediction. The
six metrics are agreement scores, so low values mean strong disagreement. Four
compare the top-k features by absolute attribution: feature agreement (same
features), rank agreement (same order), sign agreement and signed-rank agreement.
Two compare a feature set the user chooses: rank correlation and pairwise rank
agreement [arxiv:2202.01602 §3.2] [arxiv:2202.01602 app. A]. On the paper's
models, LIME and KernelSHAP on image superpixels have high rank correlation
(0.8977), while most AG News text pairs have rank and signed-rank agreement under
0.1 [arxiv:2202.01602 tab. 2] [arxiv:2202.01602 §4.3.2]. Gradient methods at pixel
level disagree strongly: rank correlation is 0.001 between Integrated Gradients and
SmoothGrad [arxiv:2202.01602 §4.3.3]. These are author-reported results on a few
models, so treat them as a reference point rather than a benchmark. Feature
agreement is trivially 1 when k equals the number of features [arxiv:2202.01602 app. C.1].

In a 25-person study, practitioners who met disagreements mostly settled them
with ad hoc heuristics, not with a test [arxiv:2202.01602 §5.2.4]. Agreement is
a coherence signal, not a check of faithfulness [arxiv:2201.08164 §2.3]. The card
is [The Disagreement Problem in Explainable Machine Learning](/papers/2202.01602).

### Baseline and adversarial tests for attention weights

These tests apply to attention weights in a model that has attention built in.
The paper works on single-layer bidirectional LSTM text classifiers and on
binary tasks only [arxiv:1908.04626 §3.1]. It runs four tests, each a
baseline or a stress test rather than a score against annotations
[arxiv:1908.04626 §5].

- **Uniform-attention baseline.** Freeze the attention to uniform weights and
  measure the drop in F1. On Diabetes the paper reports 0.775 for the
  reproduced base model against 0.706 for uniform attention, and on IMDb 0.902
  against 0.879 [arxiv:1908.04626 tab. 2]. The paper drops AG News and 20
  Newsgroups from later tests, because uniform attention matched the base model
  there [arxiv:1908.04626 §3.2].
- **Seed-variance baseline.** Train several models that differ only in their
  initialisation seed, and measure how far their attention moves. The paper uses
  eight seeds and reports that the spread depends on the dataset
  [arxiv:1908.04626 §3.3] [arxiv:1908.04626 fig. 3]. Divergence from the base
  model is read against this spread.
- **Frozen-weight MLP guide.** Impose a set of attention weights on a
  non-contextual per-token MLP and compare F1. On Diabetes the paper reports
  0.753 with base LSTM attention, 0.699 with a trained MLP's attention and 0.503
  with the adversary's attention. On Anemia the adversary scores 0.932 against
  0.931 for the base attention. The paper attributes this to the label skew
  and to sparse indicative tokens in positive examples [arxiv:1908.04626 tab. 3]
  [arxiv:1908.04626 §4].
- **Consistent adversary.** Train a second model whose predictions stay close to
  the base model while its attention is pushed away from the base attention
  [arxiv:1908.04626 §4]. The best adversary per dataset, among those with mean
  JSD above 0.4, has JSD of 0.405 to 0.514 on the four datasets kept for this
  test [arxiv:1908.04626 tab. 4].

The paper's reading is that alternative attention with similar predictions does
exist for some binary tasks, but it guides a simple model poorly, so attention
is still useful as an explanation under the paper's definitions [arxiv:1908.04626 §5]
[arxiv:1908.04626 tab. 3]. The card on the paper is
[Attention is not not Explanation](/papers/1908.04626).

## Prediction-change tests for chain-of-thought explanations

For a language model that writes its reasoning before answering, the check is
to add a feature that should not affect the answer and see whether the answer
moves and the reasoning mentions the feature. In the Suggested Answer setting, a
user line hints at a random option. On 13 BIG-Bench Hard tasks, GPT-3.5 zero-shot
accuracy falls by 36.3 percentage points under that hint, and Claude 1.0 by 30.6
[arxiv:2305.04388 tab. 8]. The hand-checked sample is small, but it shows that
the explanations rarely mention the hint: 0 of 234 BBH explanations did
[arxiv:2305.04388 app. B]. On a social-bias benchmark, the explanations for
unfaithful answers mostly support the stereotype-aligned answer [arxiv:2305.04388 tab. 5].
The authors call the test necessary but not sufficient. A failure shows
unfaithfulness; a pass does not prove faithfulness [arxiv:2305.04388 §6]. The
experiments use legacy GPT-3.5 and Claude 1.0 endpoints, and the paper does not
test other models [arxiv:2305.04388 §7]. The card is
[Language Models Don't Always Say What They Think](/papers/2305.04388).

### Rationale removal and plausibility for LLM explanations

A 2023 survey of LLM explainability groups evaluation into two kinds for
fine-tuned models. Plausibility asks whether the explanation matches human
rationales, scored with IOU, precision, recall, F1 or AUPRC. Faithfulness
asks whether the model's output depends on the rationale. For the second kind
the survey gives two formulas [arxiv:2309.01029 §5] [arxiv:2309.01029 §5.1]:

- **Comprehensiveness** compares the model's score for class j on the full input
  with its score once the rationale is removed.
- **Sufficiency** compares the full-input score with the score on the rationale
  alone.

The survey does not define the model symbol in these formulas. The card reads
it as the model's class probability, following the surrounding prose
[arxiv:2309.01029 §5.1]. The card records no threshold and no preferred
direction for either score, so check the survey before reading a result as
good or bad. For prompted models the same survey points to counterfactual
simulatability and to chain-of-thought faithfulness tests [arxiv:2309.01029 §5.2],
which the prediction-change test above is one example of. The survey lists the
lack of ground-truth explanations as an open challenge, which limits the
plausibility check [arxiv:2309.01029 §6.1]. The card is
[Explainability for Large Language Models: A Survey](/papers/2309.01029).

## When to use which

Choose checks from what you can do to the model, the data and the people
available. The sources give no ranking of methods across families.

| Situation | Use | Why |
|---|---|---|
| You can randomise or re-initialise the model's weights and re-run it | Model parameter randomisation (1) | The explanation should change when the learned parameters are destroyed [arxiv:2201.08164 §6.1] |
| The explanation is built into the model | Explanation randomisation (2) | Randomising the built-in explanation should change the output [arxiv:2201.08164 §6.1] |
| You have an interpretable model whose reasoning you know | White-box check (3) | The explanation should match the known reasoning [arxiv:2201.08164 §6.1] |
| You can generate data with a known rule and the model learns it | Controlled synthetic data (4) | Tests whether the explanation recovers the rule; the review warns that the model must be shown to follow it, e.g. by near-perfect accuracy [arxiv:2201.08164 tab. 3] |
| You can remove or mask features and re-run the model | Deletion-based checks (5, 6, 8) | Output change should follow the explanation's ranking; compare with a random order [arxiv:2201.08164 §6.1] [arxiv:2201.08164 §6.2] |
| Your explanation is for a Transformer with two inputs or an encoder-decoder | Perturbation AUC, as in the method paper | The card reports it on a VQA model and a ViT, but the AUC integration rule and step grid are not stated in the paper [arxiv:2103.15679 §5] [arxiv:2103.15679 tab. 2] |
| You have a pipeline of models and need explanations that do not need every model | Ablation curves with a shared reference set | The card reports this setting and that the reference set matters [arxiv:2105.00108 §3.2] [arxiv:2105.00108 §4.4] |
| You have ground-truth labels or an interpretable reference | Predictive performance (10) | Measures how well the interpretable model or explanation matches ground truth [arxiv:2201.08164 §6.2] |
| You have domain masks, boxes or expected explanations | Alignment with domain knowledge (27) | Compares explanations with an expected one; for text the review lists ROUGE or BLEU [arxiv:2201.08164 §6.11] [arxiv:2103.15679 tab. 1] |
| You need to know whether small input changes flip the explanation | Stability for slight variations (12) | Similar inputs should give similar explanations [arxiv:2201.08164 §6.4] |
| You need a cheap proxy for usefulness without people | Simulated user study (26) | Scores explanation utility on a synthetic task [arxiv:2201.08164 §6.10] |
| You want to compare two explainers | XAI methods agreement (28) | Agreement is a coherence signal. It does not establish faithfulness, since the review separates plausibility from correctness [arxiv:2201.08164 §2.3] |
| You are early in a design and cannot run tests yet | Qualitative criteria | Fidelity, interpretability, generalizability and explanatory power give a checklist, not scores [arxiv:1803.07517 §5.1] |
| You have a surrogate explanation and want a number for how well it matches the model | Agreement rate between surrogate and model predictions | This is the soundness metric for model-based post-hoc explanations [arxiv:2007.15911 §5.1] |
| You need to show that attribution scores account for the whole output change | None in the map; argue it from the method | Attribution completeness is marked as having no metric [arxiv:2007.15911 tab. 2] |
| You need to evaluate example-based explanations (prototypes, influential instances, counterfactuals) | None known; use the counterfactual properties for counterfactuals | The survey found no evaluation method for example-based explanations [arxiv:2007.15911 §5.3] [arxiv:2103.04244 §4.2] |
| You need a count-based measure of how simple a rule or tree explanation is | Counts of features, rules or splits, and tree depth | Parsimony of model-based explanations has these metrics [arxiv:2007.15911 §5.1] [arxiv:2007.15911 tab. 2] |
| You need to judge a counterfactual explanation | Counterfactual properties | Proximity, sparsity and plausibility are defined, but not measured [arxiv:2103.04244 §4.2] |
| You want rule-based explanations and need a stated quality bar | Fidelity, with rule confidence values | The position paper argues for both; it reports no measurements [arxiv:2012.05876 §5] |
| You want to know whether two explainers agree on a prediction | Disagreement metrics (top-k and user-chosen feature set) | Quantifies disagreement; agreement is a coherence signal, not faithfulness [arxiv:2202.01602 §3.2] [arxiv:2201.08164 §2.3] |
| You have a model with attention weights and want to know whether they do more than a uniform map | Uniform-attention baseline, with F1 drop | Tests whether the learned weights carry signal the model uses [arxiv:1908.04626 §3.2] [arxiv:1908.04626 tab. 2] |
| You want to know whether the attention you see is one of many possible ones | Consistent adversary, with a seed-variance baseline | Shows whether alternative attention can match the base predictions; the paper also reports that it guides a simple model poorly [arxiv:1908.04626 §4] [arxiv:1908.04626 tab. 4] |
| You have a chain-of-thought model and can edit its prompt | Prediction-change test with a biasing feature | Shows whether a change that should not matter moves the answer without appearing in the reasoning; passing does not prove faithfulness [arxiv:2305.04388 §2] [arxiv:2305.04388 §6] |
| You have human-annotated rationales for a fine-tuned text model | Plausibility scores (IOU, precision, recall, F1, AUPRC) against those rationales | Measures agreement with people, which the survey files under plausibility rather than faithfulness [arxiv:2309.01029 §5] |
| You have a fine-tuned text model and a selected rationale, but no human rationales | Comprehensiveness and sufficiency | Removal and keep-only tests of faithfulness; the survey gives no threshold for a good score [arxiv:2309.01029 §5.1] |
| You are designing an explanation for a user group and have no test yet | Question-driven design from user research | Maps user questions to techniques; the mapping is a suggestion and the stopping rule is not stated [arxiv:2110.10790 §3] [arxiv:2110.10790 tab. 1] |
| You have a training set you can corrupt on purpose (background shortcut, flipped labels) | Planted-bug tests | Shows whether the explanation exposes a known defect; the same paper found that a shortcut shows up but flipped labels do not [arxiv:2011.05429 §3.1] [arxiv:2011.05429 §3.2] |
| You can re-initialise the top layers of a pretrained network | Cascading re-initialisation, as in the planted-bug study | Modified back-propagation methods did not change, so an unchanged map does not show the model is intact [arxiv:2011.05429 §4] |
| You want to know whether a model's labels or its explanations drive human trust | Human study with planted defects | With labels present, participants relied mostly on labels, so a user study can say little about the explanation alone [arxiv:2011.05429 §6] |
| You are scoping an application area and need to know what has been tried | Application review, not a test | Counts methods by data and explanation type; it reports no evaluation scores [arxiv:2504.10527 §9] |
| You need evidence from human users | Not in the main text | The main text covers only functionally-grounded methods. User-study methods are in the supplement [arxiv:2201.08164 §6] |

## Implementation sketch

The review gives no code for the checks, and this sketch is not from the
authors. It shows the selection loop in a generic form. Each check's formula
must be taken from the original paper in [tab. 3] of the review.

```pseudocode
# Choose the Co-12 properties your use case needs  [arxiv:2201.08164 §3.3]
properties = choose_properties(use_case)
# Map each property to the catalogue entries that test it  [arxiv:2201.08164 §6]
checks = [c for c in catalogue if c.primary_property in properties]
for check in checks:
    # Skip checks whose explanation types do not match yours  [arxiv:2201.08164 tab. 3]
    if not check.explanation_types & my_explanation_types:
        continue
    # Run the check on the model, the explainer and the data  [arxiv:2201.08164 §6]
    score = check.run(model, explainer, data)
    # Report the score with the caveat the review gives for that check  [arxiv:2201.08164 §6]
    report(check.name, score, caveat=check.caveat)
```

## Contradictions

No paper reports numbers that contradict another on the same test. The
differences below are about framing and scope.

- **Fidelity as one criterion or as several.** The 2018 survey treats fidelity
  as the most important single property [arxiv:1803.07517 §5.1]. The 2022
  review separates Correctness (faithfulness) from Coherence (plausibility), and
  says their trade-off must be justified for each application [arxiv:2201.08164 §2.3]
  [arxiv:2201.08164 §7]. Both are kept on this page.
- **What a model-agnostic explanation shows.** The counterfactual review argues
  that model-agnostic methods capture correlations between features rather than
  the mechanism behind a prediction [arxiv:2103.04244 §1.2]. The DeepSHAP paper
  defines its attributions with an interventional expectation, which is written
  with a do-operator [arxiv:2105.00108 eq. 8]. It does not claim its attributions
  reflect a causal mechanism. The two papers do not test each other's claims,
  and this page does not resolve the question.
- **Low fidelity of local methods.** The 2020 position paper says LIME-style
  local methods have low fidelity, citing an earlier measurement [arxiv:2012.05876 §5].
  The 2018 and 2022 sources do not test that claim. The DeepSHAP paper reports
  MNIST ablation scores in which LIME and KernelSHAP fall below DeepSHAP
  [arxiv:2105.00108 §4.3]. That is a comparison within one test, not a check of
  the earlier measurement, and neither result is re-run here.
- **Mixed numbers within one paper.** The Transformer-explainability card reports
  that its method beats the baselines overall [arxiv:2103.15679 tab. 1]. On
  positive perturbation for ViT, Transformer attribution has the lower AUC
  (17.03 against 17.32 for the proposed method, for the predicted class), and
  the paper calls the gap mild [arxiv:2103.15679 tab. 2] [arxiv:2103.15679 §5].
- **Fidelity and completeness as terms.** The 2020 health-care survey splits
  fidelity into completeness (the explanation can compute the output) and
  soundness (the explanation is true to the model) [arxiv:2007.15911 §2.2]. The
  2022 review uses Correctness for faithfulness and lists Completeness as a
  separate property, about how much of the model's behaviour is described
  [arxiv:2201.08164 §3.3]. The survey's evaluation section also uses
  "completeness" for a different, sensitivity-style property [arxiv:2007.15911 §5.2].
  The same word covers three definitions here, so this page keeps them apart and
  does not map one onto another.
- **Whether attributions expose a spurious background.** The planted-bug paper
  reports high SSIM between attributions and the background-only mask, and reads
  it as the attributions showing the shortcut [arxiv:2011.05429 tab. 2]. Its user
  study does not show the same result: a majority said the attribution did not show
  the part they expected, yet they did not reject the spurious model convincingly
  [arxiv:2011.05429 §3.1] [arxiv:2011.05429 app. H]. Both results come from one
  paper and one set of planted bugs, so this page keeps both and does not resolve
  the question.
- **Which faithfulness metric to trust.** The LLM survey notes that the
  best-ranked explanation under one faithfulness metric can be the worst under
  another [arxiv:2309.01029 §5.1]. It does not resolve this, so this page does
  not pick a single faithfulness metric.
- **Whether attention is an explanation.** [Jain and Wallace](https://arxiv.org/abs/1902.10186)
  (no card here) argue that attention is not explanation, because different attention
  can give similar predictions. The 2019 paper answers that the claim depends on
  what "explanation" means, and that adversarial attention exists for some binary
  tasks [arxiv:1908.04626 §2] [arxiv:1908.04626 tab. 4]. It also reports that
  adversarial attention guides an MLP poorly on most datasets [arxiv:1908.04626 tab. 3].
  Neither side is tested beyond that paper's setup, so this page keeps both.
- **Simulated users as a proxy.** The 2022 review lists a simulated user study as
  a cheap stand-in for usefulness [arxiv:2201.08164 §6.10]. The 2021 NLP survey
  argues that simulated users inflate results, because explaining well to people
  needs interdisciplinary input [arxiv:2108.04840 §4]. Neither source tests the
  other's claim, and this page keeps both.

## Papers

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](/papers/2201.08164): defines the Co-12 properties and catalogues 29 automated evaluation methods, each mapped to the properties it checks.
- [Attention is not not Explanation](/papers/1908.04626): four tests of attention weights in LSTM text classifiers (uniform baseline, seed variance, frozen-weight MLP guide, consistent adversary); binary tasks only; author-reported.
- [Debugging Tests for Model Explanations](/papers/2011.05429): four planted bugs with known ground truth; tests whether gradient-family, path, surrogate and modified back-propagation attributions expose them, plus a 54-person user study.
- [Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review](/papers/2504.10527): semi-systematic review of XAI in food quality; counts methods by data and explanation type and names formal evaluation as a gap; no evaluation scores.
- [Neurosymbolic AI: The 3rd Wave](/papers/2012.05876): a position paper that argues for fidelity as the test of an explanation and for rule confidence values; no experiments.
- [Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges](/papers/1803.07517): qualitative criteria (fidelity, interpretability, generalizability, explanatory power) and a three-class taxonomy, with no measurements.
- [The role of explainability in creating trustworthy artificial intelligence for health care](/papers/2007.15911): narrative survey that splits explanation into soundness and completeness and maps which properties have a metric for each explanation type; no experiments.
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](/papers/2103.15679): evaluates by positive and negative perturbation AUC and DETR segmentation AP, all author-reported.
- [Explaining a Series of Models by Propagating Shapley Values](/papers/2105.00108): evaluates by ablation curves and runtime against model-agnostic methods; notes that some axioms, such as implementation invariance, are not guaranteed.
- [Counterfactuals and Causability in Explainable Artificial Intelligence](/papers/2103.04244): lists five counterfactual properties (proximity, plausibility, sparsity, diversity, feasibility) without formulas or experiments.
- [Post-hoc Interpretability for Neural NLP: A Survey](/papers/2108.04840): sorts post-hoc NLP explanation methods by how they communicate, with validation notes per family; no experiments.
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](/papers/2110.10790): survey chapter that maps explanation techniques to user questions and reviews HCI pitfalls; no experiments.
- [Explainability for Large Language Models: A Survey](/papers/2309.01029): survey of LLM explainability; its evaluation section covers plausibility and faithfulness metrics (comprehensiveness, sufficiency, counterfactual simulatability); no new experiments.
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](/papers/2202.01602): six metrics for disagreement between feature-attribution explanations, plus tabular, text and image experiments and a user study; author-reported.
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](/papers/2305.04388): prompt-change test for chain-of-thought explanations on BBH and BBQ; black-box and author-reported.