Explanation evaluation (judging whether an XAI explanation is good)

Overview

An explanation method (for example, a feature-importance or heatmap method) produces an explanation for one prediction, or for the model as a whole. The open question is how to tell whether that explanation is good. Many papers judge it with a few hand-picked examples that look plausible. The review treats this "face validity" check as insufficient, because a plausible explanation can still misrepresent the model, and the paper's reviewed set reports that 33% of its 312 introducing papers use only such evidence, by the review's own count 2201.08164 §1 2201.08164 §5.

The shared idea across the variants is to break "explanation quality" into separate, measurable properties, then pick a check for each property you care about. Notation: a predictive model is f, an explanation method is e. A local explanation is e(f(x)) for one input; a global one is e(f); a self-explaining model has e = f 2201.08164 §1.1.

Earlier work framed the same goal with a few qualitative criteria rather than a catalogue. Its main criterion is fidelity, meaning how well the explanation agrees with the model's input-output mapping 1803.07517 §5.1. Later work uses that criterion as a measurement, for example by removing features and watching the output 2103.15679 §5 2105.00108 §3.2.

A 2020 position paper on neurosymbolic AI also takes fidelity, meaning how closely an explanation matches the model rather than the data, as the test to apply. It argues that many local methods have low fidelity, citing an earlier measurement that the paper does not reproduce, and that rules extracted from a network should carry confidence values 2012.05876 §5. It runs no experiments 2012.05876 §1.

A 2021 survey of post-hoc methods for neural NLP sorts evaluation into three kinds: application-grounded (tested in the real deployment), human-grounded (tested with people or a stand-in for people) and functionally-grounded (tested for whether the explanation reflects the model). Most NLP work uses only the last two. The survey also argues that simulated users are often used in place of real ones, and that this inflates results 2108.04840 §4. A 2023 study takes a different route: it changes the prompt in a way that should not matter, then checks whether the written reasoning mentions the change 2305.04388 §2. A 2019 paper on attention weights compares them with a uniform baseline, with seed-to-seed variation and with an adversary that builds different attention for a similar prediction 1908.04626 §3.2 1908.04626 §4.

Variants

The variants fall into several families. They differ in what they measure and in how much evidence they need from you.

Co-12 properties and the automated checks

The review defines twelve explanation-quality properties, in three groups 2201.08164 §3.3 2201.08164 tab. 2:

• Content: Correctness (faithful to the model), Completeness (how much of the model's behaviour is described), Consistency (identical inputs give identical explanations), Continuity (small input changes give small explanation changes), Contrastivity (explanations differ across targets and can answer "why not?"), Covariate complexity (features are understandable and their interactions simple).
• Presentation: Compactness (the explanation is small), Composition (format and organisation), Confidence (certainty information is present and accurate).
• User: Context (relevant to the user's needs), Coherence (agrees with prior knowledge), Controllability (the user can influence the explanation).

Correctness (faithfulness) and Coherence (plausibility) are kept apart on purpose 2201.08164 §2.3. The card for this paper is From Anecdotal Evidence to Quantitative Evaluation Methods.

The review catalogues 29 automated methods, each with a primary property, the explanation types it applies to, and what it checks 2201.08164 §6 2201.08164 tab. 3. Grouped by property:

The review says the table in 2201.08164 tab. 4 lists which properties a method can measure. It does not say which were measured in each paper 2201.08164 §6.

Qualitative criteria and taxonomy

Before the catalogue, one paper set out four criteria and left them unmeasured: fidelity, interpretability (split into clarity and parsimony), generalizability across architectures, and explanatory power, meaning how many user questions a method can answer 1803.07517 §5.1. The paper calls fidelity the most important property. It gives no measurement procedure, so the ratings are labels, not scores 1803.07517 §5.1 1803.07517 §5.2. The same paper sorts methods into rule extraction, attribution and intrinsic classes and rates each against those criteria 1803.07517 §5.2. The card is Explanation Methods in Deep Learning.

The neurosymbolic position paper gives a checklist of the same kind, without ratings. Its explanation argument is that a rule set can show a proof trace, which a raw activation pattern cannot, and that fidelity is the property to judge explanations by 2012.05876 §5. It is a position, not a test, so it adds no check to the catalogue. The card is Neurosymbolic AI: The 3rd Wave.

Two more sources give a checklist and a method for designing explanations, not a test. A human-centred survey chapter sorts explanation techniques by more than 50 user questions in nine categories and pairs each category with candidate methods. The authors present that pairing as a suggestion, not a validated result 2110.10790 §3 2110.10790 tab. 1. It also lists pitfalls: an explanation may not serve the user's real goal, and people often skim them rather than use them 2110.10790 §4. The NLP survey's per-family notes say that validation is often weak or missing, especially for human-centred and faithfulness checks 2108.04840 §4 2108.04840 §17. The cards are Human-Centered Explainable AI (XAI) and Post-hoc Interpretability for Neural NLP: A Survey.

A 2025 semi-systematic review of explainable AI in food quality reaches a similar point from the application side. It counts how studies explain their models, and finds that most use visual heatmap methods on pictures, with Grad-CAM-family methods preferred over LRP, and it lists formal evaluation of explanations among its open gaps 2504.10527 §9 2504.10527 tab. 2 2504.10527 §1. Its counts describe what was used, not how well it worked, so it gives no check to apply. The card is Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review.

Which properties have a metric, by explanation type

A 2020 survey of explainable AI for health care maps each explanation type to the properties it can be checked for. Each cell is one of three marks: satisfied by definition, measurable with a generic metric, or no known metric 2007.15911 tab. 2 2007.15911 §5. The map is the authors' classification from their literature review. It shows where to look for a metric, not what any method scores.

• Surrogate fidelity. For model-based post-hoc explanations, soundness has a metric: the share of predictions on which the surrogate and the model agree 2007.15911 §5.1. A surrogate's completeness is marked as satisfied by construction, since it can always compute the model's output, so no metric is applied to it 2007.15911 §5.1 2007.15911 tab. 2.
• Attribution soundness and completeness. Attribution soundness has three kinds of test: artificial ground truth, perturbation tests, and axiomatic tests such as sensitivity and implementation invariance 2007.15911 §5.2. Attribution completeness, meaning whether the scores account for the full change in output, has no metric in the map 2007.15911 tab. 2.
• Parsimony and clarity. Parsimony of model-based explanations can be counted: features, rules or tree splits, tree depth, operation counts, and a feature-interaction complexity measure. Global clarity of rule-based explanations has an unambiguity measure built on rule overlap and coverage. Local clarity is marked as no metric for model-based explanations, but as a generic metric for attribution-based ones 2007.15911 §5.1 2007.15911 tab. 2.
• Example-based explanations. The authors know of no evaluation method for them, for either scope 2007.15911 §5.3 2007.15911 tab. 2. A check for counterfactuals therefore has to come from the counterfactual family above, not from this map.

The health-care survey is the only source here that gives this full map. Its numbers are not checked by experiment. The card is The role of explainability in creating trustworthy artificial intelligence for health care.

Perturbation and ablation curves in method papers

Two method papers use their own tests rather than the catalogue.

• Positive and negative perturbation AUC. Remove image or text tokens in order of their attribution and track accuracy. A larger area is better for negative perturbation (removing the most relevant tokens hurts), and a smaller one is better for positive perturbation (keeping only the relevant tokens preserves accuracy). This is the same deletion idea as checks 5 and 6 in the catalogue. For ViT-Base on ImageNet, the card reports negative-perturbation AUC of 54.61 for the proposed method against 54.16 for the best baseline 2103.15679 tab. 2. The method is described in Generic Attention-model Explainability.
• Ablation curves for pipelines. Replace the least important features with reference values and track the model's score. The DeepSHAP paper reports that its ablation scores are comparable to IME on MNIST and better than KernelSHAP and LIME there 2105.00108 §4.3. It also reports that a k-means reference set "substantially outperforms" a general-population reference, in a qualitative comparison without a table 2105.00108 §3.2 2105.00108 fig. 3. The card is Explaining a Series of Models by Propagating Shapley Values.

The 2103.15679 card also reports a DETR weakly supervised segmentation test on COCO validation, with AP 13.1 against 7.2 for the best baseline 2103.15679 tab. 1. This is an overlap-with-annotation test, which the catalogue files under alignment with domain knowledge (27).

Planted-bug tests with known ground truth

Instead of scoring an explanation against the model, this family plants a known bug in the model or its data and checks whether the explanation shows it. The bugs are a spurious background, mislabelled training examples, re-initialised top layers and out-of-distribution inputs 2011.05429 §1 2011.05429 §2.1. The authors compare gradient-family, path, surrogate and modified back-propagation methods on a birds-vs-dogs CNN and a VGG-16 model 2011.05429 §2.2.

• Spurious background. Against a relaxed ground-truth mask, the attribution SSIM ranges from 0.78 (KernelSHAP) to 0.98 (VGrad) 2011.05429 tab. 2. The authors read this as the attributions picking up the background shortcut 2011.05429 §3.1.
• Mislabelled training examples. SSIM between attributions from the correct and the flipped-label model ranges from 0.73 to 0.99 across the methods. The authors conclude that these methods are unlikely to identify mislabelled examples 2011.05429 §3.2 2011.05429 fig. 6.
• Re-initialised top layers. Guided Backprop, DConvNet, PatternNet, PatternAttribution and LRP-SPAF stay visually and quantitatively unchanged as higher layers are re-initialised, so they cannot show that kind of defect. The paper reports this qualitatively, with no numeric table 2011.05429 §4 2011.05429 fig. 7. This is the same randomisation idea as checks 1 and 2 in the catalogue, but on planted bugs.

The same paper runs a 54-person study. With labels present, participants rejected the random-label and re-initialised models mostly on predicted labels, and were unsure about the spurious-background model 2011.05429 §6 2011.05429 app. H. Its authors say the 10-way breed task may make people lean on labels more than on attributions 2011.05429 §6. The card is Debugging Tests for Model Explanations.

Counterfactual properties

For counterfactual explanations, a review lists five properties of a good counterfactual: proximity (close to the original input), plausibility (does not change features that cannot change), sparsity (few changed features), diversity and feasibility 2103.04244 §4.2. The paper defines each in words and gives no formula or threshold for any of them 2103.04244 §4.2. Distance is usually an L_p norm; the review notes that L1 favours sparse changes 2103.04244 §4.1. The card is Counterfactuals and Causability in Explainable AI. For counterfactuals, the properties are the design targets, and the numbers still have to come from the original algorithm's paper.

Disagreement between explanation methods

This family asks how often two explainers disagree on the same prediction. The six metrics are agreement scores, so low values mean strong disagreement. Four compare the top-k features by absolute attribution: feature agreement (same features), rank agreement (same order), sign agreement and signed-rank agreement. Two compare a feature set the user chooses: rank correlation and pairwise rank agreement 2202.01602 §3.2 2202.01602 app. A. On the paper's models, LIME and KernelSHAP on image superpixels have high rank correlation (0.8977), while most AG News text pairs have rank and signed-rank agreement under 0.1 2202.01602 tab. 2 2202.01602 §4.3.2. Gradient methods at pixel level disagree strongly: rank correlation is 0.001 between Integrated Gradients and SmoothGrad 2202.01602 §4.3.3. These are author-reported results on a few models, so treat them as a reference point rather than a benchmark. Feature agreement is trivially 1 when k equals the number of features 2202.01602 app. C.1.

In a 25-person study, practitioners who met disagreements mostly settled them with ad hoc heuristics, not with a test 2202.01602 §5.2.4. Agreement is a coherence signal, not a check of faithfulness 2201.08164 §2.3. The card is The Disagreement Problem in Explainable Machine Learning.

Baseline and adversarial tests for attention weights

These tests apply to attention weights in a model that has attention built in. The paper works on single-layer bidirectional LSTM text classifiers and on binary tasks only 1908.04626 §3.1. It runs four tests, each a baseline or a stress test rather than a score against annotations 1908.04626 §5.

• Uniform-attention baseline. Freeze the attention to uniform weights and measure the drop in F1. On Diabetes the paper reports 0.775 for the reproduced base model against 0.706 for uniform attention, and on IMDb 0.902 against 0.879 1908.04626 tab. 2. The paper drops AG News and 20 Newsgroups from later tests, because uniform attention matched the base model there 1908.04626 §3.2.
• Seed-variance baseline. Train several models that differ only in their initialisation seed, and measure how far their attention moves. The paper uses eight seeds and reports that the spread depends on the dataset 1908.04626 §3.3 1908.04626 fig. 3. Divergence from the base model is read against this spread.
• Frozen-weight MLP guide. Impose a set of attention weights on a non-contextual per-token MLP and compare F1. On Diabetes the paper reports 0.753 with base LSTM attention, 0.699 with a trained MLP's attention and 0.503 with the adversary's attention. On Anemia the adversary scores 0.932 against 0.931 for the base attention. The paper attributes this to the label skew and to sparse indicative tokens in positive examples 1908.04626 tab. 3 1908.04626 §4.
• Consistent adversary. Train a second model whose predictions stay close to the base model while its attention is pushed away from the base attention 1908.04626 §4. The best adversary per dataset, among those with mean JSD above 0.4, has JSD of 0.405 to 0.514 on the four datasets kept for this test 1908.04626 tab. 4.

The paper's reading is that alternative attention with similar predictions does exist for some binary tasks, but it guides a simple model poorly, so attention is still useful as an explanation under the paper's definitions 1908.04626 §5 1908.04626 tab. 3. The card on the paper is Attention is not not Explanation.

Prediction-change tests for chain-of-thought explanations

For a language model that writes its reasoning before answering, the check is to add a feature that should not affect the answer and see whether the answer moves and the reasoning mentions the feature. In the Suggested Answer setting, a user line hints at a random option. On 13 BIG-Bench Hard tasks, GPT-3.5 zero-shot accuracy falls by 36.3 percentage points under that hint, and Claude 1.0 by 30.6 2305.04388 tab. 8. The hand-checked sample is small, but it shows that the explanations rarely mention the hint: 0 of 234 BBH explanations did 2305.04388 app. B. On a social-bias benchmark, the explanations for unfaithful answers mostly support the stereotype-aligned answer 2305.04388 tab. 5. The authors call the test necessary but not sufficient. A failure shows unfaithfulness; a pass does not prove faithfulness 2305.04388 §6. The experiments use legacy GPT-3.5 and Claude 1.0 endpoints, and the paper does not test other models 2305.04388 §7. The card is Language Models Don't Always Say What They Think.

Rationale removal and plausibility for LLM explanations

A 2023 survey of LLM explainability groups evaluation into two kinds for fine-tuned models. Plausibility asks whether the explanation matches human rationales, scored with IOU, precision, recall, F1 or AUPRC. Faithfulness asks whether the model's output depends on the rationale. For the second kind the survey gives two formulas 2309.01029 §5 2309.01029 §5.1:

• Comprehensiveness compares the model's score for class j on the full input with its score once the rationale is removed.
• Sufficiency compares the full-input score with the score on the rationale alone.

The survey does not define the model symbol in these formulas. The card reads it as the model's class probability, following the surrounding prose 2309.01029 §5.1. The card records no threshold and no preferred direction for either score, so check the survey before reading a result as good or bad. For prompted models the same survey points to counterfactual simulatability and to chain-of-thought faithfulness tests 2309.01029 §5.2, which the prediction-change test above is one example of. The survey lists the lack of ground-truth explanations as an open challenge, which limits the plausibility check 2309.01029 §6.1. The card is Explainability for Large Language Models: A Survey.

When to use which

Choose checks from what you can do to the model, the data and the people available. The sources give no ranking of methods across families.

Implementation sketch

The review gives no code for the checks, and this sketch is not from the authors. It shows the selection loop in a generic form. Each check's formula must be taken from the original paper in tab. 3 of the review.

# Choose the Co-12 properties your use case needs  [arxiv:2201.08164 §3.3]
properties = choose_properties(use_case)
# Map each property to the catalogue entries that test it  [arxiv:2201.08164 §6]
checks = [c for c in catalogue if c.primary_property in properties]
for check in checks:
    # Skip checks whose explanation types do not match yours  [arxiv:2201.08164 tab. 3]
    if not check.explanation_types & my_explanation_types:
        continue
    # Run the check on the model, the explainer and the data  [arxiv:2201.08164 §6]
    score = check.run(model, explainer, data)
    # Report the score with the caveat the review gives for that check  [arxiv:2201.08164 §6]
    report(check.name, score, caveat=check.caveat)

Contradictions

No paper reports numbers that contradict another on the same test. The differences below are about framing and scope.

• Fidelity as one criterion or as several. The 2018 survey treats fidelity as the most important single property 1803.07517 §5.1. The 2022 review separates Correctness (faithfulness) from Coherence (plausibility), and says their trade-off must be justified for each application 2201.08164 §2.3 2201.08164 §7. Both are kept on this page.
• What a model-agnostic explanation shows. The counterfactual review argues that model-agnostic methods capture correlations between features rather than the mechanism behind a prediction 2103.04244 §1.2. The DeepSHAP paper defines its attributions with an interventional expectation, which is written with a do-operator 2105.00108 eq. 8. It does not claim its attributions reflect a causal mechanism. The two papers do not test each other's claims, and this page does not resolve the question.
• Low fidelity of local methods. The 2020 position paper says LIME-style local methods have low fidelity, citing an earlier measurement 2012.05876 §5. The 2018 and 2022 sources do not test that claim. The DeepSHAP paper reports MNIST ablation scores in which LIME and KernelSHAP fall below DeepSHAP 2105.00108 §4.3. That is a comparison within one test, not a check of the earlier measurement, and neither result is re-run here.
• Mixed numbers within one paper. The Transformer-explainability card reports that its method beats the baselines overall 2103.15679 tab. 1. On positive perturbation for ViT, Transformer attribution has the lower AUC (17.03 against 17.32 for the proposed method, for the predicted class), and the paper calls the gap mild 2103.15679 tab. 2 2103.15679 §5.
• Fidelity and completeness as terms. The 2020 health-care survey splits fidelity into completeness (the explanation can compute the output) and soundness (the explanation is true to the model) 2007.15911 §2.2. The 2022 review uses Correctness for faithfulness and lists Completeness as a separate property, about how much of the model's behaviour is described 2201.08164 §3.3. The survey's evaluation section also uses "completeness" for a different, sensitivity-style property 2007.15911 §5.2. The same word covers three definitions here, so this page keeps them apart and does not map one onto another.
• Whether attributions expose a spurious background. The planted-bug paper reports high SSIM between attributions and the background-only mask, and reads it as the attributions showing the shortcut 2011.05429 tab. 2. Its user study does not show the same result: a majority said the attribution did not show the part they expected, yet they did not reject the spurious model convincingly 2011.05429 §3.1 2011.05429 app. H. Both results come from one paper and one set of planted bugs, so this page keeps both and does not resolve the question.
• Which faithfulness metric to trust. The LLM survey notes that the best-ranked explanation under one faithfulness metric can be the worst under another 2309.01029 §5.1. It does not resolve this, so this page does not pick a single faithfulness metric.
• Whether attention is an explanation. Jain and Wallace (no card here) argue that attention is not explanation, because different attention can give similar predictions. The 2019 paper answers that the claim depends on what "explanation" means, and that adversarial attention exists for some binary tasks 1908.04626 §2 1908.04626 tab. 4. It also reports that adversarial attention guides an MLP poorly on most datasets 1908.04626 tab. 3. Neither side is tested beyond that paper's setup, so this page keeps both.
• Simulated users as a proxy. The 2022 review lists a simulated user study as a cheap stand-in for usefulness 2201.08164 §6.10. The 2021 NLP survey argues that simulated users inflate results, because explaining well to people needs interdisciplinary input 2108.04840 §4. Neither source tests the other's claim, and this page keeps both.

Papers

• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI: defines the Co-12 properties and catalogues 29 automated evaluation methods, each mapped to the properties it checks.
• Attention is not not Explanation: four tests of attention weights in LSTM text classifiers (uniform baseline, seed variance, frozen-weight MLP guide, consistent adversary); binary tasks only; author-reported.
• Debugging Tests for Model Explanations: four planted bugs with known ground truth; tests whether gradient-family, path, surrogate and modified back-propagation attributions expose them, plus a 54-person user study.
• Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review: semi-systematic review of XAI in food quality; counts methods by data and explanation type and names formal evaluation as a gap; no evaluation scores.
• Neurosymbolic AI: The 3rd Wave: a position paper that argues for fidelity as the test of an explanation and for rule confidence values; no experiments.
• Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges: qualitative criteria (fidelity, interpretability, generalizability, explanatory power) and a three-class taxonomy, with no measurements.
• The role of explainability in creating trustworthy artificial intelligence for health care: narrative survey that splits explanation into soundness and completeness and maps which properties have a metric for each explanation type; no experiments.
• Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers: evaluates by positive and negative perturbation AUC and DETR segmentation AP, all author-reported.
• Explaining a Series of Models by Propagating Shapley Values: evaluates by ablation curves and runtime against model-agnostic methods; notes that some axioms, such as implementation invariance, are not guaranteed.
• Counterfactuals and Causability in Explainable Artificial Intelligence: lists five counterfactual properties (proximity, plausibility, sparsity, diversity, feasibility) without formulas or experiments.
• Post-hoc Interpretability for Neural NLP: A Survey: sorts post-hoc NLP explanation methods by how they communicate, with validation notes per family; no experiments.
• Human-Centered Explainable AI (XAI): From Algorithms to User Experiences: survey chapter that maps explanation techniques to user questions and reviews HCI pitfalls; no experiments.
• Explainability for Large Language Models: A Survey: survey of LLM explainability; its evaluation section covers plausibility and faithfulness metrics (comprehensiveness, sufficiency, counterfactual simulatability); no new experiments.
• The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective: six metrics for disagreement between feature-attribution explanations, plus tabular, text and image experiments and a user study; author-reported.
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting: prompt-change test for chain-of-thought explanations on BBH and BBQ; black-box and author-reported.