From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

In plain words

Explanation tools try to show why a machine learning model made a particular decision. This paper asks how anyone can tell whether such an explanation is any good. Many papers that propose an explanation method simply show a few convincing examples, and a third of the papers reviewed did nothing more.

The authors argue that "is this explanation good?" is not a yes-or-no question. They break it into twelve separate properties, such as whether the explanation is faithful to the model, whether it covers enough of the model's behaviour, whether it is short, and whether a user can make sense of it. They then collect automated tests that measure these properties and link each test to the properties it checks. The aim is to replace a single good-looking picture with a set of measurable checks.

TL;DR

• Systematic review of the evaluation practice of 361 included papers, of which 312 introduce an explanation method. Scope: 12 AI/ML conferences, 2014-2020 §3.1.
• Proposes twelve explanation-quality properties, "Co-12", grouped as content, presentation and user properties §3.3 tab. 2.
• Catalogues 29 automated, quantitative evaluation methods without user studies, clustered by Co-12 property §6 tab. 3 tab. 4.
• Relevant when you need to choose or design tests for an XAI method. It is a survey, not an algorithm, so there is nothing to train or run.

Problem

Explanations are usually judged by "face validity": individual, possibly cherry-picked examples that look reasonable §1 §2.1. The authors argue this cannot show that an explanation is faithful to the model, because a plausible explanation can still misrepresent the model's reasoning, and a flawed model can produce an explanation that looks wrong although it is faithful §2.1. There is no agreed evaluation strategy, and the field lacked a broad overview of quantitative methods §1.

Doshi-Velez and Kim's taxonomy separates application-grounded evaluation (domain experts, real task), human-grounded evaluation (lay people, simplified task) and functionally-grounded evaluation (no humans, computational proxies) §2.2. This paper covers only the third group in its catalogue, and points to a supplementary summary for user studies §6.

Scope assumption: the review judges the explanation method and its explanations, not the predictive model's accuracy, run time or construction overhead §3.3.

Method

The paper has two parts. First, it defines the Co-12 properties from the conceptual literature §3.3. Second, it reviews papers in a structured way and records (a) how each paper is categorised, (b) whether its evaluation is only anecdotal, and (c) which quantitative evaluation methods it uses, mapped to the Co-12 properties §3.1-§3.3 §6.

Terminology: a predictive model is written f, an explanation method e. A local explanation is e(f(x)) for one prediction; a global explanation is e(f). For self-explaining models e = f §1.1.

Co-12 properties tab. 2. Content: Correctness (faithful to the model); Completeness (how much of the model's behaviour is described); Consistency (identical inputs give identical explanations); Continuity (small input changes give small explanation changes); Contrastivity (explanations differ between targets or events and answer "why not?"); Covariate complexity (features are human-understandable and their interactions simple). Presentation: Compactness (explanation is small); Composition (format and organisation); Confidence (certainty information is present and accurate). User: Context (relevant to the user's needs); Coherence (agrees with prior knowledge); Controllability (user can influence the explanation). The paper distinguishes Correctness (faithfulness, §3.3) from Coherence (plausibility) §2.3.

Paper selection §3.1 [supp.]. DBLP title search for "explain*", "explanat*" or "interpret*" in twelve venues and 2014-2020; companion papers removed; papers kept if they introduce, apply or evaluate a method for explaining a machine learning model; 312 of the 361 introduce a method.

Categorisation §3.2 fig. 2 tab. 1. Each paper is labelled along six dimensions: type of input data, type of explanation, type of explanation problem (four types, after Guidotti et al.), type of model being explained, type of task, and type of explanation method (post-hoc, built into the model, or supervised explanation training). Explanation types use 14 categories tab. 1. Only the main text is reviewed; appendices are excluded §3.2.

Review protocol for evaluation §3.3 fig. 3. The pseudocode below restates the protocol as the paper describes it. It is a summary for implementers, not code from the authors.

# Review protocol for one paper  [§3.1] [§3.2] [§3.3] [fig. 3]
# Input:  paper (main text only)  [§3.2]
# Output: categories, evaluation records, Co-12 labels or "anecdotal"

def review_paper(paper):
    # Keep only original work that explains a machine learning model  [§3.1]
    if not explains_ml_model(paper):
        return None
    # Six categorisation dimensions  [§3.2] [fig. 2]
    categories = categorise_six_dimensions(paper)
    # Only anecdotal examples: record it and stop  [§3.3] [fig. 3]
    if shows_only_anecdotal_evidence(paper):
        return categories, "anecdotal"
    records = []
    for evaluation in paper.evaluations:
        # Drop metrics of the predictor, run time or build cost  [§3.3]
        if evaluates_predictor_or_runtime(evaluation):
            continue
        # User studies are tagged application- or human-grounded  [§3.3]
        # Other evaluations get a short description and a qualitative or
        # quantitative label, plus the Co-12 properties they check  [§3.3]
        records.append(describe_and_label(evaluation))
    return categories, records

Catalogue of quantitative evaluation methods §6 tab. 3 tab. 4. The paper clusters the methods by Co-12 property (subsections §6.1-§6.12). Each row gives the primary property of the method's subsection. Tab. 4 also lists secondary properties; read those as "can be measured with this method" §6, not as "was measured in every paper that uses it".

For each method, the card points to the subsection of §6 for variants and to the original papers in tab. 3 for implementation details. The paper does not restate formulas for the metrics §6.

Hyperparameters

This is a survey, so there are no training hyperparameters. The review settings are listed below, because they decide which papers are counted.

Results

Author-reported. The paper is a review, so there are no model benchmarks. The numbers describe the 312 papers that introduce an explanation method, from 2014-2020, at the twelve venues §5. Baselines are prior reviews, as the paper quotes them §5.

The abstract's headline figures ("1 in 3" anecdotal, "1 in 5" with users) match the 33% and 22% above §1 §5.

Paper vs code

No official code. The paper's categorisation is published as an interactive labelled dataset of the 312 papers at a project website §5 §7. It is a dataset rather than code, and this card does not use it or its contents.

Pitfalls

• Coverage counts are not measurement counts. Tab. 4 says which Co-12 properties a method can measure. It does not say which were measured in each paper §6.
• Title-only keyword search. The query misses terms such as intelligibility, transparency, and named methods like feature attribution. The authors chose precision over recall, given time constraints [supp.]. Results describe titles that contain these words, not all XAI work.
• Snapshot, not current. The search dates from May 2021 and covers 2014-2020. Later work is not counted [supp.].
• Venue choice may bias the statistics. The authors say their higher quantitative-evaluation share may partly reflect the venues chosen §5.
• Manual labels with an asterisk. One category in the categorisation figure was added manually after review and may be incomplete (high precision, lower recall) fig. 5 caption.
• Agreement is measured only on a hard subset. Cohen's κ = 0.625 comes from 81 papers that were double-checked, often because a reviewer was unsure. The authors say this may understate agreement [supp.]. Agreement on the Co-12 mapping is not reported [supp.].
• Co-12 categories overlap. Coherence (plausibility) and Correctness (faithfulness) can conflict, and Completeness and Compactness can trade off. The authors say the trade-off must be justified per application §2.3 §7 §1.
• Controlled synthetic data needs a checked model. Tab. 3 warns that the assumption that the model follows a known reasoning must be checked, for example by reporting near-perfect accuracy.
• Anecdotal-only does not mean wrong. The paper treats it as insufficient for claiming quality, not as evidence that the explanation fails §1 §2.1.
• User-study methods are only in the supplementary material. The main text covers functionally-grounded methods only §6. Agents implementing human studies should read the supplement before relying on this card.

Lineage

• Builds on: the evaluation taxonomy of Doshi-Velez and Kim (application-, human- and functionally-grounded) §2.2; the explanation-problem taxonomy of Guidotti et al. §3.2. Neither is in this wiki.
• Compared against: Adadi and Berrada (2018) and Nunes and Jannach (2017), whose evaluation shares the paper quotes §5. Zhou et al. (2021) and Vilone and Longo (2021) are named as the closest related work §1. None is in this wiki.
• Superseded by: not stated in this wiki.