A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

In plain words

Hospitals increasingly use AI tools that suggest diagnoses or treatments, but clinicians are often hesitant to act on a recommendation they cannot question. So these tools are paired with explanations that show which inputs drove a score. This review asks a narrower question: when researchers test such explanations with the clinicians who would use them, what do they find? The authors searched three literature databases and kept 31 studies that did this kind of testing with real clinicians.

Their main answer is that explanations usually raise clinicians' trust and confidence, but they can add mental effort and do not always match how doctors reason. Most of the studies were small, and the most common explanation method was SHAP. The paper then proposes a development loop: involve the stakeholders early, agree on goals and measures, build prototypes, and evaluate and redesign in cycles.

This is a review, not a new model, so there is nothing to rebuild from it. Use it to see what has been measured, how, and where the evidence is thin.

TL;DR

• A PRISMA-guided review of 31 studies (published after 2017) in which clinicians evaluated explainable AI inside clinical decision support §3 fig. 2.
• Most studies use post hoc explanations, led by SHAP (12), with surveys as the usual evaluation method (16) § Aggregate Summary of Reviewed Studies.
• Reported perceptions are positive (20) or neutral (11), with no negative study. Samples are small: a median of about 16, and only 9 studies had 30 or more participants § Aggregate Summary of Reviewed Studies.
• The paper proposes a four-phase, stakeholder-centred development protocol. It is conceptual and no study tests it §4.3.

Problem

Clinicians remain hesitant to rely on AI recommendations, even when the models are accurate, and explanation tools are meant to close that gap §1. The paper says real-world evidence on how explanations affect clinical adoption is limited, and that evaluations of XAI in healthcare range from proxy simulations to small clinical trials §1. The authors also note that prior studies have largely emphasised algorithmic innovation over clinical usability evaluation §1, and that the lack of human-centred evaluation is identified as a main reason AI-based CDSS are not adopted §2.1.

Method

This is a systematic review, with a PRISMA flow §3 fig. 2. Records came from three databases (ACM Digital Library, Web of Science, PubMed). Keywords started broad (CDSS, XAI, healthcare, human-centred evaluation) and were refined iteratively; the full query list is in the authors' companion repository §3 app. A. A study was kept if it appeared in a top-tier venue (CORE A* or A, or an SJR Q1 journal), was published after 2017, and clearly named the XAI method, the human-centred evaluation method and the medical field §3. The 2017 cut-off follows the start of the DARPA XAI programme §3.

The paper sets out its own taxonomy of XAI methods along three axes §2.2:

• Type: intrinsic (ante hoc) or post hoc §2.2.1. Intrinsic examples are linear models, decision trees and attention weights. Post hoc examples are SHAP, LIME, Grad-CAM, DeepTaylor, integrated gradients, partial dependence, ICE, counterfactuals and surrogate models §2.2.1.
• Dependency: model-specific or model-agnostic §2.2.2. Agnostic methods treat the model as a black box and use only inputs and outputs. Specific methods depend on the architecture, for example CAM or Grad-CAM for CNNs.
• Scope: local (one prediction) or global (the whole dataset, usually the test set) §2.2.3.

Human-centred evaluations are grouped in three levels §2.4: application-grounded (real tasks with real experts), human-grounded (simplified tasks with lay people) and proxy (no humans). The authors judge application-grounded evaluation preferable in medicine, and they describe the trade-off between fidelity and cost §2.4. Common human methods are think-aloud studies, interviews, focus groups and surveys §2.4.

Each included study is placed in one of three groups: intrinsic-only models §3.1 (2 studies), post hoc methods §3.2 (15) and deep learning §3.3 (14). Studies are also grouped by medical field. The authors state that ethics approval was not required, because the work reviews published literature §3.

Thematic analysis then groups the findings into four themes §4.1 tab. 2:

1. Improvements in risk assessment, with sub-themes on novice support, reduced false learning and better clinical planning.
2. A preference for established clinical practice, with a sub-theme that XAI is often not actionable.
3. Differences by clinical role and seniority.
4. Friction between stakeholders during development.

Section 4.2 explains the themes with a socio-technical gap framing §1 §4.2. Two dimensions are used, following Liao et al. §4.2: T1, human requirement realism (who the stakeholders are and what they need), and T2, context realism (whether an evaluation measures those needs in the actual use case). The authors map each theme onto one or both dimensions §4.2.

The protocol in section 4.3 has four phases: (1) identify stakeholders and their requirements; (2) refine the clinical goals and choose metrics; (3) build prototypes, moving from low-fidelity to high-fidelity; (4) evaluate and iterate. Phases 3 and 4 are meant to repeat in a cycle §4.3.

The paper has no equations and no algorithm blocks. The pseudocode below is our reading of the review procedure and the protocol. Each step cites the text it comes from.

# Review selection procedure  [§3] [fig. 2]
# Input:  keyword queries, three databases
# Output: 31 included studies

def select_studies(queries, databases):
    # 1. Retrieve records from the three databases, plus 12 records the
    #    authors already knew  [§3] [fig. 2]
    records = []
    for db in databases:
        records += search(db, queries)
    records += known_by_authors()
    records = dedupe(records)  # 3,039 after duplicates  [fig. 2]
    # 2. Keep only top-tier venues (CORE A*/A or SJR Q1)  [§3] [fig. 2]
    venue_ok = [r for r in records if top_venue(r)]
    # 3. Title and abstract screening, then published after 2017  [§3]
    screened = [r for r in venue_ok if year(r) > 2017 and relevant(r)]
    # 4. Full-text check: XAI method, HCE method and medical field named
    #    explicitly  [§3]
    included = [r for r in screened if names_all_three(r)]
    return included  # 31 studies in the review  [§3] [fig. 2]

# Stakeholder-centred development loop  [§4.3]
# Input:  stakeholders, clinical problem
# Output: a prototype and its evaluation record

def develop_cdss(stakeholders, problem):
    # 1. Identify stakeholders and collect their requirements  [§4.3] (1)
    requirements = collect_requirements(stakeholders)
    # 2. Refine clinical goals and choose metrics; the paper cites a
    #    reference set of 14 XAI metrics  [§4.3] (2)
    goals = refine_goals(problem, requirements)
    metrics = choose_metrics(goals)
    # 3. Build prototypes, from low to high fidelity  [§4.3] (3)
    for fidelity in ["low", "retrospective", "high"]:
        prototype = build_prototype(goals, fidelity)
        # 4. Evaluate technical, human-centred and ethical-legal aspects
        #    [§4.3] (4)
        results = evaluate(prototype, metrics)
        # 5. Redesign against predefined checkpoints; values not stated
        #    [§4.3] (4)
        if not passes_checkpoints(results):
            goals = redesign(goals, results)
    return prototype, results

Note: the paper describes the phases in order and calls phases 3 and 4 a repeating cycle. The loop above is one way to read that; the paper does not give code for it.

Hyperparameters

The paper has no model hyperparameters. The table lists the review's own parameters.

Results

Author-reported. This is a systematic review, so there are no new experiments and no baselines. The figures below are the review's own tallies over the 31 studies. Study-level details are in tab. 1 (it has no sample-size column) and in the per-section text.

Examples the paper reports, one per setting:

• Intrinsic model, risk tool for postoperative complications: think-aloud sessions with physicians on 8 to 10 cases in each of two case studies. The authors report a net reclassification gain of +12% and +16% over no XAI §3.1.
• Post hoc, perioperative blood ordering: 63 clinicians in a repeated-measures design. The format with SHAP plus clinical explanation improved trust, satisfaction and acceptance over the other two formats §3.2.
• Deep learning, radiology: 12 radiographers preferred simple binary feedback over heatmaps, which often caused disagreement §3.3.
• Deep learning, radiology: a counterfactual explanation improved residents' understanding more than saliency maps did §3.3.
• Deep learning, dermatology: 116 participants; explanations did not significantly change accuracy but did raise confidence and trust §3.3.

Paper vs code

No official code.

The paper's appendix points to a companion GitHub repository, iamalegambetti/CDSS-XAI-HCE-Survey, for the search queries app. A. Its listing shows a query file, a README that points to it, and CSV data files. It has no source code and no LICENSE file. We did not clone it, because it is a search-query repository and not an implementation. It is not recorded as code.

Pitfalls

• The abstract's headline share is not in the body. The abstract says over 80% of studies use post hoc, model-agnostic methods. The body never states this figure. Counting Table 1's labels by hand gives 28 of 31 post hoc (90%) and 20 of 31 post hoc and model-agnostic (65%). The 80% figure only holds for post hoc alone tab. 1 § Aggregate Summary of Reviewed Studies.
• "Frequently increases cognitive load" is not counted. Cognitive load is raised in individual studies (for example by the less-experienced-clinician finding in §3.3 and the effort finding in §4.1), but the review gives no count of how many studies report it.
• Counts disagree between sections. The aggregate section counts 4 CAM-family studies § Aggregate Summary of Reviewed Studies. The deep-learning summary counts 5 occurrences §3.3, and it cites GradCAM for ihongbe, which Table 1 labels as LIME with others. The deep-learning summary also says 11 studies were positive but cites 9 §3.3. Table 1 gives 9 positive studies in that group, and it labels neves as neutral, while the text lists neves as positive tab. 1 §3.3. The overall 20 / 11 split matches Table 1.
• Sample-size sentence is ambiguous. Section 4.1 says "only nine of these studies" had reasonable samples (over 30). "These" refers to the 12 early-feedback studies §4.1. The aggregate section gives 9 across all 31 § Aggregate Summary of Reviewed Studies. Read the first as a subset of the second only if the nine are the same studies, which the paper does not state.
• Per-study sample sizes are mostly missing from the body. The text gives numbers for about a dozen studies, and Table 1 has no sample-size column. The range "3 to 116" can be checked only against those mentions §3.1 §3.2 §3.3.
• Evaluation level is asserted, not tabulated. The discussion says application-grounded evaluations with domain experts were the only method used §4.1. Table 1 does not record the evaluation level, so this cannot be checked from the body.
• The protocol is not tested. Section 4.3 is a conceptual proposal, and no study applies it. Its checkpoints have no values §4.3 (4). The "field-agnostic" claim rests on 31 studies, mostly from critical care and radiology §4.3 § Aggregate Summary of Reviewed Studies.
• Inclusion favours top venues. Screening drops everything outside top venues before reading any title or abstract fig. 2. The review therefore reflects high-ranking venues, and a study with the same method in a lower venue is out of scope by design §3.
• Search is a snapshot. Results were updated to October 2025 app. A. The arXiv record is v4 from January 2026, so later papers are not covered.
• Figures. The paper has four figures: the intersection diagram fig. 1, the PRISMA flow fig. 2, papers per year fig. 3 and the distribution of methods and fields fig. 4. We describe them here only. None is reproduced.

Lineage

• Builds on: the socio-technical gap theory of Ackerman §1 §4.2; the three-level evaluation taxonomy cited as doshi2017towards §2.4; the T1 and T2 dimensions cited as liao2021human §4.2; a reference set of 14 XAI-in-CDSS metrics cited as Brankovic2025 §4.3 (2).
• Compared against: three prior XAI-in-healthcare surveys cited in the introduction, whose keys are antoniadi2021prediction, loh2022application and Kim2024. The paper says those emphasise algorithms or cross-domain frameworks, not evaluation with clinicians §1.
• Superseded by: not known from wiki content.