wikillm

Paper · 2023-11-18 · cs.AI, cs.HC, cs.LG

Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework

Systematic review of interpretable and explainable AI in healthcare: 2,241 records from PubMed, Scopus and Web of Science gave 74 included studies (10 reviews, 64 experimental). Proposes a three-level interpretability process (data pre-processing, model selection, post-processing), a 12-dimension quality checklist and a roadmap. No model, no code; counts are author-reported.

Explainable AIInterpretable machine learningExplanation evaluation (judging whether an XAI explanation is good)Feature attributionhealthcareexplainable-aiinterpretable-mlsystematic-reviewclinical-decision-support

arXivPDFMarkdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Doctors and patients are asked to trust AI tools that help with diagnosis and monitoring, but many of these tools cannot say why they gave an answer. This paper is a literature review. The authors searched three research databases, kept 74 studies that fit their criteria, and read them to see how researchers make healthcare AI easier to understand.

They split "making AI understandable" into three places where it can happen: when the data is prepared, when the model is chosen, and after the model has made a prediction. The paper observes that most existing work focuses on the last stage. It then proposes a checklist of questions for each stage, a list of twelve qualities a system should be checked for, and a rough set of steps for putting this into a hospital or clinic.

The paper does not train a model or test a new method. Its value is the map of the field and the checklists, which a team can use when planning a clinical AI tool.

TL;DR

  • A systematic review (PRISMA 2020 flow, PICO questions), not a new model: 2,241 records screened down to 74 included studies, 10 of them reviews and 64 experimental §2.3 fig. 6.
  • Proposes a three-level interpretability process: data pre-processing, interpretable model selection, and post-processing explanations §3 fig. 9 tab. 1.
  • Adds a 12-dimension quality checklist (accuracy, reliability, robustness, and so on) and a "Data to Health Center" roadmap with four boards §5 tab. 5 §6 fig. 14.
  • The paper reports no measured results for its own framework or checklist. Its counts and shares are author-reported and some are read from charts (see Pitfalls).

Problem

The authors state that clinicians and patients are reluctant to use AI tools whose outputs they cannot understand, and that this reluctance can lead to mistrust §1 §1.1. They argue that most existing reviews compare explanation methods or list guidelines, while none covers the whole chain from data preparation to post-model explanation for clinical decision support §1.6 §2.5.

The paper asks five research questions §1:

  1. Which levels of interpretability have been applied in health applications?
  2. Which healthcare applications have used interpretable AI tools?
  3. What are the current approaches to interpretability in health applications?
  4. How can robust interpretable AI be implemented and evaluated for specific end-users?
  5. What future trends and key challenges exist for explainable AI in medicine?

Method

This is a systematic review, not a new algorithm. The review protocol and the proposed framework are described below.

Review protocol.

  1. Preliminary keyword searches in PubMed, Scopus and Web of Science were used to scope the field §2.1.
  2. The authors derived nine conceptual frameworks, and wrote a Boolean search string for each, based on title and abstract and specific terms §2.1. The paper's list of frameworks names ten topics, so the count is inconsistent (see Pitfalls).
  3. The final searches were run on 16 June 2023 §2.1. Results span 2012 to 2023 fig. 6.
  4. Two authors screened records. Duplicates were removed in EndNote and screening was done in Covidence §2.1 §2.3. Titles and abstracts were screened first. When in doubt, a record was kept for full-text review. Disagreements were settled by consensus §2.1.
  5. Full texts were reviewed for eligibility §2.3.
  6. Each remaining study was appraised for quality and risk of bias with the MetaQAT tool. Studies with a substantial risk of bias were excluded §2.2. Grey literature was not included §2.2.
  7. Data were extracted with a standard template in Excel, and the flow was reported as a PRISMA 2020 diagram §2.2 §2.3 fig. 6.

Proposed three-level interpretability process. The paper splits interpretability into three levels, each with its own goals §3 tab. 1 fig. 9.

  • Data pre-processing interpretability (questions Pre1 to Pre8). Understand the data before modelling: exploratory analysis, dataset description and standardisation, summarising (for example prototype selection), augmentation (for example SMOTE and autoencoders), and feature engineering §3.1 §3.1.2–§3.1.6 fig. 10.
  • Interpretable model selection (Mod1 to Mod8). Choose a model that is either inherently interpretable, a hybrid of an interpretable and a complex model, a joint model that produces predictions and explanations together, or a model whose architecture is modified for interpretability §3.2 §3.2.1–§3.2.4 tab. 2.
  • Post-processing interpretability (Pos1 to Pos7). Explain a trained model after the fact: attribution, visualisation, example-based, game-theoretic (Shapley values and SHAP), knowledge extraction and neural methods §3.3 §3.3.1–§3.3.6 fig. 11.

Quality assessment and deployment. The paper lists twelve quality dimensions for an XAI system: accuracy, reliability, robustness, interpretability, usability, human-AI interaction, ethical considerations, responsiveness to feedback, compliance with regulatory standards, clinical validation and evidence base, transparency, and scalability. Each has a guiding question §5 tab. 5. A "Data to Health Center" (D2H) is proposed with four parts: a clinician-AI communication desk, a data-scientist help desk, a quality assessment board and an ethical review board. The flow runs from explainable data pre-processing, through explainable model processing and explainable post-processing, into quality assessment, and then to end-users §6 fig. 14. Feedback from end-users and stakeholders is meant to drive refinement in an iterative cycle §8.8 §8.10.

Pseudocode
# Three-level interpretability process with a quality cycle  [fig. 14] [§3]
# Input:  clinical task, end-user groups, raw data
# Output: predictions with explanations, quality records, feedback

def interpretable_clinical_pipeline(task, end_users, raw_data):
    # Pre-processing: understand the data first  [§3.1] [tab. 1]
    data_profile = explore_dataset(raw_data)  # EDA, missing samples  [§3.1.2]
    clean_data = prepare_data(raw_data, data_profile)  # [§3.1]
    features = engineer_features(clean_data, end_users)  # [§3.1.6]
    # Model selection: prefer a model a person can read  [§3.2] [tab. 2]
    model = choose_model(task, features, end_users)  # [§3.2.1]–[§3.2.4]
    # Post-processing: explain with a method that fits each end-user's question
    # [§3.3] [fig. 9]
    explanations = explain_predictions(model, features, end_users)
    # Quality assessment: score the twelve dimensions with end-users  [§5]
    # The paper gives no pass threshold for any dimension: not stated.
    quality = assess_quality(model, explanations, end_users)  # [tab. 5]
    # Feed end-user and stakeholder feedback into the next cycle  [§8.8]
    feedback = collect_feedback(end_users, explanations)
    return model, explanations, quality, feedback

The function names are generic labels chosen for this card. The paper gives no code and no algorithm for these steps.

Hyperparameters

No model is trained, so there are no model hyperparameters. The review protocol settings that matter for reproducing the search are:

SettingValueWhere
DatabasesPubMed, Scopus, Web of Science§2.1
Search date16 June 2023§2.1
Publication years covered2012 to 2023fig. 6
Conceptual search frameworks"nine" stated; ten topics listed§2.1
Reference manager and screening toolEndNote (duplicates); Covidence (screening)§2.1 §2.3
Quality and bias toolMetaQAT§2.2
Reporting guideline and question frameworkPRISMA 2020; PICO§2.1 §2.3
Reviewerstwo authors; disagreements by consensus§2.1 §2.4
Grey literaturenot included§2.2
Selection criteriasummarised in an appendix ("Appendix 2"), not in the text§2.1
Quality thresholdexcluded if "substantial risk of bias"; no numeric cut-off§2.2

Results

Author-reported counts. The setup: a systematic review of records from three databases, screened with PRISMA 2020 §2.3 fig. 6.

StageCountSource
Records identified (PubMed 658, Web of Science 582, Scopus 1001)2,241§2.3 fig. 6
Duplicates removed (EndNote 521, Covidence 34)555fig. 6
Records screened by title and abstract1,686fig. 6
Excluded at title and abstract958§2.3 fig. 6
Full texts reviewed728fig. 6
Full texts excluded (wrong setting 210; wrong outcomes 156; wrong intervention 116)482fig. 6
Assessed with MetaQAT246fig. 6
Excluded for risk of bias172§2.3 fig. 6
Included in review74 (10 reviews, 64 experimental)§2.3 fig. 6

Other reported results:

  • Among the 10 included reviews, the year split is 10% (2020), 20% (2021), 10% (2022) and 60% (2023) fig. 8b. This chart covers the reviews only, not all 74 studies.
  • Of the 64 experimental studies, about 80% concern AI tools in general, 15% wearable medical devices and 5% medical LLMs such as ChatGPT. These shares are read from the pie chart in fig. 15a.
  • Heart disease is the largest disease category in the chart of current XAI applications in AI-based medical devices fig. 12. Smaller slices are not individually labelled.
  • Post-processing explanation methods are the most common stage. The bar chart reads roughly 60% (pre-processing), 78% (processing) and 100% (post-hoc). The text says post-hoc methods are used in "over 100%" of cases and suggests some studies apply more than one fig. 15b §7.
  • The Discussion names SHAP and LIME as the most often used XAI methods in the included studies §7. No count is given in the text.
  • The paper's gap analysis: little XAI work on wearables and on LLMs §7 §8.11 tab. 3 tab. 4. Table 3 lists nine wearable examples and Table 4 lists three LLM examples tab. 3 tab. 4.

Paper vs code

No official code. The paper links no code repository or project page, and the metadata lists none. Searches: the paper's text and metadata for links (none); a GitHub repository search for the title's key terms (no results); a second search for "interpretable healthcare trust clinician", which returned one student project by an unrelated owner (not an author's repository, not used); and a GitHub search for "clinician-AI collaboration responsible", which returned no results. The paper is a review, so there is no implementation to compare with the text.

Pitfalls

  • Appendices are missing from the source. The text relies on Appendix 2 (selection criteria), Appendix 4 (survey table) and Appendix 5 (study-by-study summary). The arXiv version used here (v2) ends at reference 195 on page 59 with no appendices. The inclusion criteria and the per-study data are therefore not checkable from this source.
  • Broken cross-references. Several references render as "Error! Reference source not found." (for example in §3, §3.1, §3.2, §4.2, §5, §6 and §8.2). Figures are cited here by their captions, not by the broken references.
  • Two conflicting counts for experimental studies. §7 says "only around 10 experimental results" are presented. The abstract, §2.3, Figure 6 and Figure 15 all give 64 experimental studies. This card uses 64.
  • Search frameworks: nine or ten? §2.1 says nine conceptual frameworks but lists ten topics (Internet of medical things, telehealth, clinical decision support, remote care, sensors, wearable medical devices, medical LLMs, digital care twins, explainable AI, interpretable ML). Which list was used to build the strings is not clear.
  • Screening rule is permissive at the first stage. Records were excluded at title and abstract only when clearly irrelevant; any doubt sent them to full-text review §2.1. Full-text exclusion reasons are given only as counts fig. 6.
  • Quality appraisal has no published threshold. Studies were excluded for a "substantial" risk of bias under MetaQAT §2.2. The text names the four MetaQAT phases (relevancy, reliability, validity, applicability) §2.2 fig. 5, but gives no per-study scores, no item-level results and no cut-off.
  • Year chart covers reviews only. Figure 8b shows the publication years of the 10 reviews. It does not describe the year distribution of the 74 included studies.
  • Percentages in Figure 15 are read from charts and are not tabulated in the text. The post-hoc bar is at 100% but the text says "over 100%". Shares are not exclusive, so they should not be added up.
  • Quality framework is a checklist, not a tested method. Section 5 and Table 5 give twelve dimensions and a guiding question for each. The text gives no scoring scale, no acceptance thresholds and no results from applying them. Figure 14 shows "Quality Assessment" as a box without a procedure.
  • Pre-processing, model and post-processing questions (Table 1) are a checklist. Pre1–Pre8, Mod1–Mod8 and Pos1–Pos7 are proposed questions. They were not validated in the paper.
  • Opinion, not evidence, in several places. §8.1 argues that post-hoc explainers may not meet legal transparency goals and that users should treat such systems as black boxes unless better methods emerge. §2.4 asserts the explainability of reviewed outcomes is "verifiable" because most papers are in top venues. Treat these as the authors' positions.
  • Date limit. Searches stopped on 16 June 2023 §2.1. Later studies are not covered.
  • Version. This card follows arXiv v2 (updated 2024-04-10, as recorded in the metadata). Other versions may differ.

Lineage

  • Builds on: none. The paper positions itself against earlier reviews of XAI in healthcare, which it cites as [24], arXiv:2304.01543, and others that cover single application areas §2.5 fig. 7.
  • Compared against: reviews of XAI in oncology, heart disease, drug discovery and healthcare generally, named in §1.6 and §2.5, and compared in fig. 7.
  • Superseded by: not stated.

Source and license

Adapted from “Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework” by Elham Nasarian, Roohallah Alizadehsani, U. Rajendra Acharya, Kwok-Leung Tsui (arXiv:2311.11055), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.