Explainable AI

Overview

Explainable AI (XAI) aims to make a model understandable to the people who use it or are affected by it. The survey treats a good explanation as depending on the receiver's knowledge and goals, not only on the model 2110.10790 §1. It proposes starting from the questions users ask, and then choosing a technique that answers them 2110.10790 §3.

Two distinctions recur across the design space:

• Interpretable by design or post hoc. Directly interpretable models, such as decision trees, rule-based models and linear models, are contrasted with opaque models that are explained after training 2110.10790 §2.
• Scope of a post-hoc explanation. Post-hoc methods are grouped as global model explanations, outcome explanations and counterfactual inspection. Each can be feature-based or example-based 2110.10790 §2. Most post-hoc methods approximate the model, and the survey notes that such approximations can be unfaithful 2110.10790 §2.

A health-care survey groups post-hoc methods the same way, by what kind of explanation they produce: a simpler surrogate model, a score per input, or selected example instances. Each kind can be global or local. Its table also lists which properties of each kind have a quantitative metric 2007.15911 §4 2007.15911 tab. 1. It files LIME under surrogate models, where other surveys file it under attribution 2007.15911 §4.1.

Narrative reviews also name explainability as a remedy for opaque predictions. A drug-discovery review proposes it for transparent predictions, with no method or test of its own 2212.08104 §7. That review also says most references in its ChatGPT-drafted text were incorrect 2212.08104 §10, so cite it only for the claim it makes, not as evidence.

A third point runs through the cards: an explanation is an output like any other, and it can be wrong in ways that look plausible. The model-agnostic pitfalls review and the debugging-test study below both test this.

A philosophical essay uses explanation in a different way. It treats a good explanation of a system as an intentional agent as evidence that the system is one, while stating that the explanation is not a definition of agency 2609.15407 §4 2609.15407 §4, p. 7, fn. 6. It also names explainable AI as a reason to build systems whose behaviour is described in agential terms 2609.15407 §4.1, p. 10. The essay has no experiments and does not test any system, so it informs the framing and not the method. Card: Can AI systems have free will?.

Variants

Interpretable models

The explanation is the model's structure: a decision tree, a rule list or a linear model 2110.10790 §2. The pitfalls review argues that such models often perform about as well as complex ones, citing earlier benchmarks it did not re-check. It recommends starting with linear models or trees, and moving to generalised additive models and then to interactions only when the gain in performance justifies it 2007.04131 §4. No card here measures the accuracy cost of this choice on a task.

The interpretable-ML survey takes a firmer position. It argues that, for high-stakes decisions, an inherently interpretable model should be tried first, and that a readable model is often as accurate as a black box 2103.11251 Introduction. Its support is a set of examples and cited experiments, not a proof; the one worked comparison is a single benchmark dataset with one tree of each kind 2103.11251 fig. 3. It lists ten open problems for such models, among them optimal sparse trees, scoring systems, generalised additive models, prototype-based reasoning and disentangled neural networks 2103.11251 §1–§10. Its "Rashomon set" challenge asks how to choose among many models of near-equal loss, which matters when the choice of model is part of the explanation 2103.11251 §9. Card: Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges.

A health-care survey offers a decision guide that starts from how much explanation matters. When it matters a lot, the authors prefer an interpretable model. A post-hoc explanation is considered only when a complex model clearly outperforms, and the choice of explanation type then depends on whether fidelity or interpretability is the priority 2007.15911 §6.1 2007.15911 §6.2 2007.15911 §6.3 2007.15911 fig. 2. The authors say the guide needs testing in practice 2007.15911 §7.1. Card: The role of explainability in creating trustworthy artificial intelligence for health care.

One hybrid shows the interpretable route on time series. BORF classifies a series from counts of short symbolic words taken from many sliding windows, then applies a ridge classifier to those counts. Each count is one feature, so the coefficients and the per-count contributions can be read directly 2311.18029 §4.2 2311.18029 §4.5. Its contribution scores use SHAP on the linear model, which keeps the explanation deterministic across runs 2311.18029 §4.5 2311.18029 §5. Card: BORF.

Global effects and importance

Partial dependence (PDP), accumulated local effects (ALE) and permutation feature importance (PFI) explain the model across a dataset. Their pitfalls are the main subject of 2007.04131:

• Generalisation first. Interpreting an under- or overfitted model can mislead about the data. The review's example has a linear model that underfits, a random forest that overfits and a tuned SVM that fits best; it recommends judging fit only on held-out data 2007.04131 §3.
• Feature dependence. Perturbation-based methods can evaluate the model at combinations of feature values that never occur. An equidistant grid also creates unrealistic values for skewed features. The review prefers quantiles or sampled observed values, and ALE plots where dependence is strong 2007.04131 §5.1.
• Marginal or conditional. Marginal methods stay true to the model but not to the data; conditional variants do the reverse. The review asks the user to decide which question is being answered before choosing 2007.04131 §5.3.
• Uncertainty and multiple testing. One PDP can show an effect that is only noise, so the review repeats the estimate over resamples 2007.04131 §7. Testing 50 unimportant features at α = 0.05 gives a probability of about 0.923 of at least one "significant" result. The review suggests a false discovery rate control or a Bonferroni correction 2007.04131 §8.3.
• Causation. A non-zero PFI does not show that a feature causes the outcome. Causal reading needs explicit assumptions and an adjustment set 2007.04131 §9. See causal inference.

Post-hoc feature attribution

Methods such as LIME, SHAP and gradient-based saliency score each input feature for one prediction. Different methods often give different scores for the same prediction, especially on text and on pixel-level image attributions 2202.01602 §4.3. Card: The Disagreement Problem in Explainable Machine Learning. Details are on feature attribution.

A CNN-specific method, FullGrad, starts from an exact split of a ReLU network's output into input-gradient and bias-gradient terms 1905.00780 eq. 1. It post-processes each term and sums them into one saliency map [arxiv:1905.00780 eq. 2]. The paper proves that for piecewise-linear functions a saliency map cannot in general be both complete (its scores add up to the output) and weakly dependent on the input 1905.00780 prop. 1. The post-processing makes the final map approximate, so it no longer adds up to the output [arxiv:1905.00780 §4.2]. The authors report it best among the compared methods on ImageNet pixel perturbation and on a CIFAR100 remove-and-retrain test; both results appear only as plots 1905.00780 §5.1 1905.00780 §5.2 [arxiv:1905.00780 fig. 2]. On the paper's MNIST tables, integrated gradients has the lowest pixel-perturbation score at all three settings, and the signed variant of FullGrad scores highest on digit flipping. The paper concludes that no single method wins both tests 1905.00780 tab. 1 1905.00780 tab. 2. The released code applies the post-processing per layer rather than per channel, so its maps differ from the equation 1905.00780 eq. 2; the card records this. Card: Full-Gradient Representation for Neural Network Visualization.

On tabular data, a perspective on SHAP and LIME reports that the feature ranking depends on the classifier. Four classifiers fitted to one UK Biobank cohort agreed on the top three SHAP features, but the lower ranks varied 2305.02012 §2.1 fig. 2. SHAP assumes features are independent, and correlated features are a stated risk for its scores 2305.02012 §2.1. The paper's airline case adds two proxy measures for this: NMR, a stability score for a ranking, and MIP, a re-ordering that accounts for correlated features. The paper gives no formula for either; the authors' code does 2305.02012 §3 2305.02012 tab. 3. Its results come from one dataset per case, with no seeds or intervals 2305.02012 §2.1 §3. Card: A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME.

Two studies show where attributions help and where they do not:

• Planted bugs. In controlled tests, attribution maps picked up a spurious background that the model had learned to use 2011.05429 §3.1 2011.05429 tab. 2. They did not separate training images with flipped labels from correct ones 2011.05429 §3.2 2011.05429 fig. 6. For several modified back-propagation methods, the maps stayed the same after the top layers were re-initialised, so these methods cannot reveal that bug 2011.05429 §4 2011.05429 fig. 7.
• Text classifiers. A linear support vector classifier on TF-IDF features was explained with LIME word weights for one generated phishing-style email. The paper does not measure whether these weights are faithful or stable 2405.11619 p. 15. LIME settings such as the sample count are not stated 2405.11619 p. 10 2405.11619 p. 11.

Visual explanations dominate in practice. A review of food-quality studies found that most use heatmaps, mostly CAM-type methods, while spectral and time-series data are rarely explained. Few studies combined more than one XAI method 2504.10527 §9 2504.10527 fig. 8. Those authors report that most works choose Grad-CAM over LRP although their costs are similar, and cite human studies for the choice 2504.10527 §3.2 2504.10527 tab. 2. The review also notes that LIME and SHAP depend on sampling, kernel width and baseline, and that Grad-CAM maps are coarse. It cites other work for these points and did not test them 2504.10527 §9.1.

Amortised explainers for graph neural networks

GNNExplainer solves a separate optimisation for each instance. PGExplainer instead trains one small network that scores the edges of a frozen GNN, and then explains new instances in one forward pass 2011.04573 §4.3. The training samples edge masks through a binary concrete relaxation and rewards masks that keep the GNN's prediction 2011.04573 §4.2. The sample count per instance is not stated 2011.04573 §4.2. Author-reported AUC against the planted motif edges is highest in all six settings, from 0.873 to 0.987 2011.04573 tab. 2. Inference is 12x to 108x faster than GNNExplainer per instance, but that figure leaves out the explainer's own training time 2011.04573 tab. 2 2011.04573 app. C.1. The benchmarks plant the ground-truth motifs, so the AUC measures recovery of those motifs 2011.04573 §5.1. Card: Parameterized Explainer for Graph Neural Network. Details are on graph neural network.

Explaining large language models

The LLM survey groups explanation methods by training paradigm. For fine-tuned models it lists local methods (feature attribution, attention, examples and natural-language explanations) and global methods (probing, neurons, concepts and circuits) 2309.01029 §3.1 2309.01029 §3.2. For prompted models it covers in-context learning, chain-of-thought and representation engineering 2309.01029 §4.1. Its evaluation section separates plausibility, measured against human rationales, from faithfulness, and reports that the best-ranked explanation under one faithfulness metric can be the worst under another 2309.01029 §5.1. The survey has no experiments of its own, and its numbers come from the works it cites [arxiv:2309.01029 Results]. Card: Explainability for Large Language Models: A Survey. For the circuit-level view, see mechanistic interpretability.

Explanations written by a generating LLM

Here the explanation is text the model writes about its own design choices, not a method applied afterwards. A TinyML paper asks open-weight LLMs to propose convolutional networks for microcontrollers and to explain their choices 2504.09685 fig. 2. The paper says these explanations guide later proposals 2504.09685 fig. 2. The authors' code prints each explanation for valid candidates, but does not store it or feed it back search.py. The card records no test of whether the explanations are accurate; its results cover accuracy, MACs, flash, parameters and search cost only 2504.09685 tab. 2. Card: Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?. Details are on neural architecture search.

Counterfactual explanations

These answer "what would have to change to get another outcome?". The survey maps the "Why not" and "How to be that" question types to counterfactual methods such as DiCE and CEM 2110.10790 tab. 1. The mapping is the authors' suggestion for discussion, not a validated result [arxiv:2110.10790 tab. 1]. Details are on counterfactual explanation. Card: Human-Centered Explainable AI (XAI).

When to use which

Data, model access and implementation effort are not compared in any card.

Implementation sketch

Run two methods on the same prediction, check their agreement, and report the disagreement rather than hiding it. The top-k agreement function is defined in 2202.01602 app. A.1. The choice of k is made per task 2202.01602 §4.3. For global plots, the perturbation step should follow the quantile option in 2007.04131 §5.1.

# Explain one prediction and check the explanation  [arxiv:2110.10790 §3]
# Input:  model, instance x, the user's question q, k for top-k agreement
# Output: attribution for the user, and agreement scores

def explain_with_check(model, x, q, k):
    # Map the question to candidate techniques  [arxiv:2110.10790 tab. 1]
    candidates = techniques_for(q)
    # Run two different methods on the same prediction  [arxiv:2202.01602 §3.2]
    attr_a = candidates[0](model, x)
    attr_b = candidates[1](model, x)
    # Top-k agreement: lower values mean more disagreement  [arxiv:2202.01602 app. A.1]
    scores = topk_agreement(attr_a, attr_b, k)
    # Show the disagreement; practitioners often use ad hoc heuristics  [arxiv:2202.01602 §5.2.4]
    return attr_a, scores

Contradictions

• Do explanations help people? The survey reports that many studies found positive effects of XAI on understanding, and that many did not find gains in performance 2110.10790 §1 2110.10790 §4. It does not resolve this. It also reports that uncertainty information about a single prediction helped people rely on AI more appropriately than local explanations did in some HCI studies 2110.10790 §4. The debugging study's participants mostly relied on predicted labels, and the authors found that they were unsure about spurious-background models 2011.05429 §3.1 2011.05429 app. H. The health-care survey says evidence that explanations help people is still lacking, and that explanations are neither necessary nor sufficient for trust 2007.15911 §8 2007.15911 §7.2.
• Which method to trust. The disagreement paper's user study shows that practitioners chose among conflicting explanations by preference, intuition or rules of thumb 2202.01602 §5.2. Its metrics measure agreement, not correctness 2202.01602 §3.2. No card establishes which method is correct for a given model.
• Which gradient method is preferred. In the debugging study, Gradient and SmoothGrad were rated above Integrated Gradients on the normal model 2011.05429 app. H. The disagreement paper's user study found participants choosing KernelSHAP most often, and Gradient×Input least often 2202.01602 §5.2.2. The studies used different methods, tasks and participants, so these results are not in direct conflict, and neither shows which method is correct.
• Interpretable models versus accuracy. The pitfalls review says simple models often reach close to the accuracy of complex ones, citing prior work 2007.04131 §4. It does not measure this, and the same card warns that the accuracy of the model must be checked before any explanation is trusted 2007.04131 §3. The interpretable-ML survey goes further and says no general accuracy trade-off has been shown 2103.11251 Introduction, while stating that its evidence is examples and cited experiments, not a theorem. No card runs a controlled comparison across tasks to settle it.
• Is attention an explanation? The LLM survey lists whether raw attention is a valid explanation as an open debate 2309.01029 §3.1.2. No card settles it.
• Do LLM design explanations feed back? The paper says the explanations guide further iterations 2504.09685 fig. 2. The authors' code prints them only and does not reuse them search.py. Which one the reported search results reflect is not settled by the card.
• Explain a black box, or replace it? The survey argues that post-hoc explanations of black boxes can be wrong or misleading, and that they are often used to justify keeping a black box that was never needed 2103.11251 Introduction. A general review of machine learning and deep learning instead lists explainability as a challenge to be met. For high-stakes systems it recommends an explainable white-box model or XAI additions to the model, and it notes that shallow models can be more interpretable than deep ones 2104.05314 p. 4 2104.05314 p. 8. That review does not test either option.

Papers

• Human-Centered Explainable AI (XAI): From Algorithms to User Experiences: survey chapter that maps XAI techniques to user questions and reviews HCI findings on explanation pitfalls.
• The role of explainability in creating trustworthy artificial intelligence for health care: narrative survey defining explainability as interpretability plus fidelity, with a taxonomy, a metric table and a decision guide; no experiments.
• Full-Gradient Representation for Neural Network Visualization: FullGrad, a CNN saliency map built from exact full-gradient terms; author-reported wins on pixel perturbation and remove-and-retrain, with a proof that completeness and weak input dependence conflict.
• The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective: defines six agreement metrics for post-hoc explanations and reports how often six methods disagree.
• General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models: review of ten pitfalls in PDP, ALE, permutation importance and SHAP, each with a worked example and a fix.
• Debugging Tests for Model Explanations: tests whether attribution methods reveal four planted bugs, and reports a 54-person user study.
• The Opportunities and Risks of Large Language Models in Mental Health: narrative review that lists transparency and explainability among the risks of LLMs in mental health care.
• Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection: TF-IDF classifier for email with LIME word explanations and a web app; the explanations are shown for one email only.
• Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review: review of XAI use in food quality, classifying studies by data type and explanation type.
• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges: position survey arguing for inherently interpretable models, with ten open technical challenges; reports no experiments of its own.
• Machine learning and deep learning: conceptual review of AI, ML and deep learning, including explainability and deployment risks; no experiments and no code.
• A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME: shows SHAP and LIME rankings shifting with the classifier and with correlated features on tabular data; proposes NMR and MIP checks, tested on two datasets.
• BORF: symbolic-word transform with a linear classifier whose explanations are repeatable; reported on 158 UCR/UEA datasets.
• Parameterized Explainer for Graph Neural Network: PGExplainer, one shared network that explains many GNN predictions, including unseen graphs, in one forward pass.
• Explainability for Large Language Models: A Survey: taxonomy of explanation methods for fine-tuned and prompted Transformer LLMs, with evaluation and open challenges; no experiments.
• Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?: LLMs propose microcontroller CNNs and explain their choices; the search, Pareto selection and distillation are reported with CIFAR-100 results.
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies: narrative review that lists explainable AI among remedies for data and bias problems; no method or experiments, drafted with ChatGPT.
• Can AI systems have free will?: philosophical essay proposing that an intentional-agent explanation, not mystery or unpredictability, is the test of free will in AI; no experiments.