---
kind: concept
slug: explainable-ai
title: Explainable AI
summary: >-
  Making a model's behaviour understandable to the people who use or are affected by it. Covers interpretable models versus post-hoc explanations, global effect plots, feature attributions that can disagree or miss planted bugs, graph and LLM explainers, and choosing a technique by question. Eighteen cards; author-reported evidence, mixed on whether explanations help people.
tags: [explainability, interpretability, evaluation]
papers: ["2110.10790", "2202.01602", "2007.04131", "2011.05429", "2403.14814", "2405.11619", "2504.10527", "2103.11251", "2104.05314", "2311.18029", "2011.04573", "2309.01029", "2212.08104", "2305.02012", "2504.09685", "2007.15911", "1905.00780", "2609.15407"]
related: [feature-attribution, explanation-evaluation, counterfactual-explanation, causal-inference, llm-ethics, interpretable-machine-learning, time-series-classification, graph-neural-network, mechanistic-interpretability]
status: draft
updated_at: 2026-11-01
---

## Overview

Explainable AI (XAI) aims to make a model understandable to the people who use
it or are affected by it. The survey treats a good explanation as depending on
the receiver's knowledge and goals, not only on the model [arxiv:2110.10790 §1].
It proposes starting from the questions users ask, and then choosing a technique
that answers them [arxiv:2110.10790 §3].

Two distinctions recur across the design space:

- **Interpretable by design or post hoc.** Directly interpretable models, such
  as decision trees, rule-based models and linear models, are contrasted with
  opaque models that are explained after training [arxiv:2110.10790 §2].
- **Scope of a post-hoc explanation.** Post-hoc methods are grouped as global
  model explanations, outcome explanations and counterfactual inspection. Each
  can be feature-based or example-based [arxiv:2110.10790 §2]. Most post-hoc
  methods approximate the model, and the survey notes that such approximations
  can be unfaithful [arxiv:2110.10790 §2].

A health-care survey groups post-hoc methods the same way, by what kind of
explanation they produce: a simpler surrogate model, a score per input, or
selected example instances. Each kind can be global or local. Its table also
lists which properties of each kind have a quantitative metric
[arxiv:2007.15911 §4] [arxiv:2007.15911 tab. 1]. It files LIME under surrogate
models, where other surveys file it under attribution [arxiv:2007.15911 §4.1].

Narrative reviews also name explainability as a remedy for opaque predictions.
A drug-discovery review proposes it for transparent predictions, with no method
or test of its own [arxiv:2212.08104 §7]. That review also says most references
in its ChatGPT-drafted text were incorrect [arxiv:2212.08104 §10], so cite it
only for the claim it makes, not as evidence.

A third point runs through the cards: an explanation is an output like any
other, and it can be wrong in ways that look plausible. The model-agnostic
pitfalls review and the debugging-test study below both test this.

A philosophical essay uses explanation in a different way. It treats a good
explanation of a system as an intentional agent as evidence that the system
is one, while stating that the explanation is not a definition of agency
[arxiv:2609.15407 §4] [arxiv:2609.15407 §4, p. 7, fn. 6]. It also names
explainable AI as a reason to build systems whose behaviour is described in
agential terms [arxiv:2609.15407 §4.1, p. 10]. The essay has no experiments and
does not test any system, so it informs the framing and not the method. Card:
[Can AI systems have free will?](/papers/2609.15407).

## Variants

### Interpretable models

The explanation is the model's structure: a decision tree, a rule list or a
linear model [arxiv:2110.10790 §2]. The pitfalls review argues that such
models often perform about as well as complex ones, citing earlier benchmarks
it did not re-check. It recommends starting with linear models or trees, and
moving to generalised additive models and then to interactions only when the
gain in performance justifies it [arxiv:2007.04131 §4]. No card here measures
the accuracy cost of this choice on a task.

The interpretable-ML survey takes a firmer position. It argues that, for
high-stakes decisions, an inherently interpretable model should be tried
first, and that a readable model is often as accurate as a black box
[arxiv:2103.11251 Introduction]. Its support is a set of examples and cited
experiments, not a proof; the one worked comparison is a single benchmark
dataset with one tree of each kind [arxiv:2103.11251 fig. 3]. It lists ten
open problems for such models, among them optimal sparse trees, scoring
systems, generalised additive models, prototype-based reasoning and
disentangled neural networks [arxiv:2103.11251 §1–§10]. Its
"Rashomon set" challenge asks how to choose among many models of near-equal
loss, which matters when the choice of model is part of the explanation
[arxiv:2103.11251 §9]. Card: [Interpretable Machine Learning: Fundamental
Principles and 10 Grand Challenges](/papers/2103.11251).

A health-care survey offers a decision guide that starts from how much
explanation matters. When it matters a lot, the authors prefer an interpretable
model. A post-hoc explanation is considered only when a complex model clearly
outperforms, and the choice of explanation type then depends on whether
fidelity or interpretability is the priority [arxiv:2007.15911 §6.1]
[arxiv:2007.15911 §6.2] [arxiv:2007.15911 §6.3] [arxiv:2007.15911 fig. 2]. The
authors say the guide needs testing in practice [arxiv:2007.15911 §7.1].
Card: [The role of explainability in creating trustworthy artificial
intelligence for health care](/papers/2007.15911).

One hybrid shows the interpretable route on time series. BORF classifies a
series from counts of short symbolic words taken from many sliding windows,
then applies a ridge classifier to those counts. Each count is one feature,
so the coefficients and the per-count contributions can be read directly
[arxiv:2311.18029 §4.2] [arxiv:2311.18029 §4.5]. Its contribution scores use
SHAP on the linear model, which keeps the explanation deterministic across
runs [arxiv:2311.18029 §4.5] [arxiv:2311.18029 §5]. Card: [BORF](/papers/2311.18029).

### Global effects and importance

Partial dependence (PDP), accumulated local effects (ALE) and permutation
feature importance (PFI) explain the model across a dataset. Their pitfalls are
the main subject of [arxiv:2007.04131]:

- **Generalisation first.** Interpreting an under- or overfitted model can
  mislead about the data. The review's example has a linear model that
  underfits, a random forest that overfits and a tuned SVM that fits best; it
  recommends judging fit only on held-out data [arxiv:2007.04131 §3].
- **Feature dependence.** Perturbation-based methods can evaluate the model at
  combinations of feature values that never occur. An equidistant grid also
  creates unrealistic values for skewed features. The review prefers quantiles
  or sampled observed values, and ALE plots where dependence is strong
  [arxiv:2007.04131 §5.1].
- **Marginal or conditional.** Marginal methods stay true to the model but not
  to the data; conditional variants do the reverse. The review asks the user to
  decide which question is being answered before choosing [arxiv:2007.04131 §5.3].
- **Uncertainty and multiple testing.** One PDP can show an effect that is only
  noise, so the review repeats the estimate over resamples [arxiv:2007.04131 §7].
  Testing 50 unimportant features at α = 0.05 gives a probability of about
  0.923 of at least one "significant" result. The review suggests a false
  discovery rate control or a Bonferroni correction [arxiv:2007.04131 §8.3].
- **Causation.** A non-zero PFI does not show that a feature causes the
  outcome. Causal reading needs explicit assumptions and an adjustment set
  [arxiv:2007.04131 §9]. See [causal inference](/concepts/causal-inference).

### Post-hoc feature attribution

Methods such as LIME, SHAP and gradient-based saliency score each input feature
for one prediction. Different methods often give different scores for the same
prediction, especially on text and on pixel-level image attributions
[arxiv:2202.01602 §4.3]. Card: [The Disagreement Problem in Explainable Machine
Learning](/papers/2202.01602). Details are on [feature attribution](/concepts/feature-attribution).

A CNN-specific method, FullGrad, starts from an exact split of a ReLU network's
output into input-gradient and bias-gradient terms [arxiv:1905.00780 eq. 1]. It
post-processes each term and sums them into one saliency map [arxiv:1905.00780
eq. 2]. The paper proves that for piecewise-linear functions a saliency map
cannot in general be both complete (its scores add up to the output) and weakly
dependent on the input [arxiv:1905.00780 prop. 1]. The post-processing makes the
final map approximate, so it no longer adds up to the output [arxiv:1905.00780
§4.2]. The authors report it best among the compared methods on ImageNet pixel
perturbation and on a CIFAR100 remove-and-retrain test; both results appear only
as plots [arxiv:1905.00780 §5.1] [arxiv:1905.00780 §5.2] [arxiv:1905.00780
fig. 2]. On the paper's MNIST tables, integrated gradients has the lowest
pixel-perturbation score at all three settings, and the signed variant of
FullGrad scores highest on digit flipping. The paper concludes that no single
method wins both tests [arxiv:1905.00780 tab. 1] [arxiv:1905.00780 tab. 2]. The
released code applies the post-processing per layer rather than per channel, so
its maps differ from the equation [arxiv:1905.00780 eq. 2]; the card records
this. Card: [Full-Gradient Representation for Neural Network
Visualization](/papers/1905.00780).

On tabular data, a perspective on SHAP and LIME reports that the feature
ranking depends on the classifier. Four classifiers fitted to one UK Biobank
cohort agreed on the top three SHAP features, but the lower ranks varied
[arxiv:2305.02012 §2.1] [fig. 2]. SHAP assumes features are independent, and
correlated features are a stated risk for its scores [arxiv:2305.02012 §2.1].
The paper's airline case adds two proxy measures for this: NMR, a stability
score for a ranking, and MIP, a re-ordering that accounts for correlated
features. The paper gives no formula for either; the authors' code does
[arxiv:2305.02012 §3] [arxiv:2305.02012 tab. 3]. Its results come from one
dataset per case, with no seeds or intervals [arxiv:2305.02012 §2.1] [§3]. Card:
[A Perspective on Explainable Artificial Intelligence Methods: SHAP and
LIME](/papers/2305.02012).

Two studies show where attributions help and where they do not:

- **Planted bugs.** In controlled tests, attribution maps picked up a spurious
  background that the model had learned to use [arxiv:2011.05429 §3.1]
  [arxiv:2011.05429 tab. 2]. They did not separate training images with flipped
  labels from correct ones [arxiv:2011.05429 §3.2] [arxiv:2011.05429 fig. 6].
  For several modified back-propagation methods, the maps stayed the same after
  the top layers were re-initialised, so these methods cannot reveal that bug
  [arxiv:2011.05429 §4] [arxiv:2011.05429 fig. 7].
- **Text classifiers.** A linear support vector classifier on TF-IDF features
  was explained with LIME word weights for one generated phishing-style email.
  The paper does not measure whether these weights are faithful or stable
  [arxiv:2405.11619 p. 15]. LIME settings such as the sample count are not
  stated [arxiv:2405.11619 p. 10] [arxiv:2405.11619 p. 11].

Visual explanations dominate in practice. A review of food-quality studies
found that most use heatmaps, mostly CAM-type methods, while spectral and
time-series data are rarely explained. Few studies combined more than one XAI
method [arxiv:2504.10527 §9] [arxiv:2504.10527 fig. 8]. Those authors report
that most works choose Grad-CAM over LRP although their costs are similar, and
cite human studies for the choice [arxiv:2504.10527 §3.2] [arxiv:2504.10527 tab. 2].
The review also notes that LIME and SHAP depend on sampling, kernel width and
baseline, and that Grad-CAM maps are coarse. It cites other work for these
points and did not test them [arxiv:2504.10527 §9.1].

### Amortised explainers for graph neural networks

GNNExplainer solves a separate optimisation for each instance. PGExplainer
instead trains one small network that scores the edges of a frozen GNN, and
then explains new instances in one forward pass [arxiv:2011.04573 §4.3]. The
training samples edge masks through a binary concrete relaxation and rewards
masks that keep the GNN's prediction [arxiv:2011.04573 §4.2]. The sample count
per instance is not stated [arxiv:2011.04573 §4.2]. Author-reported AUC against
the planted motif edges is highest in all six settings, from 0.873 to 0.987
[arxiv:2011.04573 tab. 2]. Inference is 12x to 108x faster than GNNExplainer per
instance, but that figure leaves out the explainer's own training time
[arxiv:2011.04573 tab. 2] [arxiv:2011.04573 app. C.1]. The benchmarks plant the
ground-truth motifs, so the AUC measures recovery of those motifs
[arxiv:2011.04573 §5.1]. Card: [Parameterized Explainer for Graph Neural
Network](/papers/2011.04573). Details are on [graph neural network](/concepts/graph-neural-network).

### Explaining large language models

The LLM survey groups explanation methods by training paradigm. For fine-tuned
models it lists local methods (feature attribution, attention, examples and
natural-language explanations) and global methods (probing, neurons, concepts
and circuits) [arxiv:2309.01029 §3.1] [arxiv:2309.01029 §3.2]. For prompted
models it covers in-context learning, chain-of-thought and representation
engineering [arxiv:2309.01029 §4.1]. Its evaluation section separates
plausibility, measured against human rationales, from faithfulness, and
reports that the best-ranked explanation under one faithfulness metric can be
the worst under another [arxiv:2309.01029 §5.1]. The survey has no experiments
of its own, and its numbers come from the works it cites [arxiv:2309.01029
Results]. Card: [Explainability for Large Language Models: A
Survey](/papers/2309.01029). For the circuit-level view, see [mechanistic
interpretability](/concepts/mechanistic-interpretability).

### Explanations written by a generating LLM

Here the explanation is text the model writes about its own design choices,
not a method applied afterwards. A TinyML paper asks open-weight LLMs to
propose convolutional networks for microcontrollers and to explain their
choices [arxiv:2504.09685 fig. 2]. The paper says these explanations guide
later proposals [arxiv:2504.09685 fig. 2]. The authors' code prints each
explanation for valid candidates, but does not store it or feed it back
[code: search.py#L1045-L1046]. The card records no test of whether the
explanations are accurate; its results cover accuracy, MACs, flash, parameters
and search cost only [arxiv:2504.09685 tab. 2]. Card: [Can LLMs Revolutionize
the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685).
Details are on [neural architecture search](/concepts/neural-architecture-search).

### Counterfactual explanations

These answer "what would have to change to get another outcome?". The survey
maps the "Why not" and "How to be that" question types to counterfactual
methods such as DiCE and CEM [arxiv:2110.10790 tab. 1]. The mapping is the
authors' suggestion for discussion, not a validated result [arxiv:2110.10790
tab. 1]. Details are on [counterfactual explanation](/concepts/counterfactual-explanation).
Card: [Human-Centered Explainable AI (XAI)](/papers/2110.10790).

## When to use which

| Situation | Use | Why |
|---|---|---|
| The model can be a simple structure and the explanation must be read directly | Interpretable model | Directly interpretable camp in the survey [arxiv:2110.10790 §2]; the pitfalls review advises starting here [arxiv:2007.04131 §4] |
| A high-stakes decision, and the model class is still open | Fit an inherently interpretable model and compare it with a black box on your data before explaining the black box | The survey argues the accuracy gap is often small, on examples and cited experiments [arxiv:2103.11251 Introduction] |
| Time series classification where the same explanation must come out on every run | A deterministic symbolic transform with a linear classifier, such as BORF | Author-reported: statistically tied with ROCKET and MiniRocket on 158 datasets, but not the most accurate [arxiv:2311.18029 §5] [tab. 2]. Its median runtime is below most baselines in [tab. 3]; its mean is driven by two multivariate datasets [arxiv:2311.18029 §5] |
| Effects of features across a dataset are needed | PDP or ALE on held-out data, on a quantile grid | Generalisation and dependence pitfalls [arxiv:2007.04131 §3] [arxiv:2007.04131 §5.1] |
| Tabular data with correlated features, and a SHAP ranking must be reported | Fit several classifiers, report the ranking for each, and check stability (NMR) before choosing one | Top-ranked features agreed across models, lower ranks did not; the most stable ranking came from the most accurate model [arxiv:2305.02012 §2.1] [arxiv:2305.02012 §3] [tab. 2] |
| Many features are screened for importance | Correct for multiple tests, or control the false discovery rate | At α = 0.05 with 50 noise features, about 0.92 chance of a false positive [arxiv:2007.04131 §8.3] |
| An opaque model, and a per-prediction feature score is needed | Post-hoc feature attribution, run with at least two methods | Methods disagree often; practitioners interviewed used several methods for one prediction [arxiv:2202.01602 §3.1, §4.3] |
| Checking whether a method would reveal a known defect | Plant the defect and test the method, as in the debugging tests | Only some defects were visible in the maps [arxiv:2011.05429 §3–§4] |
| Latency or compute is tight | Keep the sample count low, but check it converged | The paper's tabular runs used 2,000 perturbations; its rule is to stop when more samples no longer change the explanation [arxiv:2202.01602 app. B.2]. It reports no compute time [arxiv:2202.01602 app. B.2]. A food review's per-explanation cost table gives LIME O(k d²), SHAP O(d 2^d), and O(1) for Grad-CAM and LRP; these are authors' figures, not timings [arxiv:2504.10527 tab. 2] |
| The user asks what would change the outcome | Counterfactual explanation | Question mapping in [arxiv:2110.10790 tab. 1] |
| The saliency map must add up to the model output (completeness) | Use a completeness-based method, such as integrated gradients or DeepLIFT, and accept that its map is not weakly input-dependent | The paper proves the two properties cannot in general both hold for piecewise-linear functions [arxiv:1905.00780 prop. 1]; FullGrad is complete only before post-processing [arxiv:1905.00780 §4.2] |
| An LLM proposes a design and its rationale is wanted with each candidate | Store the explanation with its candidate, and check it against measured MACs, SRAM and accuracy | The paper's LLM explanations guide proposals, but the card reports no check of their accuracy [arxiv:2504.09685 fig. 2] [tab. 2] |
| Choosing between two explanation methods for one model | Measure agreement with the six metrics before trusting either | Metrics are agreement scores, higher means more agreement [arxiv:2202.01602 §3.2] |
| The question is causal ("does this feature cause the outcome?") | Add causal assumptions; do not read attributions as causes | Predictive models use associations [arxiv:2007.04131 §9] |
| The explanation is for people, not for model developers | Test with users, not only with proxy tasks | Proxy tasks may not predict real effectiveness [arxiv:2110.10790 §4]. In the debugging study, users mostly judged models by their predicted labels [arxiv:2011.05429 app. H] |
| A graph neural network must be explained for many instances, or for graphs unseen in training | An amortised explainer such as PGExplainer, trained once on the frozen GNN | Author-reported best AUC on the six benchmarks, with inference 12x to 108x faster than GNNExplainer; training cost excluded [arxiv:2011.04573 tab. 2] [arxiv:2011.04573 app. C.1] |
| An LLM, fine-tuned or prompted, must be explained and the method family is open | Choose the paradigm first, then local or global, and plausibility or faithfulness | The survey organises the field this way but runs no comparison [arxiv:2309.01029 §2.1–§2.2] [arxiv:2309.01029 §5.1] |
| An LLM is used in a high-stakes setting such as mental health | Treat explanation as a risk control to verify, not as a given | The review rates transparency as a medium risk for assessment and intervention, and states that no numeric thresholds are given [arxiv:2403.14814 §3.6] [arxiv:2403.14814 tab. 1] |
| Explanation is very important, as in a clinical setting | Explainable modelling: an interpretable model, not a post-hoc explanation of a black box | The health-care survey's guide returns this choice at very high importance without the complex-model check [arxiv:2007.15911 §6.1] [fig. 2]. The guide is untested [arxiv:2007.15911 §7.1] |
| A post-hoc explanation must be reported with a claim that it is faithful | Use the metric the survey lists for its kind: agreement with the model for surrogates; no known metric for example-based methods or for local clarity | Coverage of metrics per property [arxiv:2007.15911 tab. 2] [§5.1]–[§5.3] |

Data, model access and implementation effort are not compared in any card.

## Implementation sketch

Run two methods on the same prediction, check their agreement, and report the
disagreement rather than hiding it. The top-k agreement function is defined in
[arxiv:2202.01602 app. A.1]. The choice of k is made per task
[arxiv:2202.01602 §4.3]. For global plots, the perturbation step should follow
the quantile option in [arxiv:2007.04131 §5.1].

```pseudocode
# Explain one prediction and check the explanation  [arxiv:2110.10790 §3]
# Input:  model, instance x, the user's question q, k for top-k agreement
# Output: attribution for the user, and agreement scores

def explain_with_check(model, x, q, k):
    # Map the question to candidate techniques  [arxiv:2110.10790 tab. 1]
    candidates = techniques_for(q)
    # Run two different methods on the same prediction  [arxiv:2202.01602 §3.2]
    attr_a = candidates[0](model, x)
    attr_b = candidates[1](model, x)
    # Top-k agreement: lower values mean more disagreement  [arxiv:2202.01602 app. A.1]
    scores = topk_agreement(attr_a, attr_b, k)
    # Show the disagreement; practitioners often use ad hoc heuristics  [arxiv:2202.01602 §5.2.4]
    return attr_a, scores
```

## Contradictions

- **Do explanations help people?** The survey reports that many studies found
  positive effects of XAI on understanding, and that many did not find gains in
  performance [arxiv:2110.10790 §1] [arxiv:2110.10790 §4]. It does not resolve
  this. It also reports that uncertainty information about a single prediction
  helped people rely on AI more appropriately than local explanations did in
  some HCI studies [arxiv:2110.10790 §4]. The debugging study's participants
  mostly relied on predicted labels, and the authors found that they were
  unsure about spurious-background models [arxiv:2011.05429 §3.1] [arxiv:2011.05429 app. H].
  The health-care survey says evidence that explanations help people is still
  lacking, and that explanations are neither necessary nor sufficient for trust
  [arxiv:2007.15911 §8] [arxiv:2007.15911 §7.2].
- **Which method to trust.** The disagreement paper's user study shows that
  practitioners chose among conflicting explanations by preference, intuition or
  rules of thumb [arxiv:2202.01602 §5.2]. Its metrics measure agreement, not
  correctness [arxiv:2202.01602 §3.2]. No card establishes which method is
  correct for a given model.
- **Which gradient method is preferred.** In the debugging study, Gradient and
  SmoothGrad were rated above Integrated Gradients on the normal model
  [arxiv:2011.05429 app. H]. The disagreement paper's user study
  found participants choosing KernelSHAP most often, and Gradient×Input least
  often [arxiv:2202.01602 §5.2.2]. The studies used different methods, tasks
  and participants, so these results are not in direct conflict, and neither
  shows which method is correct.
- **Interpretable models versus accuracy.** The pitfalls review says simple
  models often reach close to the accuracy of complex ones, citing prior work
  [arxiv:2007.04131 §4]. It does not measure this, and the same card warns that
  the accuracy of the model must be checked before any explanation is trusted
  [arxiv:2007.04131 §3]. The interpretable-ML survey goes further and says no
  general accuracy trade-off has been shown [arxiv:2103.11251 Introduction],
  while stating that its evidence is examples and cited experiments, not a
  theorem. No card runs a controlled comparison across tasks to settle it.
- **Is attention an explanation?** The LLM survey lists whether raw attention
  is a valid explanation as an open debate [arxiv:2309.01029 §3.1.2]. No card
  settles it.
- **Do LLM design explanations feed back?** The paper says the explanations
  guide further iterations [arxiv:2504.09685 fig. 2]. The authors' code prints
  them only and does not reuse them [code: search.py#L1045-L1046]. Which one
  the reported search results reflect is not settled by the card.
- **Explain a black box, or replace it?** The survey argues that post-hoc
  explanations of black boxes can be wrong or misleading, and that they are
  often used to justify keeping a black box that was never needed
  [arxiv:2103.11251 Introduction]. A general review of machine learning and
  deep learning instead lists explainability as a challenge to be met. For
  high-stakes systems it recommends an explainable white-box model or XAI
  additions to the model, and it notes that shallow models can be more
  interpretable than deep ones [arxiv:2104.05314 p. 4] [arxiv:2104.05314 p. 8].
  That review does not test either option.

## Papers

- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](/papers/2110.10790): survey chapter that maps XAI techniques to user questions and reviews HCI findings on explanation pitfalls.
- [The role of explainability in creating trustworthy artificial intelligence for health care](/papers/2007.15911): narrative survey defining explainability as interpretability plus fidelity, with a taxonomy, a metric table and a decision guide; no experiments.
- [Full-Gradient Representation for Neural Network Visualization](/papers/1905.00780): FullGrad, a CNN saliency map built from exact full-gradient terms; author-reported wins on pixel perturbation and remove-and-retrain, with a proof that completeness and weak input dependence conflict.
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](/papers/2202.01602): defines six agreement metrics for post-hoc explanations and reports how often six methods disagree.
- [General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models](/papers/2007.04131): review of ten pitfalls in PDP, ALE, permutation importance and SHAP, each with a worked example and a fix.
- [Debugging Tests for Model Explanations](/papers/2011.05429): tests whether attribution methods reveal four planted bugs, and reports a 54-person user study.
- [The Opportunities and Risks of Large Language Models in Mental Health](/papers/2403.14814): narrative review that lists transparency and explainability among the risks of LLMs in mental health care.
- [Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection](/papers/2405.11619): TF-IDF classifier for email with LIME word explanations and a web app; the explanations are shown for one email only.
- [Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review](/papers/2504.10527): review of XAI use in food quality, classifying studies by data type and explanation type.
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](/papers/2103.11251): position survey arguing for inherently interpretable models, with ten open technical challenges; reports no experiments of its own.
- [Machine learning and deep learning](/papers/2104.05314): conceptual review of AI, ML and deep learning, including explainability and deployment risks; no experiments and no code.
- [A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME](/papers/2305.02012): shows SHAP and LIME rankings shifting with the classifier and with correlated features on tabular data; proposes NMR and MIP checks, tested on two datasets.
- [BORF](/papers/2311.18029): symbolic-word transform with a linear classifier whose explanations are repeatable; reported on 158 UCR/UEA datasets.
- [Parameterized Explainer for Graph Neural Network](/papers/2011.04573): PGExplainer, one shared network that explains many GNN predictions, including unseen graphs, in one forward pass.
- [Explainability for Large Language Models: A Survey](/papers/2309.01029): taxonomy of explanation methods for fine-tuned and prompted Transformer LLMs, with evaluation and open challenges; no experiments.
- [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685): LLMs propose microcontroller CNNs and explain their choices; the search, Pareto selection and distillation are reported with CIFAR-100 results.
- [The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies](/papers/2212.08104): narrative review that lists explainable AI among remedies for data and bias problems; no method or experiments, drafted with ChatGPT.
- [Can AI systems have free will?](/papers/2609.15407): philosophical essay proposing that an intentional-agent explanation, not mystery or unpredictability, is the test of free will in AI; no experiments.