Explainability for Large Language Models: A Survey

In plain words

Large language models answer questions well, but nobody can easily see why a given answer came out. This survey collects the research that tries to open that black box. It is a map of the field rather than a new method.

The map is organised by how a model is put to work. One way is to train a pretrained model further on labelled examples for one task. The other is to steer a model with a prompt and no extra training. For each way, the survey asks two questions: why did the model give this particular answer, and what does the model know in general? It then covers how to judge whether an explanation is any good, how explanations can be used to find and fix mistakes, and which questions remain open.

TL;DR

• A survey of explainability (interpretability) methods for Transformer-based language models, organised by training paradigm §1 §2.
• Local explanations explain one prediction; global explanations describe what the model has encoded overall §3.
• Covers evaluation metrics for plausibility and faithfulness, and lists open challenges such as the lack of ground-truth explanations §5 §6.1.
• No new experiments and no new method. The numbers quoted are the survey's summaries of other papers §3.1.3 §4.2.1 §4.3.1.

Problem

Large models are hard to inspect. Feature attribution can cost substantial computation for models with billions of parameters §1. Explanations of prompted models are harder still, because reasoning and in-context behaviour do not reduce to one input-output pair or to a simple surrogate §4.

Method

The survey has no method of its own. Its contribution is a taxonomy and a synthesis, summarised here by section.

Paradigms. Methods are split into the traditional fine-tuning paradigm (a pretrained model, then fine-tuning on labelled data for one downstream task) and the prompting paradigm (zero- or few-shot use through prompts, with base models and assistant models treated separately) §2.1 §2.2 fig. 1 fig. 2.

Local explanation of fine-tuned models §3.1 has four families fig. 3:

1. Feature attribution §3.1.1: perturbation (leave-one-out, masking), gradient-based (raw gradients, gradient × input, integrated gradients), surrogate models (LIME, SHAP) and decomposition (LRP, DTD). The survey states the raw gradient and the layer-wise decomposition in unnumbered display equations:

   \boldsymbol{s}_j = \frac{\partial f(\boldsymbol{x})}{\partial \boldsymbol{x}_j} §3.1.1

   R_j^{(l+1)} = \sum_i R_{i \leftarrow j}^{(l, l+1)} §3.1.1

2. Attention-based explanation §3.1.2 fig. 4: visualisations (bipartite graphs, heatmaps), attention blended with gradients, and a discussion of whether raw attention is a valid explanation at all.

3. Example-based explanation §3.1.3: adversarial examples, counterfactual explanations (for example Polyjuice, MiCE, CREST) and data influence (influence functions, TracIn, Data Shapley).

4. Natural-language explanation §3.1.4: a separate model is trained to write explanations alongside predictions.

Global explanation §3.2 asks what components encode:

• probing classifiers on representations and attention heads, and parameter-free probing with targeted datasets §3.2.1;
• neuron analysis, including GPT-4-written explanations of GPT-2 XL neurons scored by simulation §3.2.2 fig. 5;
• concept-based methods, for example TCAV §3.2.3;
• mechanistic interpretability: circuits, induction heads and feed-forward layers as key-value memories §3.2.4.

Using explanations §3.3: debugging for shortcuts and biases §3.3.1, and improving models through explanation regularisation, rationale-augmented prompts and user-feedback loops §3.3.2.

Prompting paradigm §4. Base models: in-context learning §4.1.1, chain-of-thought §4.1.2 and representation engineering §4.1.3. Assistant models: the role of fine-tuning §4.2.1, hallucination §4.2.2 and uncertainty quantification via consistency, verbal confidence and token-level aggregation §4.2.3. Using explanations: explanation-augmented few-shot prompting and explanation tuning §4.3.1, and downstream use cases §4.3.2.

Evaluation §5. Fine-tuned models: plausibility (measured against human rationales with IOU, precision, recall, F1, AUPRC) and faithfulness, with model-level principles and data-level metrics. The survey defines comprehensiveness and sufficiency as:

\text{comprehensiveness} = m(x_i)_j - m(x_i \setminus r_i)_j §5.1

\text{sufficiency} = m(x_i)_j - m(r_i)_j §5.1

Here m is not defined in the text; it is read as the model's probability for class j, as the surrounding prose says ("change in probability"). Prompting models: counterfactual simulatability and faithfulness of chain-of-thought explanations §5.2.

Challenges §6: missing ground truth §6.1, sources of emergent abilities §6.2, comparing the two paradigms §6.3, shortcut learning §6.4, attention redundancy §6.5, moving from snapshot to temporal analysis §6.6, and safety and ethics §6.7.

The survey gives no algorithm. The one procedure it states in full is the raw gradient attribution of §3.1.1, restated here in pseudocode; the survey does not fix the norm used to turn a gradient into a score.

# Raw gradient attribution  [§3.1.1]
# Input:  model f, input vectors inputs (n, d)
# Output: relevance score per input feature, scores (n,)

def gradient_attribution(model, inputs):
    # Partial derivative of the output with respect to each input
    # dimension  [§3.1.1]
    output = model(inputs)
    gradients = d_output_d_inputs(output, inputs)   # (n, d)
    # Magnitude of the derivative reflects sensitivity; norm not stated
    # [§3.1.1]
    scores = magnitude(gradients)                   # (n,)
    return scores

Hyperparameters

Not applicable. The survey trains nothing and specifies no method hyperparameters. Settings that appear in the text belong to the cited works (for example §3.1.3 and §4.2.1) and are listed under Results.

Results

The survey reports no new experiments. The table lists quantitative claims it summarises from other papers. These are author-reported by the cited work, not verified here. Setup is as the survey states it.

Paper vs code

No official code. The TeX source and the arXiv abstract contain no repository or project-page link, and a survey has no implementation to release. Survey reading lists are not code.

Pitfalls

• Survey, not method. Nothing here is directly implementable. The numbers in Results come from other papers, and the survey does not reproduce them. Check the original paper before relying on any figure.
• Snapshot date. The source is version v3 (November 2023). The survey does not cover later work, and claims about specific models (for example GPT-4, Claude 1.0) describe models as they were then.
• Figure 5 mismatch. The caption describes the 131st neuron of layer 5 of GPT-2. The text in §3.2.2 describes neuron explanations for GPT-2 XL. Treat the model identity for that example as unresolved.
• Abstract wording differs. The TeX abstract compares with "conventional deep learning models"; the arXiv metadata abstract says "conventional machine learning models".
• Unnumbered equations. The display equations in §3.1.1 and §5.1 carry no equation numbers in the source, so they are cited by section here.
• Undefined symbol. m in the comprehensiveness and sufficiency formulas of §5.1 is not defined. Inferred from the text as the model's class probability.
• Terminology. "Explainability" and "interpretability" are used interchangeably §1. Probing appears in the local attention discussion (§3.1.2) and as a global method (§3.2.1).
• Open debates stated as open. Whether raw attention is a valid explanation, and whether probing classifiers measure what the model encodes, are both described as unresolved §3.1.2 §3.2.1.
• Metric disagreement. The survey notes that the best-ranked explanation under one faithfulness metric can be the worst under another §5.1.
• Mixed-source claims. Several statements in §3.2.4, §4.2 and §6 are the survey's own interpretation of cited studies. Attribute them to the cited work, not to the survey.

Lineage

• Builds on: the works it cites throughout; no single predecessor method.
• Compared against: not applicable.
• Superseded by: not known from this wiki.