Post-hoc Interpretability for Neural NLP: A Survey

In plain words

Large language models are hard to inspect, and people want to know why a model gave a certain answer, especially when it is used for decisions that matter. This survey looks at one family of tools: methods that explain a model after it has been trained, without changing how it was built. It sorts them by the kind of explanation they give. Some point at the words that mattered most. Some show a small edit that would flip the answer. Some find training examples that look like the one in question. Others explain a single internal neuron or a vocabulary-wide pattern, or check whether a model has learned language rules.

The authors also ask how each family is checked. Does the explanation actually match what the model does, and is it understandable to a person? For many of these families the answer is still unclear, so the survey is as much a guide to what is missing as a catalogue of methods. It does not implement anything or report new experiments. It is useful for choosing which kind of explanation fits an application, and for knowing which checks to run on it.

TL;DR

• A survey of post-hoc (after-training) interpretability methods for neural NLP, not a new method and not an implementation.
• It organises the field by communication type (what the explanation looks like) and by the information each method uses §1 overview table.
• Each family is paired with how its authors validate it. Validation is often weak or missing, especially for human-centred and faithfulness-centred checks §4 §17.
• Gives no numeric results of its own. The one headline number in the card is a human-study result from another paper, quoted as cited by the survey §13.1.

Problem

Neural NLP models are large and hard to read, so they are treated as black boxes §1. Safety, ethics and accountability all push for explanations, especially when a wrong prediction causes harm and someone must answer for it §1 §2. Building a model that is interpretable by design is task dependent, hard, and rarely done in industry, so post-hoc methods are used instead §1. This survey limits itself to post-hoc methods, which are applied to an existing neural model after training §1.

The survey notes that the field has no agreed measure of interpretability. It separates three kinds of evaluation: application-grounded (tested in the real deployment setting), human-grounded (tested with people, or with a stand-in for people), and functionally-grounded (tested for whether the explanation reflects the model, i.e. faithfulness) §4. Most NLP work uses only the last two, because the first is slow and rarely done §4.

Method

This is a survey, so there is no algorithm to reproduce. Its organising device is the overview table in §1, which places each communication approach by how abstract it is and by which information it uses (gradients, the dataset, the embedding matrix, a second model, and so on). Two more distinctions run through the whole survey:

• Scope of the explanation. A local explanation covers one prediction, a class explanation covers one output class, and a global explanation covers the whole model §1.1.
• Post-hoc versus intrinsic. Post-hoc methods are applied after training. Intrinsic methods are interpretable by design, for example attention used as the explanation. The survey treats intrinsic claims as things to verify, and points to disputes about whether attention is an explanation §1.2 §4.

The ten method sections §6 to §15 each cover one communication approach.

Input features §6. A local explanation that scores how much each input token contributed to one class output. Covered: gradients §6.1, Integrated Gradients §6.2, LIME §6.3, Kernel SHAP §6.4, and Anchors §6.5. The survey notes the same model output can be attributed to tokens in very different ways, so faithfulness is task dependent §6.2.

# Integrated Gradients for one class c  [§6.2]
# Input:  input x (T, d); baseline b (same shape); number of steps k
# Output: attribution (T, d)

def integrated_gradients(x, b, c, k):
    # Accumulate the class-c gradient at points on the path from b to x  [§6.2]
    total_gradient = zeros_like(x)
    for i in range(1, k + 1):
        # Point at fraction i/k of the way from the baseline to the input  [§6.2]
        scaled = b + (i / k) * (x - b)
        # Gradient of the class-c logit at that point  [§6.2]
        total_gradient += grad_wrt_input(scaled, c)
    # Average the gradients, then scale by the difference from the baseline  [§6.2]
    return (x - b) * (total_gradient / k)

Adversarial examples §7. A local, contrastive explanation: a minimally changed input that the model classifies differently, with the same gold label. Covered: HotFlip §7.1, which estimates the loss change of each token swap with one backward pass, and SEA §7.2, which samples paraphrases from a paraphrase model. The survey notes that HotFlip's greedy search can miss effects that depend on several tokens together, such as double negation §7.3.

# HotFlip: score one token swap at position t with one backward pass  [§7.1]
# Input:  gradient of the loss wrt the one-hot input at position t, (V,)
# Output: index of the replacement token with the largest estimated gain

def best_swap(grad_t, current_token):
    # First-order estimate of the loss change for each candidate swap  [§7.1]
    estimated_gain = grad_t - grad_t[current_token]
    # Keep the candidate with the largest estimated increase in loss  [§7.1]
    return argmax(estimated_gain)

Several swaps are chosen by beam search over these scores §7.1. For word-level models, the candidate word must stay close to the original in the embedding space (cosine similarity at least 0.8) so the change stays a paraphrase §7.1.

Similar (influential) training examples §8. For one input, find training examples that the model treats as similar, using the model's own quantities rather than an input distance such as BLEU §8 intro. Covered: influence functions §8.1, representer point selection §8.2 and TracIn §8.3.

\mathcal{L}(y, \mathbf{x}; \tilde{\theta}) - \mathcal{L}(y, \mathbf{x}; \theta) \approx \frac{1}{n}\, \nabla_\theta \mathcal{L}(y, \mathbf{x};\theta)^\top H_\theta^{-1} \nabla_\theta \mathcal{L}(\tilde{y}, \tilde{\mathbf{x}};\theta) \quad [\S 8.1]

Here \tilde{\theta} is the parameter set when the training example \tilde{\mathbf{x}} is left out and H_\theta is the Hessian of the loss §8.1. TracIn instead sums gradient dot products over saved training checkpoints, scaled by the learning rate and batch size §8.3.

Counterfactuals §9. A minimal edit that changes the prediction and carries a different gold label, often the opposite one. This separates it from an adversarial example, which keeps the gold label §9 intro. Covered: Polyjuice §9.1, which generates edits with a fine-tuned GPT-2 and filters them by the model's prediction change, and MiCE §9.2, which masks gradient-selected tokens and fills them with a T5 model until the prediction flips.

Natural-language explanations §10. Post-hoc rationalisation: a separate generative model writes an explanation after the prediction is made. Covered: CAGE §10.1. The survey observes that this explanation model has no link to the explained model beyond the shared answer, so it need not reflect the model's behaviour §10.1.

Concepts §11. Explain a model through a human-defined concept, such as occupation words in a bias study. Covered: the Natural Indirect Effect from causal mediation analysis, which measures how much an intermediate component (here an attention head in a small GPT-2) carries a bias effect when a concept word is intervened on §11.1.

Vocabulary §12. Global explanations about the embedding vocabulary. Covered: projections such as PCA and t-SNE, and supervised projections onto a chosen direction §12.1. Also covered: rotations of the embedding space into interpretable basis dimensions, using the Crawford–Ferguson family from exploratory factor analysis §12.2.

Ensembles §13. A global explanation built by selecting several local explanations that represent different modes of the model. Covered: SP-LIME, which greedily picks a budget of observations to maximise coverage of the features that LIME explanations weight most §13.1.

Linguistic information §14. Global explanations that test whether a model has learned linguistic properties. Behavioural probes change the input and observe the model §14.1. Structural probes train a small classifier on intermediate representations to predict a linguistic label §14.2. The survey treats the critiques of probing as central: a probe may learn the task itself, not reveal what the model encodes §14.2.

Rules §15. Global explanations as simple logical rules. Covered: SEAR, which builds rules from word-level adversarial changes and keeps those with high flip rates §15.1, and Compositional Explanations of Neurons, which searches boolean rules over word presence to match one neuron's activation, scored with an intersection-over-union measure §15.2.

Hyperparameters

This survey has no experiments, so it has no training hyperparameters. The table lists the free settings that each method's own paper or the survey states, so an implementer knows what to choose. "not stated" means the survey gives no value.

Results

Author-reported. Survey only: no new experiments by these authors.

Headline results the survey reports from cited work:

The survey itself notes that the BERT and human rows are not directly comparable, and that the HANS training data (MNLI) reached 83% for BERT §14.1. Influence-function and representer-point results from the cited papers are reported only qualitatively in the survey §8.1 §8.2.

Paper vs code

No official code.

The paper's LaTeX source and abstract link no code repository. Its bibliography and template links are for citations and the ACM class only. No project page is linked. A GitHub search for the authors' accounts (user AndreasMadsen) found three repositories, which belong to other works (LLM introspection, ROAR interpretability, a talk), so none is this survey's code and none was cloned. A generic repository search for the title words returned nothing.

Source checks: the arXiv metadata (raw/arxiv/2108.04840.json) lists version v5 and CC BY 4.0, and the LaTeX source was read in full for the method chapters §6 to §15.

Pitfalls

• Survey, not a method. There is nothing to implement from this card alone. An agent using it should pick one method family and then read that method's own paper and code.
• Section numbers are derived, not checked against the PDF. Section and subsection numbers above follow the order of \input in the LaTeX source (v5). Verify them against the rendered PDF before citing from another card.
• Terminology conflicts. Adversarial and counterfactual examples are often mixed up. The distinction is the gold label: kept for adversarial, changed for counterfactual §9 intro. Contrast sets and counterfactual datasets are generated without the model and are not explanations of it §9 intro.
• Faithfulness is task-dependent for input features. Integrated Gradients and gradients have been criticised for not reflecting the model in computer vision, and the NLP evidence is also task-dependent §6.2. Attribution by multiplying the input by the gradient is a separate choice that can reduce faithfulness §6.1.
• Perturbation-based methods can be gamed. LIME and SHAP depend on perturbed inputs. The survey cites work showing that a model can be built to look fair on perturbed inputs while behaving differently on real ones §6.4.
• SHAP needs a valid masked input. Masking requires a token the model can read; for many models this is not available after training, which makes SHAP sit between post-hoc and intrinsic methods §6.4 §18.
• Influence functions are fragile. The Hessian is not guaranteed to be positive definite outside convex models, so a diagonal term is added §8.1. The gradient in the formula is near zero for well-fit training points, which causes numerical instability §8.1. The inverse-Hessian product must be computed per test example, and it is still too slow for interactive use §8.1.
• Representer point selection is a last-layer method. It depends on a regularised final layer, and its post-hoc version re-fits that layer, so it only sees the last layer's representations §8.2.
• Probes can learn the task. With a large enough probe and dataset, a probe may learn the target itself. Control tasks and information-theoretic measures such as minimum description length are the proposed fixes §14.2.
• Behavioural probes do not give faithfulness by themselves. Challenge datasets test robustness, and a good score says nothing about the mechanism §14.1.
• Rationales are not checked against the model. The explainer model in CAGE is trained on human explanations, and only its answer is tied to the explained model §10.1.
• Human validation is often replaced by simulated users. The survey argues this inflates results, since explaining well to people needs interdisciplinary input §4.
• Rule and ensemble methods are rarely validated. Compositional rules have no groundedness check §15.2. Ensemble quality depends on both the local explanations and the selection step, which also needs validation §13 intro §13.2.
• Appendix chapters are excluded from the arXiv build. The source wraps the appendix in \ifacm, which renders as empty, so no appendix content was used.

Lineage

• Builds on: the survey's own taxonomy draws on earlier surveys of interpretability for NLP, named in §1 §1. It is not an extension of any single method.
• Covers, as cited inside the survey (titles and years from the bibliography): "Why should I trust you?" (2016, LIME) §6.3; "Axiomatic attribution for deep networks" (2017, Integrated Gradients) §6.2; "A Unified Approach to Interpreting Model Predictions" (2017, SHAP) §6.4; "Anchors: High-precision model-agnostic explanations" (2018) §6.5; "HotFlip: White-Box Adversarial Examples for Text Classification" (2018) §7.1; "Understanding Black-box Predictions via Influence Functions" (2017) §8.1; "Estimating Training Data Influence by Tracing Gradient Descent" (2020, TracIn) §8.3; "Representer Point Selection for Explaining Deep Neural Networks" (2018) §8.2; "Explaining NLP Models via Minimal Contrastive Editing (MiCE)" (2021) §9.2; "Compositional Explanations of Neurons" (2020) §15.2; "Investigating Gender Bias in Language Models Using Causal Mediation Analysis" (2020, NIE) §11.1; "Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference" (2019, HANS) §14.1; "Designing and Interpreting Probes with Control Tasks" (2019) §14.2; "Information-Theoretic Probing with Minimum Description Length" (2020) §14.2.
• Compared against: no methods are compared quantitatively; the survey states there is no unified benchmark §16.
• Superseded by: not stated in the content of this card.