---
kind: concept
slug: language-model-surprisal
title: Language model surprisal
summary: >-
  Surprisal is the negative log probability a language model gives a token in its context. Word surprisal predicts human reading times, scored as held-out ΔLogLik over a baseline. Sequence perplexity of a prompt ranks zero-shot prompts without labels. Both uses score different outcomes, so their numbers are not comparable. Define tokenisation and the spillover window before comparing.
tags: [psycholinguistics, reading-time, language-model-evaluation, prompting]
papers: ["2006.01912", "2212.04037"]
related: [llm-evaluation, language-model-pretraining, likelihood-based-text-evaluation, prompt-engineering, zero-shot-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

Surprisal is the negative log of the probability a language model gives a
token after the text before it. Two uses are covered here, and they score
different things.

The first is reading time. Human reading slows on words that are hard to
predict from the text before them, so a model's word surprisal can stand in
for that difficulty [arxiv:2006.01912 §1]. The shared measure is the held-out
gain in log-likelihood that surprisal adds to a regression of reading time on
word length and frequency, reported as ΔLogLik per token [arxiv:2006.01912 §2.2].
The invariant across these variants is the baseline: each score is a
difference from a model that uses only length and frequency controls
[arxiv:2006.01912 §2.2].

The second is prompt selection. A prompt is scored by the perplexity of its
text under the model, averaged over task inputs, with no label included. The
authors use perplexity as a proxy for how familiar the prompt wording is to
the model [arxiv:2212.04037 §2]. This variant has no baseline. Candidate
prompts are ranked against each other on the same inputs [arxiv:2212.04037 §7].

## Variants

### Word surprisal from an n-gram model

Reading-time variant. A 5-gram model with Kneser-Ney smoothing, built with
SRILM [arxiv:2006.01912 §2.1]. Its perplexity is about twice the neural
models' on the same data, yet its ΔLogLik is comparable or higher on Dundee,
BLLIP-SM and BLLIP-XS [arxiv:2006.01912 §3.2]. Card: [Wilcox et al., 2020](/papers/2006.01912).

### Word surprisal from an LSTM

Reading-time variant. An LSTM language model, trained with a boilerplate
PyTorch implementation; the paper does not state its size or training schedule
[arxiv:2006.01912 §2.1]. The paper reports that Transformers and 5-grams beat
LSTMs at a given perplexity [arxiv:2006.01912 §4]. Card: [Wilcox et al., 2020](/papers/2006.01912).

### Word surprisal from a recurrent neural network grammar

Reading-time variant. An RNNG models words and their syntactic structure
jointly, trained on Penn Treebank-style constituency parses
[arxiv:2006.01912 §2.1]. The paper's training details for it are not stated
[arxiv:2006.01912 §2.1]. Card: [Wilcox et al., 2020](/papers/2006.01912).

### Word surprisal from a Transformer

Reading-time variant. A GPT-2 architecture trained from scratch on BLLIP
subsets, with 12 layers [arxiv:2006.01912 §2.1] [arxiv:2006.01912 §4]. Some
Transformers use byte-pair encoding, so they predict sub-word tokens
[arxiv:2006.01912 §2.1]. Transformers trained on larger corpora with
byte-pair encoding get the highest scores on all three corpora
[arxiv:2006.01912 §4]. Card: [Wilcox et al., 2020](/papers/2006.01912).

### Mean prompt perplexity for prompt selection

Prompt-selection variant. For each zero-shot prompt, the paper instantiates it
with about 1,000 task inputs. For classification, the input comes first and
the prompt follows. The label is left out. The perplexity of each instance is
computed, and the values are averaged [arxiv:2212.04037 §2] [arxiv:2212.04037 §4.2].
The paper gives no formula for perplexity, and it does not state tokenisation,
a start token or the context window [arxiv:2212.04037 §2].

The hypothesis is that, averaged over tasks and models, a lower-perplexity
prompt performs better [arxiv:2212.04037 §2]. Across OPT 175B, OPT 30B, OPT
1.3B and Bloom 176B, most correlations are negative and significant on the
seven classification tasks, the antonym task and the word-translation task.
The exceptions include IMDB with OPT 175B, which is positive
[arxiv:2212.04037 tab. 5]. The authors also tested prompt length as a
predictor, which gives weak positive correlations, mostly not significant
[arxiv:2212.04037 §5].

The authors turn the hypothesis into SPELL. It paraphrases seed prompts with
GPT-3 text-davinci-002 using seven meta-prompts, backtranslates them through
NLLB languages, ranks the candidates by mean perplexity and keeps the three
lowest [arxiv:2212.04037 §3] [arxiv:2212.04037 §7]. The average accuracy gain
over the manual prompts is 1.8 points with OPT and 2.3 with Bloom
[arxiv:2212.04037 §7.1]. The per-task gains vary: the AG News gain with Bloom is
−12.5 points, and the DBpedia gain with OPT is −5.7 [arxiv:2212.04037 tab. 10].
Card: [Gonen et al., 2022](/papers/2212.04037).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You cannot train a neural model and need a reading-time surprisal predictor | An n-gram model | Its ΔLogLik is comparable to the neural models' on Dundee and BLLIP-SM/XS, despite about twice the perplexity [arxiv:2006.01912 §3.2] |
| You want the best fit to reading times and can train a Transformer | A GPT-2 style Transformer on a larger corpus, with byte-pair encoding | It scores highest on all three corpora in the paper [arxiv:2006.01912 §4] |
| You are choosing between LSTM and Transformer at a fixed perplexity | A Transformer, or a 5-gram | Both beat LSTMs at a given perplexity [arxiv:2006.01912 §4] |
| You want a grammar-quality proxy as a predictor of reading times | Not supported by this paper | Once perplexity is controlled, the syntactic generalization score adds no significant value [arxiv:2006.01912 §3.3] |
| You need to compare reading-time models with different tokenisers | Sub-word models with word probability as the product of its sub-word probabilities | The paper scores sub-word models this way, and perplexity is only comparable within one vocabulary [arxiv:2006.01912 §2.1] |
| You must pick a zero-shot prompt with no labels, have about 1,000 example inputs, and can read token probabilities from the model | Rank candidates by mean prompt perplexity and keep the lowest few (SPELL used k = 3) | Lower perplexity went with a higher score on most tasks for OPT and Bloom, and SPELL gained on average [arxiv:2212.04037 tab. 5] [arxiv:2212.04037 §7.1] |
| The model or task differs from the paper's scope | Treat the prompt method as untested | Only OPT and Bloom, mostly English, human-readable prompts were tested [arxiv:2212.04037 Limitations] |
| You want to know whether a prompt appeared in pretraining data | Not this method | The paper uses perplexity only as a proxy for familiarity [arxiv:2212.04037 §2] |

## Implementation sketch

The reading-time variants compute word surprisal from sub-word probabilities,
then score a model by its held-out gain over a controls-only regression. The
prompt variant averages sequence perplexity over task inputs. Pseudocode in our
own words, with the papers' citations.

```pseudocode
# Word surprisal in bits, from a model's sub-word token probabilities
# [arxiv:2006.01912 §2.1] [arxiv:2006.01912 §2.2]
# Input:  probabilities of the sub-word tokens of one word
# Output: word surprisal in bits
def word_surprisal(subword_probs):
    # Word probability: product of its sub-word probabilities
    word_prob = product(subword_probs)
    # Surprisal is the negative log2 of the word probability
    return -log2(word_prob)
```

```pseudocode
# Predictive power of one model on one reading corpus
# [arxiv:2006.01912 §2.2]
# Input:  one row per word (reading time, controls, surprisals); 10 folds
# Output: mean per-token ΔLogLik on held-out folds
def predictive_power(rows, folds):
    deltas = []
    for fold in folds:
        # Controls-only baseline, fit on the training folds
        baseline = fit_linear(fold.train, controls)
        # Full model adds the surprisal terms, fit on the same folds
        full = fit_linear(fold.train, controls + surprisals)
        # Per-token gain in held-out log-likelihood
        deltas.extend(
            token_log_lik(full, fold.test)
            - token_log_lik(baseline, fold.test)
        )
    return mean(deltas)
```

```pseudocode
# Mean perplexity of one prompt over unlabelled task inputs
# [arxiv:2212.04037 §2] [arxiv:2212.04037 §4.2]
# Input:  prompt (str), inputs (about 1,000 task inputs, no labels)
# Output: mean of the per-instance perplexities
def mean_prompt_perplexity(prompt, inputs):
    values = []
    for x in inputs:
        # Classification: input first, then prompt; the label is left out
        # [arxiv:2212.04037 §2]
        text = x + " " + prompt
        # Perplexity of this one instance [arxiv:2212.04037 §4.2]
        values.append(perplexity(text))
    # Average the per-instance values [arxiv:2212.04037 §4.2]
    return mean(values)
```

Choose the spillover window explicitly. The reading-time paper uses the
current word and the previous word for eye-tracking, and the current word and
the previous three words for self-paced reading [arxiv:2006.01912 §2.2]. The
released code does not match this in every case; see Contradictions.

## Contradictions

- Earlier studies disagreed about whether explicit syntactic structure helps
  once perplexity is accounted for. The reading-time paper reports this
  disagreement but does not resolve it [arxiv:2006.01912 §1]. Those earlier
  studies have no card here, so this page states the disagreement only.
- Paper text versus released code, for the reading-time surprisal window. The
  text gives the current word and one or three previous words
  [arxiv:2006.01912 §2.2]. The headline linear model uses two previous words
  for eye-tracking and four for self-paced reading
  [code: scripts/analysis.Rmd#L193-L194]. The GAM for the figure uses one and
  three [code: scripts/analysis.Rmd#L649-L659]. The paper's headline numbers
  are the ones the linear model produces, so the window behind them is not
  the one the text describes.
- The reading-time paper's mixed-model random-effects structure agrees with the
  code [arxiv:2006.01912 §3.2] [code: scripts/analysis.Rmd#L564].
- SPELL's average OPT gain, as stated in the text, and the average of its
  per-task gains in the table differ. The text gives 1.8 points
  [arxiv:2212.04037 §7.1]. The seven OPT per-task differences in the table
  average 1.73 [arxiv:2212.04037 tab. 10]. The paper does not explain the
  difference. The Bloom figures agree: 2.3 in the text, and 2.29 from the table.

## Papers

- [Wilcox et al., 2020](/papers/2006.01912): compares 5-gram, LSTM, RNNG and Transformer surprisal against three reading-time corpora, and finds predictive power rises as perplexity falls.
- [Gonen et al., 2022](/papers/2212.04037): finds that lower-perplexity zero-shot prompts mostly score better on OPT and Bloom, and proposes SPELL, which selects prompts by perplexity without labels.