Language model surprisal

Overview

Surprisal is the negative log of the probability a language model gives a token after the text before it. Two uses are covered here, and they score different things.

The first is reading time. Human reading slows on words that are hard to predict from the text before them, so a model's word surprisal can stand in for that difficulty 2006.01912 §1. The shared measure is the held-out gain in log-likelihood that surprisal adds to a regression of reading time on word length and frequency, reported as ΔLogLik per token 2006.01912 §2.2. The invariant across these variants is the baseline: each score is a difference from a model that uses only length and frequency controls 2006.01912 §2.2.

The second is prompt selection. A prompt is scored by the perplexity of its text under the model, averaged over task inputs, with no label included. The authors use perplexity as a proxy for how familiar the prompt wording is to the model 2212.04037 §2. This variant has no baseline. Candidate prompts are ranked against each other on the same inputs 2212.04037 §7.

Variants

Word surprisal from an n-gram model

Reading-time variant. A 5-gram model with Kneser-Ney smoothing, built with SRILM 2006.01912 §2.1. Its perplexity is about twice the neural models' on the same data, yet its ΔLogLik is comparable or higher on Dundee, BLLIP-SM and BLLIP-XS 2006.01912 §3.2. Card: Wilcox et al., 2020.

Word surprisal from an LSTM

Reading-time variant. An LSTM language model, trained with a boilerplate PyTorch implementation; the paper does not state its size or training schedule 2006.01912 §2.1. The paper reports that Transformers and 5-grams beat LSTMs at a given perplexity 2006.01912 §4. Card: Wilcox et al., 2020.

Word surprisal from a recurrent neural network grammar

Reading-time variant. An RNNG models words and their syntactic structure jointly, trained on Penn Treebank-style constituency parses 2006.01912 §2.1. The paper's training details for it are not stated 2006.01912 §2.1. Card: Wilcox et al., 2020.

Word surprisal from a Transformer

Reading-time variant. A GPT-2 architecture trained from scratch on BLLIP subsets, with 12 layers 2006.01912 §2.1 2006.01912 §4. Some Transformers use byte-pair encoding, so they predict sub-word tokens 2006.01912 §2.1. Transformers trained on larger corpora with byte-pair encoding get the highest scores on all three corpora 2006.01912 §4. Card: Wilcox et al., 2020.

Mean prompt perplexity for prompt selection

Prompt-selection variant. For each zero-shot prompt, the paper instantiates it with about 1,000 task inputs. For classification, the input comes first and the prompt follows. The label is left out. The perplexity of each instance is computed, and the values are averaged 2212.04037 §2 2212.04037 §4.2. The paper gives no formula for perplexity, and it does not state tokenisation, a start token or the context window 2212.04037 §2.

The hypothesis is that, averaged over tasks and models, a lower-perplexity prompt performs better 2212.04037 §2. Across OPT 175B, OPT 30B, OPT 1.3B and Bloom 176B, most correlations are negative and significant on the seven classification tasks, the antonym task and the word-translation task. The exceptions include IMDB with OPT 175B, which is positive 2212.04037 tab. 5. The authors also tested prompt length as a predictor, which gives weak positive correlations, mostly not significant 2212.04037 §5.

The authors turn the hypothesis into SPELL. It paraphrases seed prompts with GPT-3 text-davinci-002 using seven meta-prompts, backtranslates them through NLLB languages, ranks the candidates by mean perplexity and keeps the three lowest 2212.04037 §3 2212.04037 §7. The average accuracy gain over the manual prompts is 1.8 points with OPT and 2.3 with Bloom 2212.04037 §7.1. The per-task gains vary: the AG News gain with Bloom is −12.5 points, and the DBpedia gain with OPT is −5.7 2212.04037 tab. 10. Card: Gonen et al., 2022.

When to use which

Implementation sketch

The reading-time variants compute word surprisal from sub-word probabilities, then score a model by its held-out gain over a controls-only regression. The prompt variant averages sequence perplexity over task inputs. Pseudocode in our own words, with the papers' citations.

# Word surprisal in bits, from a model's sub-word token probabilities
# [arxiv:2006.01912 §2.1] [arxiv:2006.01912 §2.2]
# Input:  probabilities of the sub-word tokens of one word
# Output: word surprisal in bits
def word_surprisal(subword_probs):
    # Word probability: product of its sub-word probabilities
    word_prob = product(subword_probs)
    # Surprisal is the negative log2 of the word probability
    return -log2(word_prob)

# Predictive power of one model on one reading corpus
# [arxiv:2006.01912 §2.2]
# Input:  one row per word (reading time, controls, surprisals); 10 folds
# Output: mean per-token ΔLogLik on held-out folds
def predictive_power(rows, folds):
    deltas = []
    for fold in folds:
        # Controls-only baseline, fit on the training folds
        baseline = fit_linear(fold.train, controls)
        # Full model adds the surprisal terms, fit on the same folds
        full = fit_linear(fold.train, controls + surprisals)
        # Per-token gain in held-out log-likelihood
        deltas.extend(
            token_log_lik(full, fold.test)
            - token_log_lik(baseline, fold.test)
        )
    return mean(deltas)

# Mean perplexity of one prompt over unlabelled task inputs
# [arxiv:2212.04037 §2] [arxiv:2212.04037 §4.2]
# Input:  prompt (str), inputs (about 1,000 task inputs, no labels)
# Output: mean of the per-instance perplexities
def mean_prompt_perplexity(prompt, inputs):
    values = []
    for x in inputs:
        # Classification: input first, then prompt; the label is left out
        # [arxiv:2212.04037 §2]
        text = x + " " + prompt
        # Perplexity of this one instance [arxiv:2212.04037 §4.2]
        values.append(perplexity(text))
    # Average the per-instance values [arxiv:2212.04037 §4.2]
    return mean(values)

Choose the spillover window explicitly. The reading-time paper uses the current word and the previous word for eye-tracking, and the current word and the previous three words for self-paced reading 2006.01912 §2.2. The released code does not match this in every case; see Contradictions.

Contradictions

• Earlier studies disagreed about whether explicit syntactic structure helps once perplexity is accounted for. The reading-time paper reports this disagreement but does not resolve it 2006.01912 §1. Those earlier studies have no card here, so this page states the disagreement only.
• Paper text versus released code, for the reading-time surprisal window. The text gives the current word and one or three previous words 2006.01912 §2.2. The headline linear model uses two previous words for eye-tracking and four for self-paced reading analysis.Rmd. The GAM for the figure uses one and three analysis.Rmd. The paper's headline numbers are the ones the linear model produces, so the window behind them is not the one the text describes.
• The reading-time paper's mixed-model random-effects structure agrees with the code 2006.01912 §3.2 analysis.Rmd.
• SPELL's average OPT gain, as stated in the text, and the average of its per-task gains in the table differ. The text gives 1.8 points 2212.04037 §7.1. The seven OPT per-task differences in the table average 1.73 2212.04037 tab. 10. The paper does not explain the difference. The Bloom figures agree: 2.3 in the text, and 2.29 from the table.

Papers

• Wilcox et al., 2020: compares 5-gram, LSTM, RNNG and Transformer surprisal against three reading-time corpora, and finds predictive power rises as perplexity falls.
• Gonen et al., 2022: finds that lower-perplexity zero-shot prompts mostly score better on OPT and Bloom, and proposes SPELL, which selects prompts by perplexity without labels.