On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

In plain words

When people read, they slow down on some words and move quickly over others. A big part of that is how predictable a word was from the text before it. A language model makes the same kind of guess, a probability for each next word, so its guesses can be compared with how people actually read.

This paper asks which kinds of language model best match human reading. The authors trained many models of different designs, on text of different sizes, and checked how well each model's surprise at each word explained reading times from three datasets. One dataset records eye movements. The other two are self-paced reading, where the reader presses a key to reveal each word in turn.

The main findings are that better word prediction goes with better fit to reading times, and that design still matters: deep Transformers and simple word-counting models did better than LSTMs at the same level of prediction. Models that passed targeted grammar tests did not explain reading times any better once their prediction quality was accounted for. A model that knows grammar well is therefore not automatically a good model of human reading.

TL;DR

The paper fits a fleet of language models of four architectures on four training-set sizes, and scores each model by how much its word surprisal improves a regression of human reading times over a baseline that uses only word length and frequency §2.1 §2.2. The score is ΔLogLik, the mean per-token gain in held-out log-likelihood §2.2. Author-reported: the score rises as perplexity falls, and Transformers and the 5-gram model do best for a given perplexity §3.2 §4. A syntactic generalization score adds no significant predictive value once perplexity is controlled §3.3. The paper's results are shown in figures, with no table of numbers.

Problem

Human reading times track word surprisal, the negative log probability of a word in context. Earlier work used n-gram models, which only see a short window of words §1. Neural models such as LSTMs and Transformers now reach much lower perplexity, but it was unclear which architecture best predicts human reading, or how training-data size affects that §1. Earlier studies disagreed about whether explicit syntactic structure helps once perplexity is accounted for §1. The paper argues that naturalistic reading and controlled grammar probes may need different models §4.

Method

The study has three parts: a fleet of language models with known perplexity §2.1, a regression that measures how much each model's surprisal explains human reading times §2.2, and a syntactic generalization score per model §2.3. Then the paper relates the three §3.

1. Language models §2.1

• LSTM-RNN: an LSTM language model, using the boilerplate PyTorch implementation §2.1.
• RNNG: a recurrent neural network grammar that models words and their syntactic structure jointly. It is trained with Penn Treebank-style constituency parses §2.1.
• Transformer: a GPT-2 architecture trained from scratch on the paper's corpora §2.1. Some Transformers use byte-pair encoding, so they predict sub-word tokens §2.1. The paper says the word probability is the product of its sub-word probabilities §2.1 fn..
• 5-gram: a 5-gram model with Kneser-Ney smoothing, built with SRILM §2.1.
• Training corpora: four nested subsets of BLLIP, from XS to LG §2.1. Each architecture and corpus pair has 1 to 3 random seeds §2.1.

2. Perplexity §2.1

Perplexity is the inverse geometric mean of the probability the model gives to each word of a test corpus. Lower is better §2.1. The paper notes perplexity is only comparable with the same reference vocabulary, so most analyses compare models on one vocabulary §2.1.

3. Surprisal §2.2

Surprisal of word x_i is S = -log2 P(x_i | x_<i), taken from the model's probability of the word given its preceding context §2.2.

# Word surprisal for a model with sub-word tokens  [§2.1 fn.] [§2.2]
# Input:  probabilities of the sub-word tokens of one word
# Output: word surprisal in bits (base 2 per the paper)
def word_surprisal(subword_probs):
    # Word probability: product of its sub-word probabilities  [§2.1 fn.]
    word_prob = product(subword_probs)
    # Surprisal is the negative log2 of the word probability  [§2.2]
    return -log2(word_prob)

4. Psychometric predictive power §2.2

• Human data: Dundee eye-tracking corpus, with gaze duration averaged across subjects per word §2.2. Self-paced reading data from the Brown corpus, and self-paced reading data from Natural Stories §2.2.
• Controls for every word: its character count, and a log-scale frequency counted on Wikitext-2 §2.2 §2.2 fn..
• Spillover: the surprisal of the current word and the previous word for eye-tracking, and of the current word and the previous three words for self-paced reading §2.2.
• Model: linear regression of reading time on the controls and surprisal §2.2. GAMs with a cubic regression spline basis (k = 20) give qualitatively similar results and are used for the figure fits §2.2 fn. §3.1.
• Baseline: the same regression without surprisal. Predictive power is ΔLogLik, the mean per-token difference in log-likelihood on held-out data between the full and baseline models, averaged over 10-fold cross-validation §2.2. A positive value means the surprisal helps §2.2.

# Predictive power of one LM on one reading corpus  [§2.2]
# Input:  rows (one per word) with rt, controls, surprisals; K = 10 folds
# Output: mean per-token ΔLogLik on held-out folds
def predictive_power(rows, folds):
    # Per-token ΔLogLik values, pooled over folds  [§2.2]
    deltas = []
    for fold in folds:
        # Controls-only baseline, fit on the training folds  [§2.2]
        baseline = fit_linear(fold.train, controls)
        # Full model adds the surprisals, fit on the same folds  [§2.2]
        full = fit_linear(fold.train, controls + surprisals)
        # Per-token gain: full log-lik minus baseline log-lik  [§2.2]
        delta = (
            token_log_lik(full, fold.test)
            - token_log_lik(baseline, fold.test)
        )
        deltas.extend(delta)
    # Average over all held-out words  [§2.2]
    return mean(deltas)

The full model's per-token log-likelihood uses a Gaussian density with mean from the model's prediction and standard deviation from the same model's training fit analysis.Rmd. Both the baseline and the full model use this rule.

5. Syntactic generalization score §2.3

Each model is scored on 34 targeted syntactic tests from earlier work §2.3. Each test has 20 to 30 items, and each item is used in several conditions, generally four §2.3. A model passes an item only if its surprisals satisfy all the inequality criteria for that item §2.3. The score is the mean accuracy over the tests, with chance near 0.25 §2.3.

6. Analyses §3

• Surprisal against reading time: GAM fits with bootstrapped 95% intervals for each model and corpus, using only the surprisal terms §3.1 fig. 1.
• Predictive power against perplexity: a mixed-effects regression of ΔLogLik on perplexity within each training vocabulary, with random intercepts for test corpus and model architecture §3.2 fig. 2.
• Syntactic generalization against predictive power: both quantities are residualized on perplexity within each training vocabulary, then compared §3.3 fig. 3 fig. 4.

Hyperparameters

Few training hyperparameters are stated in the paper. The table lists what the text and the released analysis code give.

Results

Author-reported. Setup: 10-fold cross-validation on three reading corpora (Dundee eye-tracking, Brown self-paced reading, Natural Stories self-paced reading) §2.2. The baseline is the controls-only regression §2.2. The paper reports results in figures; it gives no table of ΔLogLik values, so numbers below come from the text.

Paper vs code

The official code is a set of analysis scripts. It does not include the model training code, the surprisal outputs, the human reading-time files, the harmonized results, or the model metadata that holds perplexity and syntactic generalization scores. The README points to a Google Drive download that is not in this copy README.md. So the headline numbers cannot be recomputed from this repository.

Differences between paper and code, each with both citations:

• Eye-tracking surprisal window: the text says the current and previous word §2.2. The headline linear model uses the current word and two previous words analysis.Rmd. The GAM for Figure 1 uses one previous word analysis.Rmd.
• Self-paced reading window: the text says the current word and three previous words §2.2. The headline linear model uses four previous words analysis.Rmd analysis.Rmd. The GAM uses three analysis.Rmd.
• Linear formula in the footnote: the footnote lists surprisal up to the third previous word and interaction terms between frequency and length §2.2 fn.. The released linear model has no interactions and stops at the second previous word analysis.Rmd.
• Dundee measure: the text says gaze duration §2.2. The harmonization notes list "firstpass RT, SPR measure" for this column harmonize.ipynb. The Dundee file is not in the repository, so the measure cannot be confirmed.
• Model list: the code filters out an "ordered-neurons" model before any analysis analysis.Rmd. The paper names four architectures and does not mention this model §2.1.
• Row filters: the code keeps a token only when the row four back (two back for Dundee) has the expected code, so the history is contiguous analysis.Rmd. It then drops any token that is missing for some model, has zero surprisal for some model, or has zero reading time analysis.Rmd. The paper does not describe these rules.
• UNK tokens: when a model's token is an UNK token and does not match the reading-time word, the harmonization code skips that word harmonize.py. Contractions are merged, and their surprisals summed harmonize.py. The paper does not describe either rule.
• Cross-validation seed: the folds come from an unseeded random shuffle analysis.Rmd. Fold assignment, and therefore the standard errors, will differ on each run.
• Frequency and length: frequency is the natural log of the Wikitext-2 count plus one, and length is the number of characters harmonize.ipynb. The paper says "log frequency" without giving the base or the add-one §2.2. A word missing from the counts gets frequency log(1) = 0 harmonize.ipynb.
• Chance level in the figure: the code draws the dashed "random" line at 0.28 analysis.Rmd. The text gives chance as about 0.25 §2.3.
• Plot-only clipping: perplexities above 500 are set to 329.9 for display analysis.Rmd analysis.Rmd. The fits do not use this step.
• GAM resampling: the released GAM code uses 50 bootstrap resamples analysis.Rmd, and it is disabled with eval = FALSE analysis.Rmd. The README says the GAM takes over an hour per fit set README.md.
• Surprisal base: the paper says log base 2 §2.2. The surprisal values come from external model outputs that are not in this repository, so the base cannot be confirmed from the code.

The mixed model agrees with the text: random intercepts for corpus and architecture §3.2 analysis.Rmd. The per-vocabulary slopes come from its interaction term.

The code has no license file, so it is cited by path and line only. No snippet is quoted.

Pitfalls

• The surprisal window is ambiguous: the text, the headline linear model and the GAM each use a different number of previous words. Pick one explicitly and say which. The linear model is the one behind the headline ΔLogLik values.
• ΔLogLik is not a joint log-likelihood. It is a per-token mean over held-out words, so it is not comparable to the joint measure in earlier work §2.2.
• Each model's own residual standard deviation enters the density, for both baseline and full model. A model that fits better also gets a smaller standard deviation, so the comparison mixes fit and scale analysis.Rmd.
• The filters that decide which words survive are not in the paper. A reimplementation that skips them will change the sample and the numbers analysis.Rmd analysis.Rmd harmonize.py.
• Fold assignment is not seeded, so results vary between runs even with the same code analysis.Rmd.
• Perplexity is comparable only with a shared vocabulary §2.1. Sub-word models get a word probability by multiplying sub-word probabilities §2.1 fn.. Analyses that pool training vocabularies mix incomparable numbers, so this paper analyses each vocabulary separately §3.2 §3.3.
• The syntactic generalization score comes from earlier work and is taken as given §2.3. Its per-model values are not in the repository analysis.Rmd.
• Reproduction is costly: the README says the GAM fits for Figure 1 take over an hour, and the 10-fold regressions are run per model, seed, corpus and fold README.md analysis.Rmd.
• The LaTeX source contains a commented-out sentence saying the BPE was taken from GPT-2. That sentence is not in the published text, so this card does not rely on it.

Lineage

• Builds on: the surprisal-to-reading-time link studied by Smith and Levy (2013), and the perplexity-versus-fit debate of Frank and Bod (2011) and Fossum and Levy (2012). Goodkind and Bicknell (2018) found that fit tracks perplexity, and this paper extends that line to neural models §1 §2.2. The fleet of models and the syntactic tests follow Hu et al. (2020) §1 §2.1 §2.3.
• Compared against: Goodkind and Bicknell (2018), whose LSTM underperformed their n-gram models, and Aurnhammer and Frank (2019) and Merkx and Frank (2020), who compare gated RNNs or Transformers on reading data §1 §3.2 §4. This paper reports held-out ΔLogLik instead of the joint log-likelihood that Goodkind and Bicknell use §2.2.
• Superseded by: not known from content already in this wiki.