wikillm

Paper · 2023-04-28 · cs.CL

Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4

Probes which books two OpenAI models (gpt-3.5-turbo and gpt-4) have memorized with a name-cloze task: predict a masked proper name in a 40-60 token passage from 571 fiction works. Author-reported: GPT-4 scores 0.244 name-cloze accuracy on pre-1923 books, memorization follows web prevalence (Spearman up to 0.84), and memorized books give far better year-of-publication estimates.

Membership inferenceBenchmark contaminationTraining data extractionDataset curationmemorizationname-clozemembership-inferencedata-contaminationclosed-modelscultural-analyticsgpt-4chatgpt

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Chatbots are trained on huge piles of text, but the companies do not say which books are in those piles. The authors work out which books a model knows by testing its memory. They take a short passage from a novel, hide one character's name, and ask the model to fill the blank. A model that has read the book tends to produce the right name. A model that has not is guessing. Running this over hundreds of novels gives a rough map of which books the model has absorbed.

The map is uneven. Classic public-domain novels, popular science fiction and fantasy, and bestsellers are known best, and how often a book's text appears on the web tracks how well it is known. This matters for anyone who tests a model on well-known texts: if the model has memorized those texts, the test measures memory rather than skill. The same memory also changes how well the model does on related tasks, such as guessing a book's year of publication. The authors argue that only models whose training data is published can settle these questions.

TL;DR

The paper runs a name-cloze memory test on gpt-3.5-turbo and gpt-4, the models behind ChatGPT and GPT-4, over 571 works of fiction, with 100 sampled passages per book §3.2 §4. The authors report that GPT-4 knows many books still in copyright, most of them science fiction or fantasy §5.1. Name-cloze accuracy tracks how often passages appear on the web §6.2. Books the models know give much better year predictions §7.1, while the narrative-time difference is not statistically significant §7.2. Use the card for the probe design and for the contamination argument; treat the numbers as author-reported.

Problem

Closed models do not disclose their training data, so a researcher cannot tell whether a benchmark text was seen in training §1 §8. Perplexity-style tests were not available either, because the OpenAI API did not share word probabilities at the time of writing §3.1. The authors therefore use a task whose answer cannot be inferred from the passage alone: the passage contains no other names, and a human who only reads it gets 0% §3.1 §3.2. If a model recovers the name, it must know something about the book. Memorized books can then contaminate evaluations in cultural analytics, because the model's performance on them is no longer a measure of generalisation §1 §8.

Method

Name-cloze probe

  • Passage. Each candidate passage is 40 to 60 tokens, holds one single-token person name as the answer, contains no other named entity, and consists of complete sentences §3.1 §3.2. The authors run BookNLP to find entities §3.2.
  • Sample. For each book, 100 passages are drawn at random. Books with fewer than 100 usable passages are dropped from the analysis §3.2.
  • Prompt. One user message: an instruction that the answer is exactly one word and a proper name, two worked examples with their answers in name tags, then the passage with the masked token. The model must guess even when unsure fig. 2.
  • Scoring. A book's accuracy is the fraction of its sampled passages where the model's name matches the masked name §5.1. The paper does not define the matching rule (see Pitfalls).
  • Baselines. Predicting the most frequent name in the set ("Mary") gives 0.6%. One author answered the same prompt by hand, using only the passage, and scored 0% §3.2. BERT receives the passage without the instruction and is scored the same way §5.2.

Book sample

Five sources of English fiction are used, 571 works in total §4:

  • 91 novels from LitBank, published before 1923 (public domain);
  • 90 Pulitzer Prize nominees, 1924 to 2020;
  • 95 bestsellers, drawn from NY Times and Publishers Weekly lists, 1924 to 2020;
  • 101 novels by Black authors, from two collections, 1928 to 2018;
  • 95 works of Global Anglophone fiction, 1935 to 2020;
  • 99 genre novels (science fiction and fantasy, horror, mystery and crime, romance, action and spy), 1928 to 2017.

Text from the post-1923 sources was printed, scanned and OCR'd with ABBYY FineReader §4.

Memorization groups

Books are ranked by GPT-4 name-cloze accuracy. The top decile is treated as memorized (M), the bottom decile as not memorized (not-M), and a held-out set H is kept for testing §6.1. The paper also uses top and bottom quintiles for the narrative-time task §7.2.

Name-level test of seen versus unseen names

For each name the true masked answer is linked to a book by its positive pointwise mutual information. Names tied to M form set S, names tied to not-M form set U. Both sets have equal size and no overlap §6.1. On the error cases in H, the observed statistic is a log-odds ratio of predicting a name in S versus in U:

o = \log\left(\frac{\Pr(\hat{c} \in S \mid \text{error})}{\Pr(\hat{c} \in U \mid \text{error})}\right) §6.1

Here \hat{c} is the predicted character name. Significance comes from a randomization test that reshuffles names between S and U §6.1.

Pseudocode
# Shuffle-based significance for the name-level test  [§6.1]
# Input:  observed log-odds o, sets S and U, error predictions on H
# Output: p-value (shuffle count not stated in the paper)

def randomization_p_value(o, S, U, predictions):
    # Recompute the statistic after shuffling names between S and U  [§6.1]
    null = []
    for shuffle in shuffled_assignments(S, U):
        null.append(log_odds(predictions, shuffle))
    # Share of null values at least as large as the observed statistic
    return count_at_least(null, o) / len(null)

Web prevalence

For each book, 10 passages are sampled, and from each a 10-token span is taken. The span is searched as an exact string in Google, Bing, C4 and the Pile, and the hit counts are averaged per book. The paper then reports Spearman ρ between those counts and GPT-4 accuracy, separately for works published before and after 1928 §6.2 tab. 3.

Name cloze, passage builder

The pseudocode below follows the authors' generator. The paper gives the window length and the no-other-names rule; the repo adds further filters, marked in the comments.

Pseudocode
# Build name-cloze items from one book  [§3.1] [§3.2]
# Input:  tokens (list of str), sentence_ids (one per token),
#         proper_spans (list of (start, end) token indices)
# Output: list of (masked_passage, answer) pairs

def make_items(tokens, sentence_ids, proper_spans):
    items = []
    # Candidate answers are single-token person names  [§3.2]
    for index in single_token_person_indices(proper_spans):
        # Whole sentences around the name, 40 to 60 tokens, nearest 50.
        # The repo uses strict bounds and checks proper nouns only.
        # [code: generate_name_cloze/create_name_cloze_from_booknlp.py#L34-L73]
        window = closest_window(tokens, sentence_ids, index)
        if window is None:
            continue
        answer = tokens[index]
        # Repo-only filters: kinship and deity words, repeated answer.
        # [code: generate_name_cloze/create_name_cloze_from_booknlp.py#L15]
        # [code: create_name_cloze_from_booknlp.py#L167-L175]
        if answer_is_excluded(answer, window):
            continue
        items.append((mask(window, index), answer))
    return items
Pseudocode
# Ask one chat model for the masked name  [§3.2] [fig. 2]
# Input:  model (chat client), passage (str containing [MASK])
# Output: raw reply text

def ask_name(model, passage):
    # Instruction, two worked examples, then the passage  [fig. 2]
    prompt = build_name_prompt(passage)
    # Official script: one user message, temperature 0
    # [code: scripts/openai_predict_name_cloze.py#L25-L37]
    reply = model.chat(
        messages=[{"role": "user", "content": prompt}],
        # Temperature is not stated in the paper
        temperature=0.0,
    )
    return reply
Pseudocode
# Accuracy of one book  [§5.1]
# Input:  model, pairs (up to 100 (passage, answer) per book)  [§3.2]
# Output: fraction of pairs answered correctly

def book_accuracy(model, pairs):
    hits = 0
    for passage, answer in pairs:
        # Remove the name tags from the reply
        # [code: data/model_output/README.md]
        guess = strip_tags(ask_name(model, passage))
        # Exact string match is our reading; the paper does not state one
        if guess == answer:
            hits += 1
    return hits / len(pairs)

Year of first publication

A random 250-word passage (complete sentences) from each book goes into a few-shot prompt. GPT-4 is asked for a four-digit year, and may answer that the year is uncertain §7.1 fig. 3. The paper records how often the model refrains and the absolute error |ŷ − y| for each valid answer. Books are compared by top and bottom decile of GPT-4 name-cloze accuracy §7.1.

Narrative time

The same 250-word passages are annotated by hand, blind to the book and to the group, using the criteria of Underwood (2023), and given to GPT-4 with a five-part prompt: summarize, reason about elapsed time, take the midpoint, give minutes, and state certainty §7.2 fig. 4. Accuracy is Spearman ρ between predicted and annotated minutes, with a bootstrap 95% confidence interval. The groups are the top and bottom deciles and the top and bottom quintiles §7.2.

Hyperparameters

NameValueWhereNotes
Modelsgpt-3.5-turbo (ChatGPT), gpt-4 (GPT-4)§5.1 openai_predict_name_cloze.pyDated snapshots not stated; the script keeps the GPT-4 name commented out
InterfaceOne user message, no system messageopenai_predict_name_cloze.pyPaper says only "OpenAI API" §5.1
Temperature0.0openai_predict_name_cloze.pyNot stated in the paper; the year and time scripts also use 0.0 openai_predict_year.py
Passage length40 to 60 tokens§3.1 §3.2Repo: target 50, strict bounds 40 < n < 60 create_name_cloze_from_booknlp.py
Passages per book100, random§3.2Books with fewer usable passages are excluded §3.2
Passage sentencesComplete sentences, no crossing sentence boundary§3.2Implemented by sentence-id windows create_name_cloze_from_booknlp.py
Entity taggerBookNLP, pipeline entity, model big§3.2 run_booknlp.pyPaper cites BookNLP only
Excluded answers14 kinship and deity wordscreate_name_cloze_from_booknlp.pyNot in paper
Most-frequent-name baseline"Mary", 0.6%§3.2
Human baselineOne author, 0%§3.2Single annotator
Year prompt250-word passage, four-digit year, "uncertain" allowed§7.1 fig. 3571 rows in the repo data file year.passages.txt
Narrative-time passage250 words, hand-annotated minutes§7.2 fig. 4228 rows in the repo data file narrative.time.passages.txt
Decile and quintile groupsTop and bottom 10% and 20% by GPT-4 name-cloze accuracy§6.1 §7.1 §7.2
Error-analysis groupsM = top decile, not-M = bottom decile, H held out§6.1Shuffle count not stated
Confidence intervalBootstrap, 95%§7.2Resample count not stated
Web-search sample10 passages per book, 10-token spans§6.2Search API details §6.2
API costAbout $400; $0.002 per 1K tokens (ChatGPT), $0.03 per 1K tokens (GPT-4)§5.1Prices as of the paper

Results

Author-reported. Setup: 571 works of fiction §4; ChatGPT and GPT-4 queried through the OpenAI API with the same name-cloze prompt on every sampled passage §5.1; BERT given the passage without instructions §5.2.

Headline results:

  • The most memorized book is Alice's Adventures in Wonderland, at GPT-4 accuracy 0.98 tab. 1.
  • Twelve of the 20 most memorized in-copyright books are science fiction or fantasy §5.1.
  • Name-cloze accuracy on the pre-1923 LitBank books is 0.244 for GPT-4 and 0.072 for ChatGPT tab. 2.
  • BERT is near zero on almost every listed book. The exception is Fifty Shades of Grey at 0.13 tab. 1 §5.2.
  • Error analysis: o = 1.34 for ChatGPT and o = 1.37 for GPT-4, both with p < 0.0001 §6.1.
  • Web prevalence: pre-1928 Spearman ρ is 0.74 (Google), 0.70 (Bing), 0.71 (C4) and 0.84 (Pile). Post-1928 values are weaker, between 0.21 and 0.41 tab. 3.
  • Year of first publication: mean absolute error on top-decile books is 3.3 years for ChatGPT and 0.6 years for GPT-4. On bottom-decile books it is 29.8 and 14.5 years tab. 5. Refusals run 4% (top) against 9% (bottom) for ChatGPT. GPT-4 never refused §7.1.
  • Narrative time: ρ = 0.50 [0.25, 0.72] for TOP10 and 0.27 [−0.02, 0.59] for BOT10, not significant. For the quintiles, 0.47 and 0.46 §7.2.
  • Cost: about $400 for the full name-cloze run §5.1.

Table 1: the 20 highest GPT-4 scores tab. 1

GPT-4ChatGPTBERTDateAuthorTitle
0.980.820.001865Lewis CarrollAlice's Adventures in Wonderland
0.760.430.001997J.K. RowlingHarry Potter and the Sorcerer's Stone
0.740.290.001850Nathaniel HawthorneThe Scarlet Letter
0.720.110.001892Arthur Conan DoyleThe Adventures of Sherlock Holmes
0.700.100.001815Jane AustenEmma
0.650.190.001823Mary W. ShelleyFrankenstein
0.620.130.001813Jane AustenPride and Prejudice
0.610.350.001884Mark TwainAdventures of Huckleberry Finn
0.610.300.001853Herman MelvilleBartleby, the Scrivener
0.610.080.001897Bram StokerDracula
0.610.180.001838Charles DickensOliver Twist
0.590.130.001902Arthur Conan DoyleThe Hound of the Baskervilles
0.590.220.001851Herman MelvilleMoby Dick; Or, The Whale
0.580.350.001876Mark TwainThe Adventures of Tom Sawyer
0.570.300.001949George Orwell1984
0.540.100.001908L. M. MontgomeryAnne of Green Gables
0.510.200.011954J.R.R. TolkienThe Fellowship of the Ring
0.490.160.132012E.L. JamesFifty Shades of Grey
0.490.240.011911Frances H. BurnettThe Secret Garden
0.490.120.001883Robert L. StevensonTreasure Island
0.490.160.001847Charlotte BrontëJane Eyre: An Autobiography
0.490.220.001903Jack LondonThe Call of the Wild

Table 2: name-cloze accuracy by book category tab. 2

SourceGPT-4ChatGPT
pre-1923 LitBank0.2440.072
Genre: SF/Fantasy0.2350.108
Genre: Horror0.0540.028
Bestsellers0.0330.016
Genre: Action/Spy0.0320.007
Genre: Mystery/Crime0.0290.014
Genre: Romance0.0290.011
Pulitzer0.0260.011
Global0.0200.009
BBIP/BCALA0.0170.011

Table 3: Spearman ρ between GPT-4 accuracy and web hit counts tab. 3

DateGoogleBingC4Pile
pre-19280.740.700.710.84
post-19280.370.410.360.21

Sites that return copyrighted text tab. 4

The site with the most hits for copyrighted-text snippets is archive.org, at 337. The rest of the top ten mix full-text hosts with sites that carry snippets in reviews and study guides, such as Goodreads, Amazon and LitCharts §6.2 tab. 4.

Table 5: year of first publication, top and bottom 10% tab. 5

Refusal is the share of books where the model did not give a year. The bracketed values are 95% bootstrap intervals around MAE.

ModelRefusal top 10%Refusal bottom 10%ρMAE top 10%MAE bottom 10%
ChatGPT0.040.09−0.393.3 [0.1–12.3]29.8 [20.2–41.2]
GPT-40.000.00−0.370.6 [0.1–1.5]14.5 [9.5–20.8]
GPT-4ChatGPTBERTDateAuthorTitle
0.760.430.001997J.K. RowlingHarry Potter and the Sorcerer's Stone
0.570.300.001949George Orwell1984
0.510.200.011954J.R.R. TolkienThe Fellowship of the Ring
0.490.160.132012E.L. JamesFifty Shades of Grey
0.480.140.002008Suzanne CollinsThe Hunger Games
0.430.270.001954William GoldingLord of the Flies
0.430.170.001979Douglas AdamsThe Hitchhiker's Guide to the Galaxy
0.300.160.001959Chinua AchebeThings Fall Apart
0.280.120.001977J. R. R. Tolkien and Christopher TolkienThe Silmarillion
0.270.130.001953Ray BradburyFahrenheit 451
0.270.130.001996George R.R. MartinA Game of Thrones
0.260.050.012003Dan BrownThe Da Vinci Code
0.260.080.001965Frank HerbertDune
0.250.200.011937Zora Neale HurstonTheir Eyes Were Watching God
0.250.140.001961Harper LeeTo Kill a Mockingbird
0.240.030.031953Ian FlemingCasino Royale
0.220.130.001984William GibsonNeuromancer
0.200.100.001985Orson Scott CardEnder's Game
0.190.120.001932Aldous HuxleyBrave New World
0.180.070.001937Margaret MitchellGone with the Wind
0.170.050.001968Philip K. DickDo Androids Dream of Electric Sheep?
0.160.060.052009Dan BrownThe Lost Symbol
0.150.040.042013Dan BrownInferno
0.150.080.012014Veronica RothDivergent
0.150.040.001940John SteinbeckThe Grapes of Wrath
0.130.050.001983James KahnReturn of the Jedi
0.130.020.011928D. H. LawrenceLady Chatterley's Lover
0.130.030.001977Alex HaleyRoots
0.110.110.001961Irving StoneThe Agony and the Ecstasy
0.110.010.031957Ian FlemingFrom Russia with Love
0.110.040.001962Madeleine L'EngleA Wrinkle in Time
0.110.040.011939Marjorie Kinnan RawlingsThe Yearling
0.100.050.001975E. L. DoctorowRagtime
0.100.050.001929Dashiell HammettThe Maltese Falcon
0.100.080.071991Diana GabaldonOutlander
0.100.020.001989Kazuo IshiguroThe Remains of the Day
0.100.010.001983Alice WalkerThe Color Purple
0.090.020.001934Dorothy L. SayersThe Nine Tailors
0.090.030.001985Margaret AtwoodThe Handmaid's Tale
0.090.040.001988Toni MorrisonBeloved
0.080.070.001982Joe NazelEvery Goodbye Ain't Gone
0.080.060.011984Helen Hooven Santmyer...And Ladies of the Club
0.080.020.002006Max BrooksWorld War Z
0.080.020.001993Irvine WelshTrainspotting
0.080.030.001947Robert Penn WarrenAll the King's Men
0.070.020.001952Ralph EllisonInvisible Man
0.070.010.001951Isaac AsimovFoundation
0.070.010.001976Anne RiceInterview with the Vampire
0.070.040.021977Stephen KingThe Shining
0.070.010.001996Helen FieldingBridget Jones's Diary

Paper vs code

The official repository is bamman-group/gpt4-books, linked from the paper's footnote and conclusion §5.1 §9. Its owner is the authors' lab, and it contains no LICENSE file, so its code is recorded as unlicensed: nothing is copied from it. It was searched by reading the paper's links, which lead straight to it. No other repository was searched.

No other discrepancies were found.

Pitfalls

  • Matching rule. The paper does not define when a reply counts as correct. Exact string match after tag removal is the natural reading, but it does not allow for case, accents or spacing variants. State the rule you use.
  • Snapshots and API. The model versions are not dated. The authors measured "at the time of writing" §3.1. The official script uses the old openai.ChatCompletion.create interface openai_predict_name_cloze.py. Port it to a current client rather than copying it. Expect results to drift as hosted models change.
  • Cost. The full run cost about $400 at 2023 prices §5.1. The count of requests is derived here, not reported: 571 works, 100 passages each, two models, which is at most 114,200 calls §4 §3.2.
  • Thin books. Books with fewer than 100 usable passages are dropped, and the number dropped is not stated §3.2.
  • Token definition. Lengths are counted in BookNLP tokens. A different tokenizer changes which windows pass the 40 to 60 test.
  • Circular groups. The memorization deciles come from the same GPT-4 accuracy that the year and time results are then compared against §7.1. The comparison is therefore not independent of the measure.
  • Narrative time is inconclusive. The authors state that the TOP10 and BOT10 intervals overlap and that the quintile results show no effect §7.2. A small annotated set (228 rows in the repo file) gives wide intervals.
  • Single human baseline. The 0% human score comes from one annotator §3.2. Inter-annotator agreement for the narrative-time labels is not stated.
  • Web counts are correlations. The authors argue popularity drives both web prevalence and memorization, and say the link is not causal §6.2. The post-1928 correlations are weaker, 0.21 to 0.41 tab. 3.
  • BERT is not like-for-like. BERT receives the passage without the instruction §5.2, so its numbers are not a direct comparison.
  • OCR noise. Post-1923 books were OCR'd, and the paper does not measure how errors affect the cloze §4.
  • Permission for data. The repository ships passages from in-copyright books in its data files year.passages.txt, and it has no LICENSE. Do not redistribute those excerpts. Rebuild passages from text you have the right to use.
  • Randomization and bootstrap settings. Shuffle counts and resample counts are not stated §6.1 §7.2. Pick them and report them.

Lineage

  • Builds on: the notion of exact memorization §3.1, the membership-inference framing §1, the link between duplication and memorization §6.2, and the narrative-time framework and prompt of Underwood (2023) §7.2. None of these has a card in this wiki.
  • Compared against: BERT as a plain language-model baseline §5.2.
  • Superseded by: not stated in the paper or the repository.

Source and license

Adapted from “Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4” by Kent K. Chang, Mackenzie Cramer, Sandeep Soni, David Bamman (arXiv:2305.00118), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/bamman-group/gpt4-books at commit ef7b1a6c2c08 (none (no LICENSE file in repository)). Any quoted code is shown under that license.