Paper · 2023-04-28 · cs.CL
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Probes which books two OpenAI models (gpt-3.5-turbo and gpt-4) have memorized with a name-cloze task: predict a masked proper name in a 40-60 token passage from 571 fiction works. Author-reported: GPT-4 scores 0.244 name-cloze accuracy on pre-1923 books, memorization follows web prevalence (Spearman up to 0.84), and memorized books give far better year-of-publication estimates.
Membership inferenceBenchmark contaminationTraining data extractionDataset curationmemorizationname-clozemembership-inferencedata-contaminationclosed-modelscultural-analyticsgpt-4chatgpt
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Chatbots are trained on huge piles of text, but the companies do not say which books are in those piles. The authors work out which books a model knows by testing its memory. They take a short passage from a novel, hide one character's name, and ask the model to fill the blank. A model that has read the book tends to produce the right name. A model that has not is guessing. Running this over hundreds of novels gives a rough map of which books the model has absorbed.
The map is uneven. Classic public-domain novels, popular science fiction and fantasy, and bestsellers are known best, and how often a book's text appears on the web tracks how well it is known. This matters for anyone who tests a model on well-known texts: if the model has memorized those texts, the test measures memory rather than skill. The same memory also changes how well the model does on related tasks, such as guessing a book's year of publication. The authors argue that only models whose training data is published can settle these questions.
TL;DR
The paper runs a name-cloze memory test on gpt-3.5-turbo and gpt-4, the
models behind ChatGPT and GPT-4, over 571 works of fiction, with 100 sampled passages per book §3.2 §4. The
authors report that GPT-4 knows many books still in copyright, most of them
science fiction or fantasy §5.1. Name-cloze accuracy tracks how often
passages appear on the web §6.2. Books the models know give much better year
predictions §7.1, while the narrative-time difference is not statistically
significant §7.2. Use the card for the probe design and for the contamination
argument; treat the numbers as author-reported.
Problem
Closed models do not disclose their training data, so a researcher cannot tell whether a benchmark text was seen in training §1 §8. Perplexity-style tests were not available either, because the OpenAI API did not share word probabilities at the time of writing §3.1. The authors therefore use a task whose answer cannot be inferred from the passage alone: the passage contains no other names, and a human who only reads it gets 0% §3.1 §3.2. If a model recovers the name, it must know something about the book. Memorized books can then contaminate evaluations in cultural analytics, because the model's performance on them is no longer a measure of generalisation §1 §8.
Method
Name-cloze probe
- Passage. Each candidate passage is 40 to 60 tokens, holds one single-token person name as the answer, contains no other named entity, and consists of complete sentences §3.1 §3.2. The authors run BookNLP to find entities §3.2.
- Sample. For each book, 100 passages are drawn at random. Books with fewer than 100 usable passages are dropped from the analysis §3.2.
- Prompt. One user message: an instruction that the answer is exactly one word and a proper name, two worked examples with their answers in name tags, then the passage with the masked token. The model must guess even when unsure fig. 2.
- Scoring. A book's accuracy is the fraction of its sampled passages where the model's name matches the masked name §5.1. The paper does not define the matching rule (see Pitfalls).
- Baselines. Predicting the most frequent name in the set ("Mary") gives 0.6%. One author answered the same prompt by hand, using only the passage, and scored 0% §3.2. BERT receives the passage without the instruction and is scored the same way §5.2.
Book sample
Five sources of English fiction are used, 571 works in total §4:
- 91 novels from LitBank, published before 1923 (public domain);
- 90 Pulitzer Prize nominees, 1924 to 2020;
- 95 bestsellers, drawn from NY Times and Publishers Weekly lists, 1924 to 2020;
- 101 novels by Black authors, from two collections, 1928 to 2018;
- 95 works of Global Anglophone fiction, 1935 to 2020;
- 99 genre novels (science fiction and fantasy, horror, mystery and crime, romance, action and spy), 1928 to 2017.
Text from the post-1923 sources was printed, scanned and OCR'd with ABBYY FineReader §4.
Memorization groups
Books are ranked by GPT-4 name-cloze accuracy. The top decile is treated as memorized (M), the bottom decile as not memorized (not-M), and a held-out set H is kept for testing §6.1. The paper also uses top and bottom quintiles for the narrative-time task §7.2.
Name-level test of seen versus unseen names
For each name the true masked answer is linked to a book by its positive pointwise mutual information. Names tied to M form set S, names tied to not-M form set U. Both sets have equal size and no overlap §6.1. On the error cases in H, the observed statistic is a log-odds ratio of predicting a name in S versus in U:
o = \log\left(\frac{\Pr(\hat{c} \in S \mid \text{error})}{\Pr(\hat{c} \in U \mid \text{error})}\right) §6.1
Here \hat{c} is the predicted character name. Significance comes from a randomization test that reshuffles names between S and U §6.1.
# Shuffle-based significance for the name-level test [§6.1]
# Input: observed log-odds o, sets S and U, error predictions on H
# Output: p-value (shuffle count not stated in the paper)
def randomization_p_value(o, S, U, predictions):
# Recompute the statistic after shuffling names between S and U [§6.1]
null = []
for shuffle in shuffled_assignments(S, U):
null.append(log_odds(predictions, shuffle))
# Share of null values at least as large as the observed statistic
return count_at_least(null, o) / len(null)Web prevalence
For each book, 10 passages are sampled, and from each a 10-token span is taken. The span is searched as an exact string in Google, Bing, C4 and the Pile, and the hit counts are averaged per book. The paper then reports Spearman ρ between those counts and GPT-4 accuracy, separately for works published before and after 1928 §6.2 tab. 3.
Name cloze, passage builder
The pseudocode below follows the authors' generator. The paper gives the window length and the no-other-names rule; the repo adds further filters, marked in the comments.
# Build name-cloze items from one book [§3.1] [§3.2]
# Input: tokens (list of str), sentence_ids (one per token),
# proper_spans (list of (start, end) token indices)
# Output: list of (masked_passage, answer) pairs
def make_items(tokens, sentence_ids, proper_spans):
items = []
# Candidate answers are single-token person names [§3.2]
for index in single_token_person_indices(proper_spans):
# Whole sentences around the name, 40 to 60 tokens, nearest 50.
# The repo uses strict bounds and checks proper nouns only.
# [code: generate_name_cloze/create_name_cloze_from_booknlp.py#L34-L73]
window = closest_window(tokens, sentence_ids, index)
if window is None:
continue
answer = tokens[index]
# Repo-only filters: kinship and deity words, repeated answer.
# [code: generate_name_cloze/create_name_cloze_from_booknlp.py#L15]
# [code: create_name_cloze_from_booknlp.py#L167-L175]
if answer_is_excluded(answer, window):
continue
items.append((mask(window, index), answer))
return items# Ask one chat model for the masked name [§3.2] [fig. 2]
# Input: model (chat client), passage (str containing [MASK])
# Output: raw reply text
def ask_name(model, passage):
# Instruction, two worked examples, then the passage [fig. 2]
prompt = build_name_prompt(passage)
# Official script: one user message, temperature 0
# [code: scripts/openai_predict_name_cloze.py#L25-L37]
reply = model.chat(
messages=[{"role": "user", "content": prompt}],
# Temperature is not stated in the paper
temperature=0.0,
)
return reply# Accuracy of one book [§5.1]
# Input: model, pairs (up to 100 (passage, answer) per book) [§3.2]
# Output: fraction of pairs answered correctly
def book_accuracy(model, pairs):
hits = 0
for passage, answer in pairs:
# Remove the name tags from the reply
# [code: data/model_output/README.md]
guess = strip_tags(ask_name(model, passage))
# Exact string match is our reading; the paper does not state one
if guess == answer:
hits += 1
return hits / len(pairs)Year of first publication
A random 250-word passage (complete sentences) from each book goes into a few-shot prompt. GPT-4 is asked for a four-digit year, and may answer that the year is uncertain §7.1 fig. 3. The paper records how often the model refrains and the absolute error |ŷ − y| for each valid answer. Books are compared by top and bottom decile of GPT-4 name-cloze accuracy §7.1.
Narrative time
The same 250-word passages are annotated by hand, blind to the book and to the group, using the criteria of Underwood (2023), and given to GPT-4 with a five-part prompt: summarize, reason about elapsed time, take the midpoint, give minutes, and state certainty §7.2 fig. 4. Accuracy is Spearman ρ between predicted and annotated minutes, with a bootstrap 95% confidence interval. The groups are the top and bottom deciles and the top and bottom quintiles §7.2.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Models | gpt-3.5-turbo (ChatGPT), gpt-4 (GPT-4) | §5.1 openai_predict_name_cloze.py | Dated snapshots not stated; the script keeps the GPT-4 name commented out |
| Interface | One user message, no system message | openai_predict_name_cloze.py | Paper says only "OpenAI API" §5.1 |
| Temperature | 0.0 | openai_predict_name_cloze.py | Not stated in the paper; the year and time scripts also use 0.0 openai_predict_year.py |
| Passage length | 40 to 60 tokens | §3.1 §3.2 | Repo: target 50, strict bounds 40 < n < 60 create_name_cloze_from_booknlp.py |
| Passages per book | 100, random | §3.2 | Books with fewer usable passages are excluded §3.2 |
| Passage sentences | Complete sentences, no crossing sentence boundary | §3.2 | Implemented by sentence-id windows create_name_cloze_from_booknlp.py |
| Entity tagger | BookNLP, pipeline entity, model big | §3.2 run_booknlp.py | Paper cites BookNLP only |
| Excluded answers | 14 kinship and deity words | create_name_cloze_from_booknlp.py | Not in paper |
| Most-frequent-name baseline | "Mary", 0.6% | §3.2 | |
| Human baseline | One author, 0% | §3.2 | Single annotator |
| Year prompt | 250-word passage, four-digit year, "uncertain" allowed | §7.1 fig. 3 | 571 rows in the repo data file year.passages.txt |
| Narrative-time passage | 250 words, hand-annotated minutes | §7.2 fig. 4 | 228 rows in the repo data file narrative.time.passages.txt |
| Decile and quintile groups | Top and bottom 10% and 20% by GPT-4 name-cloze accuracy | §6.1 §7.1 §7.2 | |
| Error-analysis groups | M = top decile, not-M = bottom decile, H held out | §6.1 | Shuffle count not stated |
| Confidence interval | Bootstrap, 95% | §7.2 | Resample count not stated |
| Web-search sample | 10 passages per book, 10-token spans | §6.2 | Search API details §6.2 |
| API cost | About $400; $0.002 per 1K tokens (ChatGPT), $0.03 per 1K tokens (GPT-4) | §5.1 | Prices as of the paper |
Results
Author-reported. Setup: 571 works of fiction §4; ChatGPT and GPT-4 queried through the OpenAI API with the same name-cloze prompt on every sampled passage §5.1; BERT given the passage without instructions §5.2.
Headline results:
- The most memorized book is Alice's Adventures in Wonderland, at GPT-4 accuracy 0.98 tab. 1.
- Twelve of the 20 most memorized in-copyright books are science fiction or fantasy §5.1.
- Name-cloze accuracy on the pre-1923 LitBank books is 0.244 for GPT-4 and 0.072 for ChatGPT tab. 2.
- BERT is near zero on almost every listed book. The exception is Fifty Shades of Grey at 0.13 tab. 1 §5.2.
- Error analysis: o = 1.34 for ChatGPT and o = 1.37 for GPT-4, both with p < 0.0001 §6.1.
- Web prevalence: pre-1928 Spearman ρ is 0.74 (Google), 0.70 (Bing), 0.71 (C4) and 0.84 (Pile). Post-1928 values are weaker, between 0.21 and 0.41 tab. 3.
- Year of first publication: mean absolute error on top-decile books is 3.3 years for ChatGPT and 0.6 years for GPT-4. On bottom-decile books it is 29.8 and 14.5 years tab. 5. Refusals run 4% (top) against 9% (bottom) for ChatGPT. GPT-4 never refused §7.1.
- Narrative time: ρ = 0.50 [0.25, 0.72] for TOP10 and 0.27 [−0.02, 0.59] for BOT10, not significant. For the quintiles, 0.47 and 0.46 §7.2.
- Cost: about $400 for the full name-cloze run §5.1.
Table 1: the 20 highest GPT-4 scores tab. 1
| GPT-4 | ChatGPT | BERT | Date | Author | Title |
|---|---|---|---|---|---|
| 0.98 | 0.82 | 0.00 | 1865 | Lewis Carroll | Alice's Adventures in Wonderland |
| 0.76 | 0.43 | 0.00 | 1997 | J.K. Rowling | Harry Potter and the Sorcerer's Stone |
| 0.74 | 0.29 | 0.00 | 1850 | Nathaniel Hawthorne | The Scarlet Letter |
| 0.72 | 0.11 | 0.00 | 1892 | Arthur Conan Doyle | The Adventures of Sherlock Holmes |
| 0.70 | 0.10 | 0.00 | 1815 | Jane Austen | Emma |
| 0.65 | 0.19 | 0.00 | 1823 | Mary W. Shelley | Frankenstein |
| 0.62 | 0.13 | 0.00 | 1813 | Jane Austen | Pride and Prejudice |
| 0.61 | 0.35 | 0.00 | 1884 | Mark Twain | Adventures of Huckleberry Finn |
| 0.61 | 0.30 | 0.00 | 1853 | Herman Melville | Bartleby, the Scrivener |
| 0.61 | 0.08 | 0.00 | 1897 | Bram Stoker | Dracula |
| 0.61 | 0.18 | 0.00 | 1838 | Charles Dickens | Oliver Twist |
| 0.59 | 0.13 | 0.00 | 1902 | Arthur Conan Doyle | The Hound of the Baskervilles |
| 0.59 | 0.22 | 0.00 | 1851 | Herman Melville | Moby Dick; Or, The Whale |
| 0.58 | 0.35 | 0.00 | 1876 | Mark Twain | The Adventures of Tom Sawyer |
| 0.57 | 0.30 | 0.00 | 1949 | George Orwell | 1984 |
| 0.54 | 0.10 | 0.00 | 1908 | L. M. Montgomery | Anne of Green Gables |
| 0.51 | 0.20 | 0.01 | 1954 | J.R.R. Tolkien | The Fellowship of the Ring |
| 0.49 | 0.16 | 0.13 | 2012 | E.L. James | Fifty Shades of Grey |
| 0.49 | 0.24 | 0.01 | 1911 | Frances H. Burnett | The Secret Garden |
| 0.49 | 0.12 | 0.00 | 1883 | Robert L. Stevenson | Treasure Island |
| 0.49 | 0.16 | 0.00 | 1847 | Charlotte Brontë | Jane Eyre: An Autobiography |
| 0.49 | 0.22 | 0.00 | 1903 | Jack London | The Call of the Wild |
Table 2: name-cloze accuracy by book category tab. 2
| Source | GPT-4 | ChatGPT |
|---|---|---|
| pre-1923 LitBank | 0.244 | 0.072 |
| Genre: SF/Fantasy | 0.235 | 0.108 |
| Genre: Horror | 0.054 | 0.028 |
| Bestsellers | 0.033 | 0.016 |
| Genre: Action/Spy | 0.032 | 0.007 |
| Genre: Mystery/Crime | 0.029 | 0.014 |
| Genre: Romance | 0.029 | 0.011 |
| Pulitzer | 0.026 | 0.011 |
| Global | 0.020 | 0.009 |
| BBIP/BCALA | 0.017 | 0.011 |
Table 3: Spearman ρ between GPT-4 accuracy and web hit counts tab. 3
| Date | Bing | C4 | Pile | |
|---|---|---|---|---|
| pre-1928 | 0.74 | 0.70 | 0.71 | 0.84 |
| post-1928 | 0.37 | 0.41 | 0.36 | 0.21 |
Sites that return copyrighted text tab. 4
The site with the most hits for copyrighted-text snippets is archive.org, at 337. The rest of the top ten mix full-text hosts with sites that carry snippets in reviews and study guides, such as Goodreads, Amazon and LitCharts §6.2 tab. 4.
Table 5: year of first publication, top and bottom 10% tab. 5
Refusal is the share of books where the model did not give a year. The bracketed values are 95% bootstrap intervals around MAE.
| Model | Refusal top 10% | Refusal bottom 10% | ρ | MAE top 10% | MAE bottom 10% |
|---|---|---|---|---|---|
| ChatGPT | 0.04 | 0.09 | −0.39 | 3.3 [0.1–12.3] | 29.8 [20.2–41.2] |
| GPT-4 | 0.00 | 0.00 | −0.37 | 0.6 [0.1–1.5] | 14.5 [9.5–20.8] |
Table 6: top 50 in-copyright books (published after 1928) tab. 6
| GPT-4 | ChatGPT | BERT | Date | Author | Title |
|---|---|---|---|---|---|
| 0.76 | 0.43 | 0.00 | 1997 | J.K. Rowling | Harry Potter and the Sorcerer's Stone |
| 0.57 | 0.30 | 0.00 | 1949 | George Orwell | 1984 |
| 0.51 | 0.20 | 0.01 | 1954 | J.R.R. Tolkien | The Fellowship of the Ring |
| 0.49 | 0.16 | 0.13 | 2012 | E.L. James | Fifty Shades of Grey |
| 0.48 | 0.14 | 0.00 | 2008 | Suzanne Collins | The Hunger Games |
| 0.43 | 0.27 | 0.00 | 1954 | William Golding | Lord of the Flies |
| 0.43 | 0.17 | 0.00 | 1979 | Douglas Adams | The Hitchhiker's Guide to the Galaxy |
| 0.30 | 0.16 | 0.00 | 1959 | Chinua Achebe | Things Fall Apart |
| 0.28 | 0.12 | 0.00 | 1977 | J. R. R. Tolkien and Christopher Tolkien | The Silmarillion |
| 0.27 | 0.13 | 0.00 | 1953 | Ray Bradbury | Fahrenheit 451 |
| 0.27 | 0.13 | 0.00 | 1996 | George R.R. Martin | A Game of Thrones |
| 0.26 | 0.05 | 0.01 | 2003 | Dan Brown | The Da Vinci Code |
| 0.26 | 0.08 | 0.00 | 1965 | Frank Herbert | Dune |
| 0.25 | 0.20 | 0.01 | 1937 | Zora Neale Hurston | Their Eyes Were Watching God |
| 0.25 | 0.14 | 0.00 | 1961 | Harper Lee | To Kill a Mockingbird |
| 0.24 | 0.03 | 0.03 | 1953 | Ian Fleming | Casino Royale |
| 0.22 | 0.13 | 0.00 | 1984 | William Gibson | Neuromancer |
| 0.20 | 0.10 | 0.00 | 1985 | Orson Scott Card | Ender's Game |
| 0.19 | 0.12 | 0.00 | 1932 | Aldous Huxley | Brave New World |
| 0.18 | 0.07 | 0.00 | 1937 | Margaret Mitchell | Gone with the Wind |
| 0.17 | 0.05 | 0.00 | 1968 | Philip K. Dick | Do Androids Dream of Electric Sheep? |
| 0.16 | 0.06 | 0.05 | 2009 | Dan Brown | The Lost Symbol |
| 0.15 | 0.04 | 0.04 | 2013 | Dan Brown | Inferno |
| 0.15 | 0.08 | 0.01 | 2014 | Veronica Roth | Divergent |
| 0.15 | 0.04 | 0.00 | 1940 | John Steinbeck | The Grapes of Wrath |
| 0.13 | 0.05 | 0.00 | 1983 | James Kahn | Return of the Jedi |
| 0.13 | 0.02 | 0.01 | 1928 | D. H. Lawrence | Lady Chatterley's Lover |
| 0.13 | 0.03 | 0.00 | 1977 | Alex Haley | Roots |
| 0.11 | 0.11 | 0.00 | 1961 | Irving Stone | The Agony and the Ecstasy |
| 0.11 | 0.01 | 0.03 | 1957 | Ian Fleming | From Russia with Love |
| 0.11 | 0.04 | 0.00 | 1962 | Madeleine L'Engle | A Wrinkle in Time |
| 0.11 | 0.04 | 0.01 | 1939 | Marjorie Kinnan Rawlings | The Yearling |
| 0.10 | 0.05 | 0.00 | 1975 | E. L. Doctorow | Ragtime |
| 0.10 | 0.05 | 0.00 | 1929 | Dashiell Hammett | The Maltese Falcon |
| 0.10 | 0.08 | 0.07 | 1991 | Diana Gabaldon | Outlander |
| 0.10 | 0.02 | 0.00 | 1989 | Kazuo Ishiguro | The Remains of the Day |
| 0.10 | 0.01 | 0.00 | 1983 | Alice Walker | The Color Purple |
| 0.09 | 0.02 | 0.00 | 1934 | Dorothy L. Sayers | The Nine Tailors |
| 0.09 | 0.03 | 0.00 | 1985 | Margaret Atwood | The Handmaid's Tale |
| 0.09 | 0.04 | 0.00 | 1988 | Toni Morrison | Beloved |
| 0.08 | 0.07 | 0.00 | 1982 | Joe Nazel | Every Goodbye Ain't Gone |
| 0.08 | 0.06 | 0.01 | 1984 | Helen Hooven Santmyer | ...And Ladies of the Club |
| 0.08 | 0.02 | 0.00 | 2006 | Max Brooks | World War Z |
| 0.08 | 0.02 | 0.00 | 1993 | Irvine Welsh | Trainspotting |
| 0.08 | 0.03 | 0.00 | 1947 | Robert Penn Warren | All the King's Men |
| 0.07 | 0.02 | 0.00 | 1952 | Ralph Ellison | Invisible Man |
| 0.07 | 0.01 | 0.00 | 1951 | Isaac Asimov | Foundation |
| 0.07 | 0.01 | 0.00 | 1976 | Anne Rice | Interview with the Vampire |
| 0.07 | 0.04 | 0.02 | 1977 | Stephen King | The Shining |
| 0.07 | 0.01 | 0.00 | 1996 | Helen Fielding | Bridget Jones's Diary |
Paper vs code
The official repository is bamman-group/gpt4-books, linked from the paper's
footnote and conclusion §5.1 §9. Its owner is the authors' lab, and it
contains no LICENSE file, so its code is recorded as unlicensed: nothing is
copied from it. It was searched by reading the paper's links, which lead
straight to it. No other repository was searched.
- Passage length. The paper says 40 to 60 tokens §3.1 §3.2. The generator keeps spans strictly between 40 and 60 tokens and aims for 50 create_name_cloze_from_booknlp.py create_name_cloze_from_booknlp.py create_name_cloze_from_booknlp.py. Whether exactly 40 or 60 is allowed is ambiguous in the paper and excluded in the code.
- Answer filters not in the paper. The generator drops 14 kinship and deity words as answers create_name_cloze_from_booknlp.py create_name_cloze_from_booknlp.py. It also drops a passage when the answer appears again in it create_name_cloze_from_booknlp.py.
- Other entities. The paper requires no other named entity §3.2. The code checks only spans tagged as proper nouns create_name_cloze_from_booknlp.py.
- Window choice. The paper does not say which window to take. The code picks the one closest to 50 tokens create_name_cloze_from_booknlp.py.
- Temperature. The paper does not state one. The scripts use 0.0 openai_predict_name_cloze.py openai_predict_year.py openai_predict_literary_time.py.
- Prompt quoting. The figure shows the worked examples without quotation marks. The name-cloze script wraps them in double quotes openai_predict_name_cloze.py. The wording is otherwise the same.
- No scoring code. The repository contains prediction scripts only . Accuracy, ρ, MAE, the log-odds statistic and the bootstrap intervals are not in it. The model-output README lists the columns kept (raw reply, reply with tags removed, true masked token, passage) README.md, which is enough to recompute accuracy if the matching rule is exact.
- Model output. Per-book outputs for both models are in the repository . I did not check every file against the 100-passage rule §3.2.
- Year and narrative data. The year file has 571 rows, matching the 571 works year.passages.txt §4. The narrative file has 228 rows; the paper does not give that count narrative.time.passages.txt.
No other discrepancies were found.
Pitfalls
- Matching rule. The paper does not define when a reply counts as correct. Exact string match after tag removal is the natural reading, but it does not allow for case, accents or spacing variants. State the rule you use.
- Snapshots and API. The model versions are not dated. The authors
measured "at the time of writing" §3.1. The official script uses the
old
openai.ChatCompletion.createinterface openai_predict_name_cloze.py. Port it to a current client rather than copying it. Expect results to drift as hosted models change. - Cost. The full run cost about $400 at 2023 prices §5.1. The count of requests is derived here, not reported: 571 works, 100 passages each, two models, which is at most 114,200 calls §4 §3.2.
- Thin books. Books with fewer than 100 usable passages are dropped, and the number dropped is not stated §3.2.
- Token definition. Lengths are counted in BookNLP tokens. A different tokenizer changes which windows pass the 40 to 60 test.
- Circular groups. The memorization deciles come from the same GPT-4 accuracy that the year and time results are then compared against §7.1. The comparison is therefore not independent of the measure.
- Narrative time is inconclusive. The authors state that the TOP10 and BOT10 intervals overlap and that the quintile results show no effect §7.2. A small annotated set (228 rows in the repo file) gives wide intervals.
- Single human baseline. The 0% human score comes from one annotator §3.2. Inter-annotator agreement for the narrative-time labels is not stated.
- Web counts are correlations. The authors argue popularity drives both web prevalence and memorization, and say the link is not causal §6.2. The post-1928 correlations are weaker, 0.21 to 0.41 tab. 3.
- BERT is not like-for-like. BERT receives the passage without the instruction §5.2, so its numbers are not a direct comparison.
- OCR noise. Post-1923 books were OCR'd, and the paper does not measure how errors affect the cloze §4.
- Permission for data. The repository ships passages from in-copyright books in its data files year.passages.txt, and it has no LICENSE. Do not redistribute those excerpts. Rebuild passages from text you have the right to use.
- Randomization and bootstrap settings. Shuffle counts and resample counts are not stated §6.1 §7.2. Pick them and report them.
Lineage
- Builds on: the notion of exact memorization §3.1, the membership-inference framing §1, the link between duplication and memorization §6.2, and the narrative-time framework and prompt of Underwood (2023) §7.2. None of these has a card in this wiki.
- Compared against: BERT as a plain language-model baseline §5.2.
- Superseded by: not stated in the paper or the repository.
Source and license
Adapted from “Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4” by Kent K. Chang, Mackenzie Cramer, Sandeep Soni, David Bamman (arXiv:2305.00118), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/bamman-group/gpt4-books at commit ef7b1a6c2c08 (none (no LICENSE file in repository)). Any quoted code is shown under that license.