The Pile: An 800GB Dataset of Diverse Text for Language Modeling

In plain words

Language models learn from large piles of text, and most of the big ones have relied mostly on a single web crawl. This paper releases the Pile, a roughly 825 GiB English text collection built from 22 different sources: scientific papers, code, court opinions, patents, books, email, chat logs, subtitles and more. Some sources are deliberately given more weight so that the careful writing in them is seen more than once during training.

The authors also tested it. 1.3B-parameter models trained on the Pile did better than models trained on other web-based collections, both on the Pile's own parts and on a standard word-prediction benchmark. They also documented the data in detail, including its bias, profanity and consent issues, so that users can decide what to use it for.

TL;DR

The Pile is a fixed, documented mixture of 22 English text sources with a per-source epoch weight. Use it when you want a broad, multi-domain pretraining corpus. The paper reports that the authors' 1.3B models beat CC-100 and raw Common Crawl on the Pile components. Several sources carry consent or terms-of- service caveats that the paper itself documents, so check Table 5 before reuse.

Problem

Large language models were mostly trained on Common Crawl derivatives. The authors argue that mixing many smaller, high-quality and diverse sources gives better cross-domain knowledge and downstream generalisation §1. They also note that no large training dataset of that time had been documented in depth by its creators §6.

Method

The Pile has three parts: the constituent datasets §2, the way they are mixed and split app. D, and an evaluation protocol §3, §4. The paper also gives extensive documentation §6.

1. Constituent datasets §2 tab. 1 app. C. Twenty-two components are used: Pile-CC, PubMed Central, Books3, OpenWebText2, ArXiv, GitHub, FreeLaw, Stack Exchange, USPTO Backgrounds, PubMed Abstracts, Project Gutenberg (PG-19), OpenSubtitles, Wikipedia (en), DeepMind Mathematics, Ubuntu IRC, BookCorpus2, EuroParl, HackerNews, YouTube Subtitles, PhilPapers, NIH ExPorter and Enron Emails. Each component is processed separately; the per-component steps are in app. C. The authors filtered for English in several components app. C.1 §5.2.

2. Epoch weights tab. 1 §2 app. D. Each component gets a weight expressed as the number of passes ("epochs") over it in one full pass over the Pile. Higher-quality components get more than one epoch. Wikipedia has 3.0 epochs, and the other components have 1.0 to 2.5 tab. 1.

3. Mixing app. D. Documents are drawn at random from the components until the target size is reached. The draw probability for a component is proportional to its number of documents times its epoch weight app. D. The output is written as 30 interleaved piles app. D.

# Mix components into the Pile  [app. D]  [tab. 1]
def draw_pile_document(components, epochs, num_docs, rng):
    # Weight of each component: documents times epochs  [app. D]
    weights = [num_docs[c] * epochs[c] for c in components]
    # Pick one component at random with those weights  [app. D]
    component = rng.choices(components, weights=weights, k=1)[0]
    # Return the next document of the chosen component  [app. D]
    return next_document(component)

4. Holdout and leakage removal app. D §3.1. About 10 GiB is held out; about 2 GiB of it is used for validation and test splits, and the rest is kept in reserve app. D. Training documents that appear verbatim in the held-out data are removed app. D.

# Remove training documents that match a held-out document  [app. D]
def remove_holdout_copies(train_docs, holdout_docs):
    # Fingerprint every held-out document  [code: processing_scripts/dedupe_train.py#L40-L45]
    seen = {sha256(doc) for doc in holdout_docs}
    # Keep only training documents whose fingerprint is unseen  [app. D]
    return [doc for doc in train_docs if sha256(doc) not in seen]

5. Deduplication app. D. Document-level near-duplicate removal with MinHash LSH is applied only to OpenWebText2 and Pile-CC; Pile-wide deduplication was not performed for memory reasons app. D. The paper reports a duplicate rate of 28% for OpenWebText2 and 26% for Pile-CC app. D.

6. Evaluation §3.1 §3.2 app. E.2. Models are scored in bits per UTF-8 byte (BPB), which makes results comparable across tokenisers §3.1. The formula is given in the Hyperparameters table. Perplexity is computed per document, not over concatenated text §3.2 app. E.2.

7. Relative GPT-3 difficulty proxy §3.3. To estimate how much GPT-3 underperforms on each component, the paper compares GPT-3 and a GPT-2 model trained on the Pile, each measured relative to OpenWebText2. The equation is below §3.3.

# Relative difficulty of a component vs OpenWebText2  [§3.3]
def relative_gap(component, loss_gpt3, loss_gpt2_pile):
    # Gap between the component and OpenWebText2 for GPT-3  [§3.3]
    gpt3_gap = loss_gpt3[component] - loss_gpt3["owt2"]
    # Same gap for the GPT-2 model trained on the Pile  [§3.3]
    pile_gap = loss_gpt2_pile[component] - loss_gpt2_pile["owt2"]
    # Lower values mean GPT-3 does relatively better  [§3.3] [fig. 3]
    return gpt3_gap - pile_gap

8. Documentation §6 app. E app. F. The paper documents the data using datasheets and data statements §6.1. It adds topic models §6.2, profanity §6.3, co-occurrence bias §6.4, author consent tiers §6.5, legal discussion §7.1, and dataset samples app. F.

Hyperparameters

Values stated in the paper. Anything not listed was not stated.

Results

Author-reported. All numbers come from the paper's own runs. Models: GPT-2 (small to xl) and GPT-3 (ada to davinci) for perplexity §3.2; 1.3B-parameter models trained from scratch on each dataset for §4 tab. 3 tab. 4.

Test perplexity of GPT-2 and GPT-3 on Pile components, bits per UTF-8 byte tab. 2. One-tenth of each test set, scored per document. Bold in the paper marks the best model per row (davinci in every row).

Scaling: the authors fit a line to GPT-3 family perplexity on the Pile with coefficient −0.1674 and intercept 2.5516 §3.2 fig. 2. The paper states that GPT-2 and GPT-3 were not trained on the Pile §3.2.

Size-controlled comparison of 1.3B models tab. 3. Each dataset is deduplicated against all evaluation sets and subsampled to about 40 GB §4.1. WikiText and LAMBADA use the variant of GPT-2 §4.1 tab. 3.

The Pile row lists "825 GiB" as dataset size in the paper's table, while the caption and §4.1 say the subsample is about 40 GB tab. 3 §4.1. See Pitfalls.

Per-component BPB on the Pile test set tab. 4. Columns are the training set, rows the evaluation component. The paper marks the best model per row in bold, and the Pile model is best on every row.

Headline text, as stated by the authors: the Pile improves significantly on WikiText and shows negligible change on LAMBADA §4.2. Models trained on the Pile beat both Raw CC and CC-100 on every Pile component, per the authors §4.2. The authors single out significant gains over CC-100 on academic sets (ArXiv, PubMed Central, FreeLaw, PhilPapers), code sets (GitHub, Stack Exchange), EuroParl and DM Mathematics §4.2 fig. 4. They also note that raw Common Crawl beats CC-100 on the Pile and hypothesise that CC-100's perplexity-based filtering is the cause §4.2.

Dataset consent tiers tab. 5. This table is a statement of the authors' own assessment, not legal advice. Columns: Public (freely available on the internet), ToS (obtained consistent with the host's terms of service), Author (authorial consent, or a reasonable expectation of research use) §6.5.

The authors state that five components (Books3, OpenWebText2, OpenSubtitles, BookCorpus2 and YouTube Subtitles) were not collected and distributed in a terms-of-service compliant way, and that the authors of those works had no ability to consent §6.5.

Tokens per byte app. D. These values convert between tokens and bytes for each component. They are needed for BPB.

Filtering ratios for the Pile-CC classifier app. C.1 tab. 6. Kept fraction of Common Crawl documents at each Pareto α. α = 3 was chosen.

Other results, in words. The Pile is 97.4% English by fastText §5.2. The mean document length and bytes per token are plotted per component §5.1 fig. 5 fig. 6. Profanity: the Pile as a whole is less profane than Pile-CC by the authors' word-level proxy §6.3 fig. 7. Gender and religion co-occurrence and sentiment are reported as tables and figures in the appendix; the racial sentiment averages are White −0.114, Black −0.148, Asian −0.028, Hispanic −0.024 §6.4.3.

Paper vs code

Official repository: https://github.com/EleutherAI/the-pile, linked from the paper's footnote §1 and maintained by the authors' lab. Checked at commit df97f8651ae3da658b19659b3ceaa6a34b0fc014, licence MIT.

• Epoch weights agree. The paper's epoch column tab. 1 matches the per-component weights in the code, and Pile-CC is added with weight 1.0 pile.py pile.py.
• Mixing rule agrees. Paper: draw by documents times epochs app. D. Code: weight times documents, normalised pile.py.
• Holdout fraction disagrees. Paper: validation and test are 0.1% each, about 2 GiB of about 10 GiB held out §3.1 app. D. Code: 1% of each shuffled chunk goes to a holdout directory [code: processing_scripts/pass2_shuffle_holdout.py#L30-L35]. The validation and test split step is not in the files reviewed.
• Total size constant disagrees. Code's ThePile.size() returns 1200 GiB pile.py. The paper gives 825.18 GiB raw and 1254.20 GiB effective tab. 1.
• Holdout removal is exact. Paper: remove training documents that appear verbatim in the holdout app. D. Code: SHA-256 of the whole document must match dedupe_train.py. Near-duplicates are not removed by this step.
• GitHub file limit is characters, not bytes. Paper: 100 kB file limit app. C.6. Code: keeps files with len(text) < 100000 characters [code: processing_scripts/github_reduce.py#L15].
• Evaluation decontamination. Paper: 13-gram overlap filter as in GPT-3 §4.1. Code: 13-grams from evaluation texts are excluded, and each match removes 200 characters on each side [code: processing_scripts/ablation_dedupe/make_deduped.py#L31-L43].
• Not in this repository. No code was found in the repository for the Pile-CC extraction (jusText), the OpenWebText2 and Pile-CC MinHash deduplication, pycld2 filtering, or the 1.3B model training. A search of the repository for minhash, datasketch, justext and pycld2 returned nothing. The paper says construction code is public app. C. It may live in other repositories not reviewed here.

Pitfalls

• Three different sizes. The Pile is 825.18 GiB raw, 1254.20 GiB after epoch weights tab. 1, and the code constant says 1200 GiB [code: the_pile/pile.py#L206-L207]. Say which one you mean.
• Epoch limits contradict each other. Appendix D says no data was allowed above 3 epochs and that the authors avoided more than 2 epochs app. D. Table 1 lists Wikipedia at 3.0 and Gutenberg at 2.5 tab. 1.
• Holdout size contradicts itself. Section 3.1 gives 0.1% for each of validation and test. Appendix D says about 10 GiB is held out. The code holds out 1% per chunk pass2_shuffle_holdout.py.
• Table 3's size column conflicts with its caption. The Pile row says 825 GiB, while the caption and §4.1 say each dataset is subsampled to about 40 GB tab. 3.
• BPB needs the mean loss in nats per token. The formula is BPB = (L_T / L_B) × ℓ / ln 2, where ℓ is a per-token loss in nats §3.1. The paper does not say explicitly whether ℓ is the mean; treat it as the mean per token. Use the tokens-per-byte values for each component, not the global 0.29335, when reporting per-component BPB §3.1 app. D.
• Perplexity is per document. Documents are scored one at a time and not concatenated §3.2 app. E.2. Segments are non-overlapping, so early tokens of each segment see less context. Reproducing the paper's numbers needs the same per-document rule app. E.2.
• GPT-3 sizes are assumptions. The paper assumes ada, babbage, curie and davinci are 2.7B, 6.7B, 13B and 175B parameters §3.2 footnote. The API does not publish sizes.
• Test evaluation is on one-tenth of the test set for most components §3.2 app. E.2. Three components (Ubuntu IRC, BookCorpus2, PhilPapers) are scored on the full test set app. E.2. Compare numbers only on the same subset.
• GitHub search bounds in the paper look wrong. The paper says the first query covers repositories between 0 and 5 bytes and returns 965 results app. C.6. That range is almost certainly a unit typo, and the paper does not say what unit was intended. The next-bound formula is given in the same section app. C.6. Do not reuse the bounds without checking the GitHub API.
• GitHub sampling is lossy. Only 95.0 GiB of 630.64 GiB was kept app. C.6. Repositories exceeding 300 seconds to clone or extract are partially extracted app. C.6.
• Epoch weighting raises duplication. Components with 2 to 3 epochs are seen two or three times in one pass tab. 1. That affects memorisation and should be considered when interpreting results.
• Deduplication is partial. Only OpenWebText2 and Pile-CC are deduplicated app. D, so duplicates across components are not removed. The paper says some documents may be duplicated across train, validation and test §3.1.
• No downstream leakage removal. The authors chose not to remove overlap with downstream benchmarks app. D. Remove overlaps yourself when you evaluate on other benchmarks.
• Several components are not ToS- or consent-compliant. Five components are flagged in tab. 5 and §6.5. The paper's fair-use argument §7.1 is the authors' view and explicitly not legal advice. Check with counsel before commercial use.
• Scaling-law fit is on GPT-2 and GPT-3, not on trained Pile models. The line has coefficient −0.1674 and intercept 2.5516 §3.2.
• The relative-difficulty equation is a proxy. The authors say it can be confounded by dataset-specific scaling effects §3.3. The GPT-2-Pile model is a stand-in because GPT-3 was not retrained §3.3.
• Cross-reference error in the paper. The text refers to "Section 3.7" for the language classifier §6.3. The language section is §5.2.
• Training details are missing. The paper trains 1.3B models "based on" GPT-3 but gives no optimiser, learning rate, batch size, step count or training tokenizer §4.1. Mark these as not stated when you implement.
• Figure-based results are not tabulated. The relative GPT-3 chart fig. 3 and the CC-100 improvement chart fig. 4 have no table in the paper.

Lineage

• Builds on: the datasets it re-packages. Books3 (Bibliotik), PG-19, OpenSubtitles, OpenWebText, BookCorpus, Wikipedia, DM Mathematics, EuroParl, Enron Emails §2. These are named and cited in the paper but not carded here.
• Compared against: GPT-2 and GPT-3 perplexity §3.2; CC-100 and raw Common Crawl training §4.2. Not carded here.
• Superseded by: not stated in this card.