Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

In plain words

Large language models learn from huge piles of text scraped from the web. C4 is one such pile. It was built from one 2019 snapshot of the Common Crawl web archive, with a set of filters that keep English text and drop junk. This paper does not build a model. It documents what is inside C4 and what the filters throw away.

It asks three questions: where the text comes from, what the text says, and what was removed. Some findings are surprising. The sources are very uneven, and substantial amounts of the text are patents. A long tail of pages dates back ten to twenty years. Some benchmark test examples appear word for word in the text. The word-list filter removes a disproportionate share of text in African American English and text about gay and lesbian topics. The authors argue that anyone building a web corpus should report these things, and should not filter with word blocklists.

TL;DR

• The paper documents the English C4 corpus: the cleaned April 2019 Common Crawl snapshot used to train T5 §1 §2.
• Sources are skewed. patents.google.com is the single most represented website, and .gov, Wikipedia and news domains are prominent §3.1. Internet Archive first-index dates for a sample of 1,000,000 URLs put 92% in 2011–2019 §3.2.
• Benchmark examples appear verbatim: 1.87% to 24.88% of generation targets and 1.8% to 53.6% of GLUE-style inputs occur in C4 §4.2 tab. 2.
• The bad-word blocklist removes 42% of documents classed as African American English and 32% of Hispanic-aligned ones, versus 6.2% and 7.2% for White-aligned and other English §5.3. The authors recommend against blocklist filtering §6.
• All numbers are the authors' own measurements. There is no training method or model to reproduce; reproducing the analyses needs the data, which the authors host.

Problem

Models pretrained on web text are common, but the corpora behind them are often described only briefly §1. The original C4 scripts were published, but the authors say that running them costs thousands of dollars §1. So a downloadable copy is the practical route to reproduction §1.

The authors argue that documentation for a web-scraped, filtered corpus needs three views: the metadata (the domains and dates), the included text, and the excluded text §1 fig. 1 §8. Standard dataset documentation, such as reporting the number of examples and a download link, applies only in part, because facts about text authors are usually missing from web text §1.

Method

The paper has no training method. Its method is a set of analyses over C4 and two related versions.

Corpus §2. C4 starts from the April 2019 Common Crawl snapshot. It applies a set of filters that remove non-natural text and keep English, including a blocklist of "bad" words. The filter list is given in full in Raffel et al. §2; this paper summarises it. Two other versions are hosted: C4.en.noBlocklist, which applies all filters except the blocklist, and C4.en.noClean, which is the English-identified snapshot with no other filters §2 tab. 1. The pseudocode below follows the filter list in §2. The paper does not state the order of the filters, so the order used here is one valid choice.

# C4.en from one Common Crawl snapshot (April 2019)  [§2]
# Input:  documents (text scraped from one URL each), blocklist (set)
# Output: kept documents, with some lines removed inside them

def build_c4_en(documents, blocklist):
    kept = []
    for document in documents:
        # 1. Keep documents langdetect rates English at p >= 0.99  [§2]
        if english_probability(document) < 0.99:
            continue
        # 2. Drop documents containing any word on the blocklist  [§2] [§5]
        if any(word in blocklist for word in words(document)):
            continue
        # 3. Drop documents containing Lorem ipsum placeholder text  [§2]
        if "lorem ipsum" in document.lower():
            continue
        # 4. Remove lines without terminal punctuation or under 3 words  [§2]
        lines = [line for line in lines_of(document) if keep_line(line)]
        # 5. Drop documents with fewer than five sentences  [§2]
        if count_sentences(lines) < 5:
            continue
        kept.append("\n".join(lines))
    return kept

Metadata analyses §3. Token counts per top-level domain and per website use the SpaCy English tokenizer §3.1 tab. 1. Writing dates are approximated by the earliest Internet Archive index date of each URL, which is a proxy and not the real writing date §3.2. Locations are approximated by the country of the IP address that hosts each URL, using an IP-to-country database §3.3.

Text analyses §4. Machine-generated text is looked for in patents.google.com, where patents are machine-translated into English and some are first OCR'd §4.1 tab. 3. Benchmark contamination counts test items, and their generation targets, that occur exactly in C4, after normalising case and punctuation §4.2. Demographic bias reproduces the UNQOVER question set of Li et al. with UnifiedQA, a T5 model fine-tuned on question answering §4.3 app. A.7. The authors then measure sentiment-bearing words that co-occur with "Jewish" and "Arab" in C4, and in two domains, using lexicons §4.3 app. A.7.

# Benchmark contamination, exact-match variant  [§4.2]
# Input:  corpus documents, benchmark examples (one string each)
# Output: fraction of examples found verbatim in the corpus

def exact_match_rate(corpus_documents, examples):
    # 1. Normalise capitalisation and punctuation  [§4.2]
    corpus_text = [normalise(document) for document in corpus_documents]
    matched = 0
    for example in examples:
        # 2. Count an example as matched if it occurs in a document  [§4.2]
        #    (granularity of the match is not stated in the paper)
        if any(normalise(example) in document for document in corpus_text):
            matched += 1
    return matched / len(examples)

# Sentiment gap between two identity terms in a corpus  [§4.3] [app. A.7]
# Input:  paragraphs, a term, a sentiment lexicon (positive and negative)
# Output: share of sentiment-bearing word occurrences that are positive

def positive_share(paragraphs, term, lexicon):
    # 1. Keep paragraphs in which the term is a whitespace-delimited token
    #    [app. A.7]
    matched = [p for p in paragraphs if has_token(p, term)]
    positive = 0
    negative = 0
    for paragraph in matched:
        # 2. Count lexicon words of each sign in those paragraphs  [app. A.7]
        for word in strip_punctuation(tokenize(paragraph)):
            if lexicon.is_positive(word):
                positive += 1
            elif lexicon.is_negative(word):
                negative += 1
    # 3. Positive share of sentiment-bearing occurrences  [app. A.7]
    return positive / (positive + negative)

# Gap reported in [§4.3]: share("Jewish") minus share("Arab")

Excluded-text analyses §5. The authors take 100,000 documents removed by the blocklist, embed them with TF-IDF, project with PCA, and cluster them with k-means (k = 50) §5.1 app. A.6. They then count mentions of 22 identity regular expressions covering gender, sexual orientation, race and religion §5.2 tab. 6. Each identity is scored by pointwise mutual information (PMI) between a mention and removal by the blocklist §5.2; the paper cites Church and Hanks for PMI and does not write the formula out. Dialect is estimated with a topic model trained on geolocated tweets: each document gets posterior probabilities for African American English (AAE), Hispanic-aligned English, White-aligned English (WAE) and an "other" class, and is assigned its most likely dialect §5.3.

Recommendations §6. The authors recommend reporting the domains a web scrape comes from, examining benchmark contamination, measuring social bias as part of dataset creation, reporting excluded content, and not using blocklist filtering to build web corpora. They also recommend a public channel for reporting issues §6 §8.

Hyperparameters

Analysis settings, thresholds and sample sizes. Only the filter thresholds are design choices of C4; the rest are the authors' measurement choices.

Results

Author-reported measurements. The paper does not benchmark a method, so there is no baseline; the comparisons are internal (between domains, datasets and dialects).

Corpus sizes tab. 1.

Metadata §3.

Patent offices in the patents.google.com subset tab. 3. The US office is the largest source at 70,489 patents, followed by EP (4,583), JP (4,554), CN (2,283) and WO (2,154). The paper says that more than ten percent of these patents come from offices that require a language other than English §4.1.

Benchmark contamination tab. 2, counts from tab. 4.

The paper found no input-and-label contamination for GLUE and SuperGLUE classification sets in the unfiltered Common Crawl, after checking GitHub-hosted copies §4.2 app. A.5.

Demographic bias tab. 7 §4.3. Share of UNQOVER questions on which UnifiedQA picks an ethnicity as having positive sentiment: Jewish 67.1%, Asian 60.6%, Caucasian 60.5%, European 60.5%, White 56.5%, Alaskan 55.9%, Hispanic 50.8%, Native American 50.6%, South-American 44.4%, African-American 44.3%, Latino 43.1%, Middle-Eastern 42.6%, Black 39.3%, Arab 37.0%, African 36.6%.

Corpus co-occurrence, SocialSent lexicon §4.3 app. A.7:

Gap in all of C4 with other lexicons: 1.4 points (manual lexicon) and 2.0 points (VADER) app. A.7.

Excluded text §5.

• The 100,000-document sample is largely clustered into 16 of 50 clusters with sexual content, which the paper says make up 31% of removed documents §5.1.
• Sexual-orientation mentions (lesbian, gay, heterosexual, homosexual, bisexual) have the highest PMI with removal §5.2 fig. 5.
• In a hand check of 50 documents per term, 22% of "lesbian" documents and 36% of "gay" documents were non-offensive and non-sexual §5.2.
• Removal rate by dialect: AAE 42%, Hispanic-aligned 32%, WAE 6.2%, other 7.2% §5.3.
• Dialect share in C4.en: WAE 97.8%, AAE 0.07%, Hispanic-aligned 0.09% §5.3.

Paper vs code

The paper links the repository https://github.com/allenai/c4-documentation at the commit recorded above (6167da0). The owner, allenai, is the Allen Institute for AI, one of the authors' affiliations on the title page. The repository's only file is a README. It has no LICENSE file, no filter code and no analysis code, so the paper's numbers cannot be checked against code. The README also names the paper as "Documenting the English Colossal Clean Crawled Corpus", while the arXiv title is "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus". The README points to a raw-data download on allennlp discussions and to a search index; the paper names the same search index in its footnote §1.

The arXiv source also contains notes after \end{document}. They are not part of the compiled paper. They say the English filter is CLD3, not langdetect, and give different thresholds (5 words per line, 3 sentences per document). They also describe deduplication steps that the body does not mention. The notes point to Google's TensorFlow Datasets code, which was not read for this card. The body of the paper, which this card follows, names langdetect and gives 3 words and 5 sentences.

Pitfalls

• Filter definitions conflict in the source. The body says langdetect, at least 0.99, 3 words per line and 5 sentences per document §2. Notes after \end{document} say CLD3, 5 words per line and 3 sentences, and add deduplication. This card uses the body numbers, since the notes are not part of the paper. The full filter list is in Raffel et al. The paper does not state the order of the filters.
• The paper is not a method. There is nothing to train. Reproducing the analyses needs the raw corpus, which the paper hosts (as the README says). The paper says the search index is hosted only until 2021-12-31 §1, so check that it still works before relying on it.
• No licence for the data. The paper says C4 is likely to contain personal information and copyrighted text, which it did not quantify §6. This card's licence covers the paper only. Check the licence of the hosted data before reuse.
• Contamination is exact-match only. The match is case- and punctuation-normalised §4.2. The paper does not state whether it matches a whole document or a substring. Only exact copies count, so paraphrases and near-duplicates are not measured. The method differs from GPT-3's 8- to 13-gram overlap, so the rates are not comparable with GPT-3's numbers §4.2.
• Dates are proxies. The Internet Archive first-index date is not the writing date. The archive indexed only about 65% of URLs, and can index a page months after it was created §3.2.
• Locations are proxies. Hosting location is not the location of the authors. Shared hosting, CDNs and ISP caches all break the link, as the paper notes §3.3.
• Dialect rates lack a stated denominator. The removal rates in §5.3 are per dialect, but the paper does not write out the denominator. The dialect model was trained on US tweets with census-based topics, and the "other" class was meant for Asian-aligned English §5.3.
• The bias result is a correlation. The authors say they have not shown a causal link from C4 sentiment to model bias §4.3 §6. The corpus sentiment gap depends on the lexicon: 7.5 points with SocialSent, 2.0 with VADER and 1.4 with the manual lexicon app. A.7.
• Al Jazeera numbers contradict the text. The text says there is no gap for Al Jazeera §4.3. The table gives 42.5% for "Jewish" and 42.8% for "Arab", so the Arab figure is 0.3 points higher.
• Appendix table errors. SST-2 appears twice in the extended contamination table tab. 4. Its LAMA Google-RE count (314 / 5,528) differs from an older count in a commented-out row (184 / 5,527) in the source; the percentage (5.7) matches 314.
• Blocklist effects are not a causal test. The paper shows that removal is uneven. It does not show the effect of removal on a trained model §5 §6.
• Blocklist and tokenizer. The T5 vocabulary was trained on the cleaned corpus, so blocklisted words are missing from it. For example, "sex" is tokenized as "s" plus "ex" app. A.1.

Lineage

• Builds on: Raffel et al. (2020), the T5 paper and the origin of C4 (arXiv:1910.10683, as linked in the repository README). This paper documents the same corpus §1 §2.
• Compared against: GPT-3 (Brown et al. 2020) for benchmark contamination method §4.2. The paper gives no arXiv id for GPT-3.
• Builds on the UNQOVER bias method of Li et al. (2020), reproduced with UnifiedQA §4.3.
• Related documentation work cited by the paper: Datasheets for Datasets, and the documentation frameworks in §1 and §7.