Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

In plain words

Large language models learn from huge amounts of web text. OSCAR is a public collection of such text, built from Common Crawl, a large snapshot of the web. Its earlier versions decided the language of each line on its own. A page that mixed languages was therefore cut into pieces that no longer read as one document.

This paper rebuilds that pipeline so that each whole document gets one language label, and the rules for keeping a document look at the document as a whole. It also trims short lines at the start and end of a page, and drops pages that are mostly short lines. Where it can, it adds tags such as "tiny", "noisy" or "adult" to a document rather than deleting it, so users can filter later. The result is a new corpus, OSCAR 22.01. The authors caution that they cannot yet state its quality.

TL;DR

Rebuilds the OSCAR multilingual web corpus from one Common Crawl dump (November/December 2021) at document level instead of line level §1 §6. Each document gets one language label, multilingual pages go to a separate group, and length-based filters plus non-destructive annotations (tiny, short_sentences, header, footer, noisy, UT1 categories) are added §3 §4 §5. Author-reported: mid- and high-resource sizes stay close to OSCAR 21.09 while several low-resource corpora shrink §6.1. Relevant if you build or filter pretraining web text, but the paper's rules and the released code differ in several thresholds (see Paper vs code).

Problem

• Line-level language classification, used by the earlier OSCAR pipelines and by Ungoliant, destroys the integrity of multilingual documents, because the "documents" are just runs of consecutive lines that share a language §2.
• Crawled web text is noisy, with boilerplate from headers, footers and sidebars that appears in Common Crawl dumps §7.2. The authors argue that line-level processing breaks document integrity and readability, as in their case against line-level deduplication §6.1.4.
• Earlier OSCAR releases carried no per-document annotations or line identifications §6.1.1, so users could not easily judge quality.
• Document-level similarity deduplication is costly in compute and time §6.1.4.
• The method assumes the full Common Crawl dump is already downloaded; the carbon estimate is for generating OSCAR 22.01 from that dump app. A.

Method

Processing order in the paper follows the section order: head/tail trimming and the short-line proportion filter §3, then document language identification §4, then annotations §5. Section numbers below count the LaTeX sections, so the bibliography is §8 and the appendices are A and B. Equations in the source are unnumbered, so they are cited by section. The released code applies the steps in a slightly different order (see Paper vs code).

1. Split. A line is a string separated by a newline character. Documents are sets of lines §4.
2. Head/tail trim. Remove contiguous short lines at the start and the end of the document. A line counts as long when it is longer than 100 characters §3.1 §6.2.5.
3. Short-line proportion filter. Compare the short-line bin with the long-line bin. Discard the document if the short bin is bigger §3.2.
4. Line identification. Each remaining line gets its most likely language from a FastText classifier, with its confidence and its size in bytes. A line whose confidence is below 0.8 gets the "no identification" label g0 §4.
5. Multilingual test. A document is multilingual when every identified language covers at least a fixed share of the document and the unidentified share stays below the same bound. The paper uses the share 1/(n+1), with n the number of lines §4.1. It needs at least 5 lines and 2 to 5 languages §4.1 §6.2.2.
6. Monolingual label. Otherwise take the language with the most bytes and its byte-weighted confidence P_j. Keep the document if P_j is at least 0.6 §4.2.
7. Annotations. Add these to the document metadata without removing it: tiny (5 lines or fewer), short_sentences (at least half the lines short), header and footer (short lines at the start or the end), noisy (too few letters or marks) §5.1 §5.2, and adult from the UT1 blocklist by URL §5.3.
8. Write. Output the document with its language label and annotations, grouped by language §4 §5. The corpus is generated from the November/December 2021 Common Crawl dump §6.

# Document labelling as printed in the paper  [§3] [§4] [§5]
# Input:  lines of one document (text split on newlines)
# Output: (label, confidence), or None when the document is dropped
def label_document(lines):
    # Step 2: keep the lines from the first to the last long line  [§3.1]
    lines = trim_short_head_and_tail(lines, min_chars=100)
    if not lines:
        return None
    # Step 3: drop the document if short lines dominate  [§3.2]
    if short_lines_dominate(lines):
        return None
    # Step 4: per-line language and confidence; below 0.8 is unknown  [§4]
    line_labels = [identify_line(line, threshold=0.8) for line in lines]
    # Size of each line in bytes  [§4]
    line_sizes = [len(line.encode("utf-8")) for line in lines]
    # Step 5: multilingual test on the whole document  [§4.1]
    if is_multilingual(line_labels, line_sizes):
        return ("multi", 0.5)  # fixed value used by the released code
    # Step 6: largest language by bytes, with confidence P_j  [§4.2]
    language, confidence = largest_language(line_labels, line_sizes)
    # Keep only when P_j reaches 0.6  [§4.2]
    if language is None or confidence < 0.6:
        return None
    return (language, confidence)

# Multilingual test as printed  [§4.1]
# Input:  per-line labels (None = unknown) and sizes in bytes
# Output: True when the document is multilingual
def is_multilingual(line_labels, line_sizes):
    # At least 5 lines in the paper (the released code uses 10)  [§4.1]
    if len(line_labels) < 5:
        return False
    languages = {label for label in line_labels if label is not None}
    # Between 2 and 5 languages  [§4.1] [§6.2.2]
    if not 2 <= len(languages) <= 5:
        return False
    # Threshold |D|/(n+1), n = number of lines  [§4.1]
    threshold = sum(line_sizes) / (len(line_labels) + 1)
    for language in languages:
        # Every identified language reaches the threshold  [§4.1]
        if size_of(language, line_labels, line_sizes) < threshold:
            return False
    # Unknown lines do not exceed the threshold  [§4.1]
    if size_of(None, line_labels, line_sizes) > threshold:
        return False
    return True

Key equations, as printed (they are unnumbered in the source). The byte-weighted confidence is written here with the paper's typos corrected (the printed form uses |P_j| and s_j p_j); see Pitfalls.

Document as a pair of lines and labels, with size in bytes §4:

\mathcal{D} = (\mathcal{L}, \mathscr{L}), \qquad |\mathcal{D}| = \sum_{i} s_i

Byte-weighted confidence of language j over a document §4:

P_j = \frac{1}{|\mathcal{D}|} \sum_{i:\; g_i = g_j} s_i \, p_i

Multilingual condition, for all identified languages j and the unknown label g_0 §4.1:

|g_j| \ge \frac{|\mathcal{D}|}{n+1} \quad \forall\, g_j \ne g_0, \qquad |g_0| \le \frac{|\mathcal{D}|}{n+1}

Hyperparameters

Results

Author-reported. Setup: OSCAR 22.01 generated from the November/December 2021 Common Crawl dump; compared with OSCAR 21.09 from the February 2021 dump §6 §6.1.3. Generation ran on one machine with 192 GB RAM and two Intel Xeon Gold 5218 processors, in 42.6 hours app. A. The paper reports no benchmark scores for models trained on the corpus.

Other reported observations:

• The low-resource corpora show more variance in the lower end of the size distribution §6.1.1 fig. 1. Content sizes of the selected languages are similar for mid- and high-resource languages and slightly lower overall §6.1.1 fig. 2.
• In the low-resource group, the tiny annotation is frequent and short_sentences is rare, which the authors read as long lines inside short documents §6.2.1 fig. 3 fig. 4.
• Multilingual documents hold between 2 and 5 languages. English is present in many of them, which the authors suggest may come from boilerplate such as menus or footers §6.2.2 fig. 5.
• On an English sample of 183,497 documents (1.3 GB), clean (unannotated) documents have a mean length of 6537 and standard deviation 14983, versus 8606 and 49874 for annotated documents. After trimming the top and bottom 5%, the values are 3582 (sd 3202) and 3686 (sd 4047). Units are not stated. The authors say these results do not establish intrinsic quality §6.2.3 fig. 6.
• Adult annotation: 32,870 of 52,037,098 French documents are annotated adult. 1063 documents from one LGBTQI+ news site are labelled adult, about 3.2% of the adult set; the authors read this as a sign that more LGBTQI+ sites are on the blocklist §6.2.4.

Per-language size of OSCAR 22.01 tab. 2

Reproduced in full from the paper's appendix (Size of the OSCAR corpus by language; sizes in bytes, words as space-separated tokens) app. B tab. 2. Source table names are kept as printed.

Paper vs code

Code: the repository linked in the paper's introduction, oscar-corpus/ungoliant at commit b26f11d9b28e25ab72999356b9a91d38a6535e04 (Apache-2.0) §1. The paper does not name the code version used for OSCAR 22.01, and this commit is later than the paper, so it may not be the code that produced that release. The line-level threshold of 0.8 §4 and the document threshold of 0.6 §4.2 match the paper pipeline.rs pipeline.rs. Other differences:

• Multilingual test, unit and denominator. The paper weights by bytes and divides by n+1 with n the number of lines §4.1. Its worked example divides by the number of languages plus one instead. The code counts lines and divides by the number of distinct labels, which includes the unknown label only when present multilingual.rs.
• Multilingual minimum size. The paper says 5 lines §4.1; the code requires 10 multilingual.rs multilingual.rs.
• Multilingual confidence gate. The code requires that more than 80% of lines have confidence at least 0.8 before the language test runs multilingual.rs multilingual.rs. The paper has no such gate §4.1.
• Multilingual label and confidence. The paper does not name the label §4.1. The code stores "multi" with a fixed confidence of 0.5 and returns before the 0.6 check pipeline.rs.
• Unknown as largest bin. The paper takes the largest represented language §4.2. The code takes the largest bin by bytes including the unknown bin, and drops the document when the unknown bin is the largest pipeline.rs.
• Head/tail trim. The paper removes contiguous short lines at the head and tail §3.1. The code removes every line before the first long line and after the last long line, and drops a record with no long line sentence_filter.rs sentence_filter.rs pipeline.rs. A windowed variant (window 5) exists but the pipeline does not call it sentence_filter.rs.
• Order of filters. The code trims the head and tail first, so the proportion filter sees the trimmed record pipeline.rs. The paper does not state an order.
• Short-line proportion filter. The paper counts short and long lines and discards the document when the short bin is bigger §3.2. The code sums characters, not lines, and keeps a document only when long lines hold at least 60% of characters record.rs record.rs.
• Short-line boundary. The proportion filter counts exactly 100 characters as long record.rs. The sentence filter treats a line as long only when it has more than 100 characters sentence.rs. The header check counts bytes, not characters header.rs.
• tiny. The paper flags 5 lines or fewer §5.1. The code flags fewer than 5 lines tiny.rs tiny.rs.
• short_sentences. The paper flags at least 50% short lines §5.1. The code flags when short lines are strictly more than floor(0.5 × lines) sentence_filter.rs. For example, 2 short lines out of 4 is 50% but is not flagged.
• header and footer. The paper gives no window §5.1. The code checks the first and last 20% of lines and flags when more than half of that window is short, where short means fewer than 100 bytes header.rs header.rs header.rs.
• noisy. The paper's footnote lists "Ms" as the spacing-mark category, but the category list uses Mc; the code counts letters and all marks (Mn, Mc, Me) as letters §5.2 noisy.rs. The code flags when non-letter characters exceed floor(0.5 × characters) noisy.rs noisy.rs.
• Removal of noisy and tiny documents. The paper treats annotations as non-destructive §5. The code drops documents whose annotation list is exactly ["noisy", "tiny"] pipeline.rs. Tiny is added before noisy pipeline.rs, so the order may not match. The annotation store is in the external oscar-io crate, which is not in this repository, so whether the removal ever fires is not verified.
• Adult annotation. The paper names an "adult" annotation §5.3. The code stores the UT1 categories for the URL in the metadata categories field, and the category names are not limited to "adult" content_detector.rs.
• Hashing. The paper does not describe a per-document hash §6.1.4. The code computes a TLSH hash and stores it in metadata but does not remove any documents on that basis lsh.rs pipeline.rs.
• Perplexity annotation. The code adds an optional KenLM perplexity annotation behind a feature flag, which the paper does not describe. With the feature on and no KenLM path, the code panics pipeline.rs pipeline.rs.
• Language identification model. The paper does not name the FastText model file. The code takes its path as an input pipeline.rs.

Pitfalls

• Printed equations have typos. The paper writes |P_j| and s_j p_j for the byte-weighted confidence §4.2. The intended form, which the code also uses, is the sum of s_i p_i over the lines labelled j, divided by the total bytes model.rs.
• Multilingual denominator is ambiguous. The formula has n+1 with n lines, while the worked example has m+1 with m languages §4.1. The code uses the number of distinct labels. Pick one and test it on a sample before you rely on the counts.
• Unnumbered equations. None of the display equations has a number in the source, so this card cites them by section.
• Strict and non-strict comparisons. short_sentences uses "at least 50%" in the paper and a strict "more than floor(0.5 × lines)" in the code. tiny uses "5 lines or fewer" in the paper and "fewer than 5" in the code. Both differ at the boundary.
• Units. The paper uses characters for the length bound §6.2.5 and the code uses codepoints in the sentence filter and bytes in the header check. For non-ASCII text the two differ, and the boundary of exactly 100 is treated differently by the two filters.
• Hard bounds. Many documents sit exactly at the minimum size, because the first filter removes short lines §6.2.5. Thresholds on line counts and lengths therefore put documents on either side of a cliff.
• Carbon numbers do not match the formula. With t = 42.6 hours, c = 2 CPUs at 125 W each and 20 W of DRAM app. A, the printed formula gives about 18.2 kWh. The reported 0.4266 kWh equals 1.58 × 270 / 1000, so the time factor t seems to be missing. The reported 16.48 g CO2e follows from 0.4266 kWh. Treat both figures as unverified.
• Quality is not established. The authors say their results are not enough to state the intrinsic quality of the clean documents §6.2.3. "Clean" means "carries no annotation", not "high quality".
• Adult annotation is weak. Of the first 100 French adult documents, 21 were false positives and 2 were LGBTQI+ sites §6.2.4. The authors suggest model-based annotation instead §7.2.
• Multilingual documents are one group. They sit in the "Multilingual" group in the per-language table, not in the per-language groups tab. 2.
• Boilerplate survives filtering. Headers, footers and sidebars remain in many documents; the authors note this as a source of English in multilingual documents §6.2.2 §7.2.
• Deduplication. The main corpus is not deduplicated. A line-level deduplicated English version is distributed separately §6.1.4.
• Figure numbers are counted from the source order and should be checked against the PDF.

Lineage

• Builds on: the OSCAR 2019 pipeline (goclassy, from Ortiz Suarez et al.) and Ungoliant, which produced OSCAR 21.09 (Abadji et al.) §2. The paper gives these as author-year references without arXiv ids.
• Compared against: CCNet and mC4, which classify Common Crawl at document level §2. The paper gives no arXiv ids for them.
• Superseded by: not known from content already in this wiki.