Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

In plain words

Many public text collections for dozens of languages were built by downloading the web and sorting pages by an automatic language detector. The authors asked whether those collections actually contain the language they claim. They took random samples from five large collections and had people read them.

Many of the small-language collections were mostly junk: navigation menus, product codes, text in another language, or sentence pairs that do not match. Some were labelled with wrong or invented language codes. Two quick checks catch most of these problems: reading about a hundred lines, and looking at the language codes. The authors recommend running them before a dataset is used or published.

TL;DR

A human audit of 205 language subsets from five web-crawled multilingual corpora finds that quality is very uneven, and that the lowest-resource languages are the worst. About 100 sentences per language, judged by one rater each, were enough to show the problems. Automatic language filters did not fix them, and language codes were often wrong. Author-reported.

Problem

Public web-mined corpora (CCAligned, ParaCrawl v7.1, WikiMatrix, OSCAR, mC4) cover hundreds of languages, but their quality had rarely been measured directly §1. Downstream results are reported for a few high-resource languages, so problems in small languages go unseen §4. The paper also notes a publication bias against negative findings about corpora §4.

Method

Corpora. Five releases were audited. CCAligned, ParaCrawl and WikiMatrix are parallel (sentence-pair) data; OSCAR and mC4 are monolingual. Their filtering and language-ID choices differ (document or sentence level; FastText, CLD2 or CLD3 for LangID) tab. 1 §3.

Sampling. For each language in each corpus the authors drew a random sample of up to 100 lines. A line can be a word or a paragraph §4.1. They audited the smallest languages (at least the smallest 10 per corpus) and languages with an available proficient speaker §4.1. Because every language gets the same sample cap, the audited sample is not an unbiased sample of the corpus §4.1.

Raters. 51 volunteers from the NLP community covered about 70 languages. Each sentence was labelled by one rater. Where no speaker was available, raters used dictionaries and web search. Raters were told to be lenient with translation errors if the meaning mostly came through §4.1 app. C.

Error taxonomy §4.1 tab. 2 app. B:

• CC: correct, natural sentence (fragments are fine if about 5 words or more).
• CS: correct but a single word or short phrase; also highly repeated phrases.
• CB: correct but boilerplate or formulaic (for example auto-generated text).
• X: incorrect translation; only for parallel data, both sides in the right language.
• WL: source or target in the wrong language, but both contain linguistic content.
• NL: not linguistic content on at least one side; a lone proper noun counts as NL.
• U: unknown, to be checked with a native speaker. Used as a temporary label.
• C is the union of CC, CS and CB. Raters also flagged offensive and pornographic content.

Aggregation. Per language, each label's share is computed over its sentences §4.2. The macro average gives each language the same weight. The micro average weights each language by its share of sentences §4.2 tab. 3.

# Audit one language subset  [§4.1] [app. B]
# Input:  corpus lines, language code, taxonomy
# Output: one label per sampled line

def audit_language(lines, language, taxonomy):
    # Draw up to 100 lines at random  [§4.1]
    sample = random_sample(lines, max_size=100)
    labels = []
    for line in sample:
        # One rater per line; non-speakers may search the web  [§4.1] [app. C]
        label = rate(line, language, taxonomy)
        labels.append(label)
    return labels

# Aggregate label shares across languages  [§4.2] [tab. 3]
# Input:  labels: dict language -> list of labels
#         weights: dict language -> size weight (see Pitfalls)
# Output: macro and micro share per label

def aggregate(labels, weights):
    # Share of each label inside one language  [§4.2]
    shares = {}
    for language in labels:
        shares[language] = label_fractions(labels[language])
    # Macro: every language counts equally  [§4.2]
    macro = mean_over_languages(shares)
    # Micro: languages weighted by their size  [§4.2] [tab. 3]
    total = sum(weights[language] for language in shares)
    micro = {}
    for language in shares:
        micro[language] = shares[language] * weights[language] / total
    return macro, micro

Automatic filtering experiments §4.3. The authors evaluated the CLD3 n-gram LangID on CCAligned sentence pairs against the human labels. They also trained a Transformer-based LangID model (the paper cites Caswell et al. for the approach) and applied it as a filter to CCAligned sentences.

Language codes §5 app. A. Codes were checked against BCP-47 and ISO 639-3. Mislabels, sign-language codes that carry text in another language, superset codes and deprecated codes were recorded.

Downstream check §6. Audit scores were compared with spBLEU from the M2M-124 model on FLORES-101 for the 21 languages that appear in both.

Hyperparameters

Audit and filter settings. The paper gives no training hyperparameters for a method; these are the parameters of the audit protocol.

Results

Author-reported. Setup: 205 language-specific subsets from five public corpora, each sampled as described in Method. Labels are from one rater per line, so the numbers are rough estimates §4.2 tab. 3.

Languages in each corpus with 0% C: CCAligned 7, ParaCrawl 0, WikiMatrix 1, OSCAR 7, mC4 0. Languages with under 50% C: 44, 4, 19, 11, 9. Across the 205 audited subsets, 15 contain no correct sentence and 87 are under 50% correct §4.2 §7 tab. 3.

Other findings:

• Rank correlation (Spearman) between C% and corpus size is positive for every corpus: mC4 0.66, CCAligned 0.53, WikiMatrix 0.49, ParaCrawl 0.43, OSCAR 0.37 §4.2.
• Non-linguistic content is more common than wrong-language content in the parallel corpora. CCAligned has the highest NL share in the table. mC4 has the highest WL share, with 4 of 48 languages above 50% other-language content §4.2.
• Annotator agreement against experts, on a subset of languages. CCAligned, 10 languages: Acc-6 mean 0.66, Acc-4 0.72, Acc-2 0.79 tab. 4. OSCAR, 6 languages: Acc-6 mean 0.98 tab. 5 §4.2.
• Transformer LangID filter on CCAligned, for corpora below 50% C: median precision rose from 13.8% to 43.9%, and recall fell by 77.5% §4.3.
• CLD3 on CCAligned sentence pairs: average precision 40.6% §4.3.
• Correlation with spBLEU (M2M-124, FLORES-101) over 21 languages: C% rho = 0.44 (p = 0.041). Expected count of good sentences (C% times size) gives rho = 0.73 (p = 0.00013) §6.
• Language codes: 83 nonstandard codes overall (8 in CCAligned, 3 in OSCAR, 1 in mC4, 1 in WikiMatrix, 70 in JW300), plus 59 codes affected by superset issues. At least 48 of the 83 affected corpora are entirely spurious §5 §7. 12% (48 of 417) of JW300 codes are sign-language codes whose text is another language §5.

Paper vs code

No official code. The paper links an annotation download (footnote 1 §1), a data file, not code. Searches: the paper and its footnotes name no code repository and no project page. GitHub repository search for "quality at a glance multilingual audit" and "masakhane audit annotations" returned no results.

Pitfalls

• Macro and micro give different answers. CCAligned is 29% C macro but 54% micro; OSCAR is 87% macro and 99% micro. Say which one you quote tab. 3 §4.2.
• The sample is small and not random across languages. At most 100 lines per language, one rater each, and audited languages were chosen by size and rater availability, so per-language numbers are rough §4.1 §4.2.
• Micro weights are ambiguous. The text says "according to their presence in the overall dataset" §4.2; the caption says "the overall annotated corpus" tab. 3. The paper does not say which was used. The weights input above follows the text.
• Annotation accuracy drops for fine labels. Acc-6 is 0.66 on CCAligned but Acc-2 is 0.79. The taxonomy lacks an "in-language but unnatural" class, CS and CB were not applied consistently, and dialects were not handled §4.2 §7.
• X is not in the monolingual tables. Monolingual corpora have no X label, so the table shows "not applicable" tab. 3.
• Automatic LangID filters are not a fix. CLD3 precision is 40.6%, and the Transformer filter loses 77.5% of recall. Gains are concentrated in a few languages (Lingala 8% to 80%; Oromo 2% to 33%) and come with a loss of about half the correct sentences §4.3.
• Language codes must be checked, not trusted. Superset codes such as sh (Serbo-Croatian) and ambiguous ones such as ar are common. Some codes are deprecated (sh, iw, eml, daf) §5 app. A.
• Downstream correlation is weak evidence. It rests on 21 languages and one translation model §6.
• Recommendations §7: read about 100 sentences from a dataset before use; do not release corpora with near-zero in-language content; publish a per-language quality score and known issues. The paper also suggests that machine-generated text in a corpus is not measured here §7.

Lineage

• Builds on: manual data-quality checks of web corpora (for example Caswell et al. on LangID in web-crawled data) and data statements and datasheets, which the paper cites for documentation §2.
• Compared against: CLD3 (n-gram LangID) and a Transformer LangID model §4.3.
• Related work: an automatic analysis of C4 that finds machine-translated content §2.
• Superseded by: not stated.