Language identification

Overview

Language identification (LangID) predicts which language a piece of text or speech is in. It is the step that sorts a web crawl into per-language collections, and it is a task in its own right for speech models. The paper cards in this wiki use it in three ways: labelling lines or documents of web text 2201.06642 §4, filtering sentence pairs in parallel corpora 2103.12028 §4.3, and recognising the spoken language of a clip 2205.12446 §4.2.

The invariant all variants share is that the label is a prediction with a confidence score, not a fact. The web-corpus papers show what that means in practice: the labels that come out of automatic LangID are often wrong, and the errors are not spread evenly. Small languages and short lines suffer most 2103.12028 §4.2 2103.12028 §7.

Variants

Line-level identification with a confidence threshold

Each line of a crawled page gets its most likely language from a FastText classifier, together with its probability. A line whose top probability is below 0.8 gets an "unknown" label 2201.06642 §4. The paper argues that this approach splits multilingual pages into runs of lines that share a language, which breaks the document apart 2201.06642 §2. Card: Towards a Cleaner Document-Oriented Multilingual Crawled Corpus.

Document-level labelling with a multilingual test

The same per-line labels are used, but the decision is taken for the whole document. Lines are weighted by their size in bytes. A document counts as multilingual when two to five languages each cover at least a set share of its bytes, and the unknown share stays under the same bound [arxiv:2201.06642 §4.1]. Otherwise the largest language is kept if its byte-weighted confidence reaches 0.6 2201.06642 §4.2. Before labelling, leading and trailing short lines are removed 2201.06642 §3.1. Card: Towards a Cleaner Document-Oriented Multilingual Crawled Corpus.

N-gram and Transformer LangID as corpus filters

The audited corpora were filtered with an automatic LangID. The CLD3 n-gram model was run on CCAligned sentence pairs and compared with human labels 2103.12028 §4.3. The authors also trained a Transformer-based LangID model and applied it to remove sentences 2103.12028 §4.3. Card: Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets.

Speech language identification

A pre-trained speech encoder is fine-tuned to output one of 102 languages. The benchmark paper uses two 600M-parameter encoders for this: a speech-only model and a model pre-trained on speech and text 2205.12446 §3.1 [arxiv:2205.12446 §4.2]. Speakers in the training data differ from those in the dev and test data, because the authors consider speaker overlap a risk for this task 2205.12446 §4.2. Card: FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech.

When to use which

Speech LangID results depend strongly on the language group. Accuracy is about 52% for the SA group and about 85% for WE in the speech-only model 2205.12446 tab. 4, so a single average hides the weak groups.

Implementation sketch

Document-level labelling as the paper describes it. Thresholds are the paper's values; the released code uses some different ones (see Contradictions).

# Document-level language label from per-line LangID  [arxiv:2201.06642 §3-§4]
# Input:  lines of one document; line_lid(line) -> (language, probability)
# Output: (label, confidence), or None when the document is dropped

def label_document(lines, line_lid):
    # Drop short lines at the head and tail  [arxiv:2201.06642 §3.1]
    lines = trim_short_head_and_tail(lines, min_chars=100)
    if not lines:
        return None
    # Per-line label; probability below 0.8 becomes "unknown"  [arxiv:2201.06642 §4]
    labels = [line_lid(line, threshold=0.8) for line in lines]
    sizes = [len(line.encode("utf-8")) for line in lines]
    # Multilingual test with threshold |D| / (n + 1)  [arxiv:2201.06642 §4.1]
    if is_multilingual(labels, sizes):
        return ("multi", 0.5)  # value used by the released code
    # Largest language by bytes, kept only at confidence 0.6 or more  [arxiv:2201.06642 §4.2]
    language, confidence = largest_language(labels, sizes)
    if language is None or confidence < 0.6:
        return None
    return (language, confidence)

Sampling check before using or releasing a labelled corpus. The audits used about 100 random lines per language, rated by one person each [arxiv:2103.12028 §4.1] 2103.12028 §7.

# Manual audit of a labelled corpus  [arxiv:2103.12028 §4.1] [arxiv:2103.12028 §7]
# Input:  lines grouped by language label
# Output: share of lines per label, for each language

def audit(lines_by_language, rate, sample_size=100):
    report = {}
    for language, lines in lines_by_language.items():
        # Draw up to 100 random lines  [arxiv:2103.12028 §4.1]
        sample = random_sample(lines, min(sample_size, len(lines)))
        # One human label per line, from the taxonomy  [arxiv:2103.12028 §4.1]
        labels = [rate(line, language) for line in sample]
        report[language] = label_shares(labels)
    return report

Contradictions

No cross-paper disagreement on the same claim. The three cards measure different things (documents, sentence pairs and speech), so their numbers are not directly comparable.

• Within one paper: the document-level rules differ from the released code. The paper requires 5 lines for the multilingual test 2201.06642 §4.1. The released code requires 10 lines and a further confidence gate, and it labels multilingual documents with a fixed confidence of 0.5 [arxiv:2201.06642 §4.1]. The card records the code values as found; this page does not decide which one is correct.
• Automatic LangID as a fix versus as a building block. One paper uses line-level and document-level LangID as the core of its corpus pipeline and reports no accuracy for it 2201.06642 §4. Another paper measures automatic LangID on sentence pairs and finds 40.6% average precision 2103.12028 §4.3. The two sets of measurements cover different units, so the gap is not resolved here.

Papers

• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets: audits 205 language subsets of five web-crawled corpora, and tests CLD3 and a Transformer LangID filter.
• Towards a Cleaner Document-Oriented Multilingual Crawled Corpus: rebuilds OSCAR with one language label per document and a multilingual test.
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech: 102-language speech benchmark with a speech language-identification task.