---
kind: concept
slug: language-identification
title: Language identification
summary: >-
  Assigning a language label to text lines, documents or speech, used to sort web-crawled corpora and to benchmark speech models. Covers line-level and document-level labelling, n-gram and Transformer LangID filters, and speech LangID. Every variant guesses, and the audited corpora show that wrong labels remain common, so check labels before use.
tags: [data-quality, web-crawl, multilingual, speech, data-filtering]
papers: ["2103.12028", "2201.06642", "2205.12446"]
related: [dataset-curation, multilingual-evaluation, language-model-pretraining]
status: draft
updated_at: 2026-11-01
---

## Overview

Language identification (LangID) predicts which language a piece of text or
speech is in. It is the step that sorts a web crawl into per-language
collections, and it is a task in its own right for speech models. The paper
cards in this wiki use it in three ways: labelling lines or documents of web
text [arxiv:2201.06642 §4], filtering sentence pairs in parallel corpora
[arxiv:2103.12028 §4.3], and recognising the spoken language of a clip
[arxiv:2205.12446 §4.2].

The invariant all variants share is that the label is a prediction with a
confidence score, not a fact. The web-corpus papers show what that means in
practice: the labels that come out of automatic LangID are often wrong, and
the errors are not spread evenly. Small languages and short lines suffer most
[arxiv:2103.12028 §4.2] [arxiv:2103.12028 §7].

## Variants

### Line-level identification with a confidence threshold

Each line of a crawled page gets its most likely language from a FastText
classifier, together with its probability. A line whose top probability is
below 0.8 gets an "unknown" label [arxiv:2201.06642 §4]. The paper argues
that this approach splits multilingual pages into runs of lines that share a
language, which breaks the document apart [arxiv:2201.06642 §2]. Card:
[Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642).

### Document-level labelling with a multilingual test

The same per-line labels are used, but the decision is taken for the whole
document. Lines are weighted by their size in bytes. A document counts as
multilingual when two to five languages each cover at least a set share of
its bytes, and the unknown share stays under the same bound [arxiv:2201.06642
§4.1]. Otherwise the largest language is kept if its byte-weighted confidence
reaches 0.6 [arxiv:2201.06642 §4.2]. Before labelling, leading and trailing
short lines are removed [arxiv:2201.06642 §3.1]. Card: [Towards a Cleaner
Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642).

### N-gram and Transformer LangID as corpus filters

The audited corpora were filtered with an automatic LangID. The CLD3 n-gram
model was run on CCAligned sentence pairs and compared with human labels
[arxiv:2103.12028 §4.3]. The authors also trained a Transformer-based LangID
model and applied it to remove sentences [arxiv:2103.12028 §4.3]. Card:
[Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](/papers/2103.12028).

### Speech language identification

A pre-trained speech encoder is fine-tuned to output one of 102 languages. The
benchmark paper uses two 600M-parameter encoders for this: a speech-only model
and a model pre-trained on speech and text [arxiv:2205.12446 §3.1] [arxiv:2205.12446
§4.2]. Speakers in the training data differ from those in the dev and test
data, because the authors consider speaker overlap a risk for this task
[arxiv:2205.12446 §4.2]. Card: [FLEURS: Few-shot Learning Evaluation of
Universal Representations of Speech](/papers/2205.12446).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Building a corpus from web dumps where pages can mix languages | Document-level labelling with the multilingual test | Line-level labels cut mixed documents apart [arxiv:2201.06642 §2] [arxiv:2201.06642 §4.1] |
| You need a cheap per-line label to threshold | Line-level identification at 0.8 | This is the paper's choice [arxiv:2201.06642 §4]; the paper reports no accuracy for this step |
| Filtering sentence pairs of a parallel corpus | An n-gram LangID, with a manual sample | CLD3 average precision on CCAligned pairs was 40.6% [arxiv:2103.12028 §4.3] |
| Removing off-language text automatically | Not as a sole fix | The Transformer filter raised median precision on low-quality corpora from 13.8% to 43.9%, but lost about 77.5% of recall [arxiv:2103.12028 §4.3] |
| Spoken-language ID over many languages, one model | Speech LangID fine-tuned on all 102 languages | Average accuracy is 71.4% (speech-only) and 73.3% (speech and text) [arxiv:2205.12446 tab. 4] |
| Choosing an encoder for a region | Check the per-region rows | Speech-only was ahead in WE, SA and CJK; speech and text was ahead in EE, CMN, SSA and SEA [arxiv:2205.12446 tab. 4] |
| Any corpus release | Read about 100 sampled lines per language, and check the language codes | The authors recommend this check [arxiv:2103.12028 §7]; many codes were nonstandard, superset or deprecated [arxiv:2103.12028 §5] |

Speech LangID results depend strongly on the language group. Accuracy is
about 52% for the SA group and about 85% for WE in the speech-only model
[arxiv:2205.12446 tab. 4], so a single average hides the weak groups.

## Implementation sketch

Document-level labelling as the paper describes it. Thresholds are the
paper's values; the released code uses some different ones (see Contradictions).

```pseudocode
# Document-level language label from per-line LangID  [arxiv:2201.06642 §3-§4]
# Input:  lines of one document; line_lid(line) -> (language, probability)
# Output: (label, confidence), or None when the document is dropped

def label_document(lines, line_lid):
    # Drop short lines at the head and tail  [arxiv:2201.06642 §3.1]
    lines = trim_short_head_and_tail(lines, min_chars=100)
    if not lines:
        return None
    # Per-line label; probability below 0.8 becomes "unknown"  [arxiv:2201.06642 §4]
    labels = [line_lid(line, threshold=0.8) for line in lines]
    sizes = [len(line.encode("utf-8")) for line in lines]
    # Multilingual test with threshold |D| / (n + 1)  [arxiv:2201.06642 §4.1]
    if is_multilingual(labels, sizes):
        return ("multi", 0.5)  # value used by the released code
    # Largest language by bytes, kept only at confidence 0.6 or more  [arxiv:2201.06642 §4.2]
    language, confidence = largest_language(labels, sizes)
    if language is None or confidence < 0.6:
        return None
    return (language, confidence)
```

Sampling check before using or releasing a labelled corpus. The audits used
about 100 random lines per language, rated by one person each [arxiv:2103.12028
§4.1] [arxiv:2103.12028 §7].

```pseudocode
# Manual audit of a labelled corpus  [arxiv:2103.12028 §4.1] [arxiv:2103.12028 §7]
# Input:  lines grouped by language label
# Output: share of lines per label, for each language

def audit(lines_by_language, rate, sample_size=100):
    report = {}
    for language, lines in lines_by_language.items():
        # Draw up to 100 random lines  [arxiv:2103.12028 §4.1]
        sample = random_sample(lines, min(sample_size, len(lines)))
        # One human label per line, from the taxonomy  [arxiv:2103.12028 §4.1]
        labels = [rate(line, language) for line in sample]
        report[language] = label_shares(labels)
    return report
```

## Contradictions

No cross-paper disagreement on the same claim. The three cards measure
different things (documents, sentence pairs and speech), so their numbers are
not directly comparable.

- **Within one paper: the document-level rules differ from the released code.**
  The paper requires 5 lines for the multilingual test [arxiv:2201.06642 §4.1].
  The released code requires 10 lines and a further confidence gate, and it
  labels multilingual documents with a fixed confidence of 0.5 [arxiv:2201.06642
  §4.1]. The card records the code values as found; this page does not decide
  which one is correct.
- **Automatic LangID as a fix versus as a building block.** One paper uses
  line-level and document-level LangID as the core of its corpus pipeline and
  reports no accuracy for it [arxiv:2201.06642 §4]. Another paper measures
  automatic LangID on sentence pairs and finds 40.6% average precision
  [arxiv:2103.12028 §4.3]. The two sets of measurements cover different units,
  so the gap is not resolved here.

## Papers

- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](/papers/2103.12028): audits 205 language subsets of five web-crawled corpora, and tests CLD3 and a Transformer LangID filter.
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642): rebuilds OSCAR with one language label per document and a multilingual test.
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](/papers/2205.12446): 102-language speech benchmark with a speech language-identification task.