XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

In plain words

Speech models that learn from unlabelled audio are usually built for one language. Languages with little data gain little from them, which is the gap this paper targets.

XLS-R tries to cover 128 languages with one model. It is trained on audio from parliament recordings, audiobooks, read sentences, YouTube clips and telephone conversations. Training works by hiding parts of the audio and asking the model to pick the right hidden piece from a set of wrong ones. Languages with little data are sampled more often than their size alone would give, so they are not drowned out.

Afterwards the model is adapted to specific tasks: translating speech into English or from English, transcribing speech, and recognising which language is spoken or who is speaking. Bigger versions of the model gained more on the low-resource translation directions than on the high-resource ones.

TL;DR

XLS-R is a family of cross-lingual wav2vec 2.0 models (0.3B, 1B and 2B parameters) pretrained on about 436K hours of unlabelled speech in 128 languages §3.1 tab. 2. Languages are sampled with an upsampling exponent of 0.5 §4.1. Fine-tuned for speech translation, the 2B model gets 22.1 average BLEU on CoVoST-2 into English versus 14.7 for the prior best. On English-to-X translation its 15-direction average (27.8) is above an English-only 720M model's (26.6) §5.1. It also reports new best results on several speech recognition sets and VoxLingua107 language ID §5.2 §5.3. All numbers are author-reported. Read the Pitfalls section before reimplementing: several numbers in the text do not match the tables, and the released fine-tuning config differs from the paper in places.

Problem

• Self-supervised speech models were mostly English or monolingual. The largest cross-lingual model before this work, XLSR-53, used about 50K hours of data in 53 languages and about 300M parameters §1.
• Scaling was studied for monolingual speech models with a billion or more parameters, but cross-lingual speech pretraining stayed much smaller §1.
• The question is whether one cross-lingual model can serve many languages, including low-resource ones, without losing much on high-resource ones. The paper notes that a model that must represent many languages is at a disadvantage against an English-only model on English input §5.1.2.

Method

XLS-R is wav2vec 2.0 pretraining on a large multilingual pool of unlabelled speech, with more model capacity than earlier cross-lingual work §1 §2.

0. Training data. Five corpora: VoxPopuli (parliamentary speech, 372K h), MLS (audiobooks, about 50K h), CommonVoice (read speech), VoxLingua107 (YouTube, 6.6K h) and BABEL (telephone conversations, about 1K h) §3.1. Together they cover 128 languages, 436K hours in total fig. 2 tab. 1.
1. Architecture. A convolutional feature encoder maps raw audio to latent frames (25 ms windows every 20 ms), a Transformer produces context vectors, and a quantiser turns the latents into discrete targets using Gumbel-softmax codebooks §2. The Transformer follows BERT §2. The three sizes differ in depth, width and feed-forward size tab. 2.
2. Self-supervised objective. Spans of 10 time steps are masked at random start indices. For each masked step the model must identify the true quantised latent among K = 100 distractors drawn from other masked steps, scored by cosine similarity. A codebook diversity penalty is added §2. The weighting of the penalty and the masking probability are not stated in this paper.
3. Multilingual sampling. Batches mix languages. The probability of a language is proportional to (n_l / N) raised to alpha, where n_l is its unlabelled hours, and alpha = 0.5 §2 §4.1. The same rule is then applied to corpora, treating each corpus as a language §4.1. See the pseudocode.
4. Large-scale training. Models are optimised with Adam, with 32K warm-up steps followed by polynomial decay to zero, for one million updates. Audio is cropped to 320K samples (20 s). Pretraining uses a fully sharded backend and activation checkpointing to fit memory §4.1 tab. 2.
5. Downstream fine-tuning. Speech recognition adds a linear CTC head §4.3. Speech translation adds a 12-layer Transformer decoder initialised from mBART; the encoder is updated after the first 10K fine-tuning updates §4.2.

# Self-supervised pretraining step (wav2vec 2.0 objective)  [§2] [§4.1]
# Input:  raw audio batch (B, samples), drawn across languages and corpora
# Output: contrastive loss plus codebook diversity penalty (scalar)

def pretraining_step(audio, model):
    # Map 25 ms windows, strided by 20 ms, to latent frames  [§2]
    latents = model.feature_encoder(audio)           # (B, T, d)

    # Quantise latents to discrete targets with Gumbel-softmax codebooks  [§2]
    targets = model.quantizer(latents)               # (B, T, d)

    # Mask spans of 10 steps at random start indices  [§2]
    masked = mask_spans(latents, span_length=10)

    # Transformer context for every time step  [§2]
    contexts = model.transformer(masked)             # (B, T, d)

    # For each masked step, pick the true target among K = 100 distractors
    # sampled from other masked steps  [§2]
    contrastive = 0.0
    for step in masked_steps:
        distractors = sample_distractors(targets, step, k=100)
        contrastive += contrastive_loss(
            contexts[step], targets[step], distractors
        )

    # Codebook diversity penalty, so that all codebook entries get used  [§2]
    diversity = diversity_penalty(model.quantizer)

    # Relative weighting of the two terms is not stated  [§2]
    return contrastive + diversity

# Language and corpus sampling probabilities  [§2] [§4.1]
# Input:  unlabelled hours per language (n_l), alpha = 0.5
# Output: sampling probability per language

def sampling_probabilities(hours, alpha):
    # N is the total of all hours; the paper does not define it explicitly
    total = sum(hours)
    # Unnormalised weight (n_l / N) ** alpha  [§2]
    weights = [(n_l / total) ** alpha for n_l in hours]
    # Normalising the weights to sum to one is an assumption, not stated  [§2]
    return [weight / sum(weights) for weight in weights]

# ASR fine-tuning learning-rate schedule  [§4.3]
# Input:  update index, total updates, peak learning rate
# Output: learning rate for this update

def asr_learning_rate(update, total_updates, peak_lr):
    # Phases are 10% warm-up, 40% hold and 50% decay  [§4.3]
    warmup_steps = 0.10 * total_updates
    hold_end = 0.50 * total_updates
    decay_steps = 0.50 * total_updates

    # Warm-up start: 0.01 * peak, the scheduler default (paper: not stated)
    # [code: examples/wav2vec/xlsr/config/finetune.yaml#L49-L52]
    # [code: fairseq/optim/lr_scheduler/tri_stage_lr_scheduler.py#L38-L40]
    start_lr = 0.01 * peak_lr
    if update < warmup_steps:
        # Linear ramp from start to peak
        # [code: fairseq/optim/lr_scheduler/tri_stage_lr_scheduler.py#L69-L73]
        return start_lr + (peak_lr - start_lr) * update / warmup_steps

    # Hold at the peak rate  [§4.3]
    if update < hold_end:
        return peak_lr

    # Exponential decay to 0.05 * peak; the paper says "decayed to zero"  [§4.3]
    # [code: fairseq/optim/lr_scheduler/tri_stage_lr_scheduler.py#L81-L86]
    decay_factor = -math.log(0.05) / decay_steps
    if update < total_updates:
        return peak_lr * math.exp(-decay_factor * (update - hold_end))
    return 0.05 * peak_lr

Hyperparameters

Pretraining values come from §4.1 unless noted. Fine-tuning values are split between speech translation (ST) and speech recognition (ASR). "Not stated" means the paper does not give the value.

Results

Author-reported. Setup: multilingual pretraining on the corpora in §3.1, then task-specific fine-tuning as in §4. Baselines are the previous results each table names, and the ablations use a smaller decoder and a shorter setup, so compare rows only within one table.

Speech translation, CoVoST-2 X to English (21 directions, BLEU)

Average over the 21 directions, from tab. 3 (rows: §5.1.1). Full per-direction numbers are in tab. 14 (appendix).

Speech translation, CoVoST-2 English to X (BLEU)

Four directions and the 15-direction average, from tab. 4 (rows: §5.1.2).

The text gives the four-direction average as 25.6 for LV-60K with self-training and LM, and 25.5 for XLS-R 2B §5.1.2.

Speech translation ablations (BLEU, XLS-R 1B)

From tab. 5 and tab. 6 (§5.1.3). Multilingual fine-tuning versus bilingual fine-tuning uses a reduced set of directions and a smaller decoder, and the caption marks these numbers as not directly comparable to the rest of the paper.

Speech recognition

Word error rate (WER) unless stated; lower is better. Author-reported.

BABEL (5 languages, WER, no LM for the table values; §5.2.1, tab. 7) Labeled data per language is 55h (as), 76h (tl), 30h (sw), 59h (lo), 46h (ka).

CommonVoice (10 languages, phoneme error rate, one hour of labels per language, no LM; §5.2.2, tab. 8)

MLS (8 languages, 10 hours of labels per language unless marked full, WER with a 4-gram LM except Polish; §5.2.3, tab. 9)

Labeled hours in the full row: 44.7K (en), 2K (de), 1.6K (nl), 1.1K (fr), 918 (es), 247 (it), 161 (pt), 104 (pl) tab. 9.

VoxPopuli (14 languages, WER, no LM; §5.2.4, tab. 10)

LibriSpeech (English, WER without LM; §5.2.5, tab. 11)

Speech classification

VoxLingua107 language ID (error rate in %, development set of 33 languages; §5.3.1, tab. 12)

VoxCeleb1 speaker identification (accuracy %; §5.3.2, tab. 13)

The paper notes that the SUPERB numbers are not strictly comparable, because SUPERB does not fine-tune the underlying model §5.3.2.

Paper vs code

The code is the official fairseq repository (pytorch/fairseq, commit 3d262bb25690e4eb2e7d3c1309b1e9c406ca4b99), in its examples/wav2vec/xlsr directory. Paths below are relative to the repository root. The directory holds the ASR fine-tuning config (config/finetune.yaml), a README with checkpoint links and recipes, and scripts for language ID and speaker classification. It does not contain the XLS-R pretraining config or the speech translation recipe, so the pretraining and ST values above are not checked against code.

Differences and inconsistencies found:

• BABEL updates. The paper gives 20K updates for BABEL §4.3; the README gives 26000 README.md.
• Learning-rate end point. The paper says the rate decays to zero §4.3. The config sets final_lr_scale: 0.05 finetune.yaml. The tri-stage scheduler decays exponentially to 0.05 times the peak, then holds that value tri_stage_lr_scheduler.py.
• Warm-up start. The paper says the rate is warmed up §4.3 and does not give a start value. The scheduler ramps linearly from 0.01 times the peak, its default, as the config does not override it tri_stage_lr_scheduler.py tri_stage_lr_scheduler.py.
• Freezing for ASR. The paper gives a 10K-update encoder freeze only for speech translation §4.2. The ASR config also sets freeze_finetune_updates: 10000 finetune.yaml.
• Layer-level regularisation. The paper uses stochastic depth of 10%, and 15% for 2B §4.3. The config sets layerdrop: 0.1 finetune.yaml. The README lists the 2B changes README.md but does not give a layer-drop value for 2B.
• Masking probability. The paper's range is 30% to 75% §4.3. The config default is 0.75 finetune.yaml, and the README says the CommonVoice 2B runs chose 0.30 or 0.40 per language README.md. These are consistent but the default is not the paper's choice for every benchmark.
• Batch size. The paper gives 0.44 h for the 0.3B and 1B models §4.3. The config gives max_tokens: 1280000 and update_freq: [5] on 4 GPUs finetune.yaml finetune.yaml finetune.yaml. That is 25.6M samples, about 0.44 h at 16 kHz (derived). For 2B, the README gives max_tokens of 2560000 and a total of 2560000 * 24 samples README.md, about 1.07 h (derived), against the paper's 1.06 h §4.3.
• Language identification split. Section 3.2.3 says the test set has 33 languages §3.2.3, Section 5.3.1 says the development set has 33 languages §5.3.1. The README's inference command uses --gen-subset test and reports 5.66% error overall README.md README.md. The paper's average of 5.7% matches that number, so the split label in the text is likely the mistake, but the paper does not say.
• Released checkpoints. The README links checkpoints for 300M, 1B and 2B pretraining, speech translation models for 21 to English and 15 from English, a joint 21-plus-15 checkpoint for 2B, and a VoxLingua107 fine-tuned checkpoint README.md README.md. The joint checkpoint and the LID checkpoint are not described in the paper.

No further discrepancies were found in the checked ASR settings: the optimiser, the three-phase 10/40/50 structure and the learning-rate ranges match §4.3.

Pitfalls

• Numbers in the text that do not match the tables.
  • BABEL: the text says XLS-R 0.3B beats XLSR-53 by an average of 1.4 WER §5.2.1. The table gives 1.2, 0.0, 2.2 and 3.1 for as, tl, sw and ka, about 1.6 on average. XLSR-53 has no value for lo, so the paper's average cannot be reproduced from the table.
  • MLS: "2.9 WER on average" for 1B §5.2.3 is measured against XLSR-53 (13.8 to 10.9). Against XLS-R 0.3B (12.8) the gain is 1.9.
  • Language-ID comparison: the text cites Ravanelli et al. 2020 §5.3.1, but the table row is labelled SpeechBrain tab. 12. The 1% absolute and 15% relative gains match 6.7 to 5.7.
  • High-resource X to English languages: the text lists the high-resource group as 136 to 264 h and includes Spanish §3.2.1. The appendix table lists Spanish at 113 h and German at 184 h tab. 14.
• Abstract wording. The arXiv metadata abstract says cross-lingual pretraining "can outperform" English-only pretraining for English-to-X translation. The LaTeX abstract says it "can perform as well as". The 15-direction average (27.8 vs 26.6) supports the stronger wording only for that average, and the four-direction comparison is at parity (25.5 vs 25.6). Do not claim more than the tables show.
• Ablations are not the main setup. The bilingual baseline uses a 16M-parameter decoder with a 10K BPE vocabulary, while the multilingual model uses the 459M mBART-initialised decoder §4.2. The gap combines two changes, and the caption marks the comparison as not directly comparable tab. 5.
• Pretraining details are not in this paper. The Gumbel temperature, the codebook sizes, the mask probability, the loss weight of the diversity penalty and the peak learning rate are not stated. They come from wav2vec 2.0, which this card does not cover. The reimplementer must take them from the wav2vec 2.0 paper or its code, and should say so.
• Language sampling normalisation. The formula uses a proportionality sign and does not define N or state normalisation §2. The pseudocode normalises and defines N as the total hours, which is an assumption.
• Corpus licences. The training corpora have their own terms. BABEL is distributed by the LDC under catalogue IDs listed in the paper's footnote §3.2.2. This card's licence covers the paper, not those datasets.
• Hidden compute. Pretraining used 128 to 200 GPUs §4.1. Fine-tuning the 2B model needs a fully sharded backend and activation checkpointing README.md, and its batch is about 1 h of audio per update §4.3.
• Few-shot claims rest on small data. The few-shot speech translation gains are measured on 2 h of training data for some directions §5.1.1 tab. 14. Single runs per setting are not reported, so variance is unknown.
• Language-model decoding is tuned per benchmark. The LM weight and word penalty come from Bayesian optimisation with 128 trials per benchmark §4.3. Results with and without an LM are not comparable.

Lineage

• Builds on: wav2vec 2.0 (cited as baevski2020wav; no arXiv id is given in the source, so no card link) and XLSR-53 (cited as conneau2021xlsr; no arXiv id given in the source). XLS-R extends XLSR-53 with more data, more languages and larger models §1.
• Multilingual speech translation design: multilingual fine-tuning and mBART decoder initialisation follow li2021multilingual §5.1.3.
• Compared against: XLSR-53, VP-100K (wang2021voxpopuli), XMEF (li2021multilingual), LV-60K with self-training (wang2021st), VoxLingua107 baselines (valk2020voxlingua107), and SUPERB (yang2021superb) §5.
• Superseded by: not stated in this card's sources.