FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

In plain words

To test whether a speech model works in many languages, you need the same spoken content recorded in each language, so that results can be compared from one language to another. FLEURS is such a collection: 102 languages, built on sentences that have already been translated by hand, so every language covers the same material.

Each sentence was read aloud by native speakers, and other people then checked the recordings and threw out the bad ones. The whole collection is about 1.4k hours of speech, so each language gets only a modest amount.

The paper also reports starting results on three jobs: turning speech into text, guessing which language is being spoken, and matching a spoken sentence to its written text. Two large pre-trained multilingual models were used as starting points. The one trained on speech alone did slightly better on transcription; the one trained on speech and text did slightly better on language guessing. Only that second model was tested on matching.

TL;DR

FLEURS is a 102-language, n-way parallel corpus of read speech, about 1.4k hours in total. The paper (CC BY 4.0) is meant for few-shot evaluation of multilingual speech models on speech recognition (CER), spoken language identification (accuracy) and speech-text retrieval (P@1). Author-reported: the 600M speech-only w2v-BERT baseline reaches 14.1% average CER and the 600M speech-text mSLAM reaches 14.6%. Use it to benchmark low-resource coverage; expect large gaps between language groups and check the internal inconsistencies in Pitfalls.

Problem

Existing multilingual speech corpora cover about 8 to 60 languages each, and most are aligned at the document level rather than sentence by sentence §1 tab. 1. The authors want an n-way parallel benchmark, meaning the same sentences in every language, that covers over 100 languages and supports several speech tasks at once §1. They say it is the only dataset of more than 100 languages that enables this, "to the best of our knowledge" §1. The benchmark also needs enough speakers, quality checks and a split with speakers held out, so that a model cannot just recognise a voice it has heard before §2.1 §4.2.

Method

The paper contributes a dataset and baselines, not a new model. The sections below describe the data pipeline first, then the three baseline setups.

Sentence source and split §2.1. The sentences are the dev and devtest parts of FLoRes-101, 2009 sentences in total, because the FLoRes test set is not public. They are re-split into 1509 train, 150 dev and 350 test sentences. The sentences come from English Wikipedia and were translated by humans §2.1.

Recording and validation §2.1. For each sentence and language, three native speakers each made one recording. Other workers then judged whether each recording matched its sentence, and invalid recordings were discarded. Each sentence therefore keeps between zero and three recordings. In the first release, about 21.5% of sentences have none left, and the authors plan to fill these gaps later §2.1. Recordings were kept as they were, with no data augmentation such as speed perturbation or simulated reverberation §2.1. The sex ratio was aimed at at least 30/70 where possible §2.1.

Speaker split §2.1. The speech is divided into train, dev and test parts, aiming for a 7:1:2 proportion. Test speakers are disjoint from train and dev speakers. All segments are at most 30 seconds, sampled at 16 kHz §2.1.

Text §2.2. The transcripts are the human translations from FLoRes-101, with the English versions kept. Each sentence comes in three forms: the raw transcript (src_raw), a normalised one (src_norm) and a character-level form (src_char). The paper says src_char should be used for ASR §2.2. The normalisation steps, in the order the paper lists them:

1. Unicode NFC normalisation §2.2.
2. "FST normalisation". The paper does not describe the rules §2.2.
3. Lower-casing, then normalising and removing punctuation §2.2.
4. Splitting words into characters, with a | token marking word boundaries §2.2.

The authors chose characters as the modelling unit because they need the fewest resources for a universal vocabulary and they match the character error rate metric §2.2.

Coverage and statistics §2.3. The 102 languages span 17 language families and 27 writing systems §2.3 fig. 1 fig. 2. The tables sort them into seven regional groups, labelled WE, EE, CMN, SSA, SA, SEA and CJK. The groups cover western and eastern Europe, the Central Asia to North Africa belt, sub-Saharan Africa, South and South-East Asia, and four East Asian languages (CJK: Cantonese, Mandarin, Japanese and Korean) §2.3 tab. 8. The sentence index from 1 to 2009 links the same sentence across languages §2.3. Per-region hour and token counts are in tab. 2.

Seen and unseen languages §3.2. A language is seen if its speech was in the pre-training data of the baseline models, and unseen otherwise. The paper lists the languages in each group by region §3.2. Section 3.2 counts 54 seen and 48 unseen languages, but the lists contain 55 and 47. See Pitfalls.

Baseline models §3.1. Both baselines are 600M-parameter models:

• w2v-BERT (0.6B, called w2v-bert-51 in the paper): speech-only pre-training on 429k hours of unlabelled speech in 51 languages, drawn from VoxPopuli, MLS, CommonVoice and BABEL §3.1.
• mSLAM (0.6B): pre-trained jointly on the same speech and on more than 10 TiB of unlabelled text in 101 languages §3.1.

The paper defers the fine-tuning details to the mSLAM paper §3.1 §4.1 §4.2.

Speech recognition §4.1. The baseline puts two LSTM layers on top of the pre-trained model and trains with a CTC loss over a 6100-character vocabulary built from src_norm. No language model is used during scoring, and no language-ID label is fed to the model. Each model is fine-tuned on all 102 locales. The paper reports character error rate (CER).

# ASR baseline: pre-trained encoder, 2 LSTM layers, CTC  [§4.1]
# Input:  waveforms (B, T) at 16 kHz  [§2.1]
# Output: per-frame character scores for a CTC loss  [§4.1]

def asr_forward(waveforms):
    # Encode the audio with the pre-trained model (w2v-BERT or mSLAM)  [§3.1]
    frames = pretrained_encoder(waveforms)     # (B, T', d)
    # Two LSTM layers over the frames  [§4.1]
    # Hidden size and frame rate are not stated
    hidden = lstm_stack(frames, num_layers=2)  # (B, T', h)
    # Score the 6100-character vocabulary at each frame  [§4.1]
    # Whether a CTC blank symbol is added is not stated
    scores = output_projection(hidden)         # (B, T', 6100)
    return scores

Language identification §4.2. Fine-tuning follows the mSLAM paper. The task covers all 102 languages tab. 4. Speakers differ between the train set and the dev and test sets, and the authors say avoiding overfitting to speaker identity is essential for this task §4.2.

Speech-text retrieval §4.3. Only mSLAM (0.6B) is fine-tuned, on the training data of all languages. Speech and text embeddings are trained with an additive-margin softmax and in-batch negatives, and the loss is applied in both directions (speech to text and text to speech) §4.3. Text embeddings are built from src_norm §4.3. Scores are reported as P@1 against a database of text keys from the FLEURS test set §4.3.

# Speech-text retrieval, additive-margin softmax  [§4.3]
# Input:  a batch of matched speech and text pairs  [§4.3]
# Output: a bi-directional training loss  [§4.3]

def retrieval_loss(speech, text):
    # Embed both sides with the fine-tuned mSLAM encoder  [§4.3]
    speech_emb = embed_speech(speech)       # (B, d); pooling not stated
    text_emb = embed_text(text)             # (B, d); input is src_norm [§4.3]
    # Score every speech against every text in the batch  [§4.3]
    # The similarity function is not stated in the paper
    scores = similarity(speech_emb, text_emb)  # (B, B)
    # The matching pair is on the diagonal; the rest are negatives  [§4.3]
    # The additive margin value is not stated  [§4.3]
    loss_s2t = softmax_loss(scores, margin)    # speech -> text
    loss_t2s = softmax_loss(scores.T, margin)  # text -> speech  [§4.3]
    # Both directions are combined; the weighting is not stated  [§4.3]
    return loss_s2t + loss_t2s

The paper says "we add" the two directions, but gives no weighting. Hyperparameters for the retrieval fine-tuning appear only in a LaTeX table that the build does not include (see Hyperparameters).

Hyperparameters

The retrieval hyperparameter table is defined in the LaTeX source (raw/src/2205.12446/tables.tex, lines 996 to 1016), but template.tex never includes it. It may therefore be absent from the published PDF. Check the PDF before you rely on these values.

Results

Author-reported. Setup: FLEURS, evaluated on the test split §4.1.2; baselines are the two 600M models in §3.1; hardware is not stated.

Speech recognition, CER (lower is better) tab. 3. Both models are fine-tuned on all 102 languages. The source's seen-versus-unseen breakdown is commented out, so only the per-region numbers for all languages appear here.

Per-language CER is in tab. 6.

Speech language identification, accuracy (higher is better) tab. 4. The text calls the overall figure a macro-average §4.2. The table only says "Avg.".

Speech-text retrieval, P@1 (higher is better) tab. 5 tab. 7. Only mSLAM is fine-tuned for this task, so there is no second model to compare with.

Per-language retrieval scores are in tab. 7.

Paper vs code

No official code.

The arXiv abstract page lists no code repository and no project page. The paper only names dataset hosts: TensorFlow Datasets and Hugging Face, both under the name xtreme_s [abstract, footnote]. These are data releases, not code.

What I searched:

• Links in the paper and on the arXiv page: none to code.
• GitHub repository search for fleurs. Matches were unrelated or third-party (e.g. a benchmark of forced aligners, PalabraAI), so none was used.
• GitHub search inside the google-research organisation, which the authors' affiliation points to. The one hit, google-research/xtreme (Apache-2.0), is the original XTREME benchmark. I shallow-cloned it at commit 838c13b69daafb9328785d16caae2711e4012123. A text search found no FLEURS and no XTREME-S code (only a README image caption about XTREME). The clone was then removed, and no code field is recorded.
• Hugging Face dataset card for google/xtreme_s. It links to a fine-tuning script in the huggingface/transformers research projects. That script is maintained by Hugging Face staff, not the authors, so it was not used.

Pitfalls

• Data hours disagree. The abstract says about 12 hours of speech per language. tab. 2 totals 1390 h (987 train, 120 dev, 283 test), about 13.6 h per language, and tab. 1 says 1.4k hours. Use the table totals, and say which split you mean.
• Seen and unseen counts disagree with the lists. Section 3.2 says 54 seen and 48 unseen languages §3.2. The lists in that section have 55 and 47, and the per-region totals in tab. 3 match the lists. Use the lists.
• Which characters feed ASR. Section 2.2 says src_char "should be used for ASR" §2.2. Section 4.1 says the vocabulary is built from src_norm §4.1. The paper does not resolve this. Pick one, and say which in your report.
• The published numbers may not match a re-release. The first release has missing sentences (about 21.5%) and the authors say gaps will be filled in later §2.1. The paper does not say which version the results used. Pin the version you train and test on.
• "Gains" in the text do not match the table. The text says most gains from multimodal pre-training appear in WE, EE and CMN §4.1. tab. 3 shows mSLAM ahead only in WE (10.6 against 10.7). It is behind in EE (10.0 against 9.9) and CMN (14.8 against 14.5).
• "0.5% regression" is 0.5 CER points. The gap between 14.6 and 14.1 is 0.5 absolute CER points, not a relative percentage §4.1 tab. 3.
• Averages are not defined. The paper does not say whether the 14.1 / 14.6 CER average and the 71.4 / 73.3 accuracy average are macro (per language) or micro (per utterance). The text names the LangID average as macro §4.2; the table only says "Avg." tab. 4.
• Retrieval definitions are loose. The text-to-speech score is described as "retrieving any of the speakers who speaks the correct textual query" §4.3. It is unclear whether a hit means any utterance of that text by any speaker. The size of the candidate database is not stated.
• Retrieval hyperparameters may be missing from the PDF. The table is in the source but is not inserted into the build (see Hyperparameters).
• Unstated fine-tuning details. The ASR and LangID fine-tuning values are deferred to the mSLAM paper §3.1 §4.1 §4.2, which is not in this wiki. An implementation must take them from that paper or set them itself, and results will not be exact reproductions otherwise.
• Tokenisation is under-specified. "FST normalisation" has no rules in the paper §2.2. Exact text matching against the released src_norm strings is the only safe way to reproduce the tokens.
• Large group gaps are real. CJK CER is about 24.6 to 25.0 against about 10.6 to 10.7 for WE tab. 3. Retrieval for CJK falls to about 5% P@1 tab. 5. The authors blame tokenisation mismatch between pre-training and fine-tuning §4.3; this is their explanation and was not tested in the paper.
• Language-specific oddities. Odia (or) is seen in speech but unseen in text, and Urdu the other way round §4.3. Urdu is hard to recognise from speech because it sounds like Hindi and is often written in Devanagari §4.1.1. Check per-language rows before you draw conclusions about a region.
• The data host uses a different name. The data is distributed as xtreme_s on TFDS and Hugging Face [abstract, footnote]. Searching for "FLEURS" alone may miss it.
• Licence applies to the paper, not to the dataset host. The arXiv licence is CC BY 4.0 (recorded at fetch). The Hugging Face dataset card also lists cc-by-4.0 at the time of checking (2026-10-18). Confirm the dataset licence on the host before you redistribute audio.

Lineage

• Builds on: FLoRes-101, the source of the sentences and translations (arXiv:2106.03193) §2.1.
• Builds on: w2v-BERT, the speech-only pre-training method for the first baseline (arXiv:2108.06209) §3.1.
• Builds on: mSLAM, the joint speech-and-text pre-training for the second baseline. No arXiv id is given in the source bibliography §3.1.
• Related: XLS-R, the multilingual wav2vec 2.0 model that the introduction cites for few-shot gains across many languages (arXiv:2111.09296) §1.
• Compared against, as datasets (Table 1, tab. 1): BABEL, CommonVoice, CMU Wilderness, MLS, CoVoST-2, VoxLingua-107, Europarl-ST, MuST-C, mTEDx, VoxPopuli and CVSS tab. 1.
• Superseded by: not known from wiki content.