Self-Supervised Speech Representation Learning: A Review

In plain words

Speech recordings are easy to collect, but labelled transcripts are expensive, and many languages and domains have almost none. This review looks at a way around that. A model is first trained on a lot of unlabelled audio, by hiding or predicting parts of it, and then reused for speech tasks such as recognition or speaker identification with only a small amount of labelled data.

The authors sort the many methods into three families. The first family rebuilds hidden or future parts of the audio. The second teaches the model to tell the true next piece of audio apart from random pieces. The third predicts targets that come from a helper model. The review also covers the datasets and shared benchmarks used to compare these models, what the learned features contain, and work that learns speech without any text. It is a map of the field, not a new method. It also warns that numbers from different papers are hard to compare, because each paper uses its own fine-tuning recipe.

TL;DR

This is a survey: it proposes no method of its own. Its main categories are generative (reconstruction), contrastive and predictive approaches §3. The models it singles out as the field's go-to choices are wav2vec 2.0 and HuBERT §7. The authors report that these reach competitive word error rates on LibriSpeech with 1 hour or 10 minutes of labels §4.5. Results are author-reported and use different recipes, so they are not directly comparable §4.5. Many datasets the review lists are non-commercial, which matters for any reuse §4.3.

Problem

Supervised speech models need large labelled corpora, which restricts progress for languages and domains with little labelling investment §1. Pretext tasks that work for text and images do not transfer directly to speech, for three reasons the review gives. Speech is a long sequence without segment boundaries. It is continuous, so there is no fixed vocabulary of units. And different tasks need different information: recognition needs content and ignores the speaker, while speaker recognition needs the reverse §3, intro list. The review covers self-supervised speech only. It leaves out pseudo-labelling and general transfer learning §2.4.

Method

The framework has two stages. In the first, a representation model is pre-trained with a pretext task that uses only unlabelled data fig. 1. In the second, downstream tasks either use the frozen representation or fine-tune the whole model with labels fig. 1 §1. Hidden-layer features are often better for downstream use than the output layer §3.1. The notation (§3.1) is:

• f is the representation model kept after pre-training.
• g is an auxiliary head used only for the pretext task. It is discarded afterwards.
• m is a masking function, used only in pre-training.
• A representation is contextualised if each frame depends on a long part of the input, and localised if it depends only on a short window §3.1.

The review says the Transformer encoder is widely used across all three families §3.2.2.

Generative (reconstruction) §3.2

"Generative" here means that the pretext task reconstructs the input or a masked, future or corrupted view of it. It does not mean a model that samples new data §3.2.1.

• Autoencoders and VAEs learn from a bottleneck and reconstruction. VQ-VAE quantises each encoder output to its nearest codebook vector, copies gradients through the quantiser with a straight-through estimator, and adds a codebook term and a commitment term §3.2.2.
• Autoregressive predictive coding (APC) predicts the frame c steps ahead with an L1 loss. The review notes that next-frame prediction is trivial for speech, and that the original work found c = 3 works well §3.2.2.
• DeCoAR uses a forward LSTM over the past and a backward LSTM over the future, then reconstructs the frames between them, so it sees both sides §3.2.2.
• Masked reconstruction follows BERT. For speech, masking single frames is too easy because neighbouring frames are similar, so chunks of consecutive frames are masked instead §3.2.2. Some models also mask along frequency §3.2.2.
• Phonetic masking (pMPC) masks spans chosen by phonetic segmentation. That segmentation needs labels §3.2.2.
• PASE and PASE+ reconstruct several targets at once, including waveform, spectra and prosody §3.2.2.
• Challenge: a model that reconstructs everything also encodes information a given task does not need. Little is known about how the choice of masking and target affects what is learned §3.2.3.

Contrastive §3.3

Contrastive models pull an anchor towards its positive sample and away from negatives §3.3.1.

• CPC builds localised frames with a convolutional module, then context with a recurrent module. A separate linear projection per step k predicts the frame k steps ahead. The InfoNCE loss puts the true frame and N-1 sampled negatives in one softmax. The review reports K = 12 for phoneme classification §3.3.2.
• wav2vec replaces the softmax with independent binary classifications: the positive must score high and each negative low. The target index is excluded from the negatives §3.3.2.
• wav2vec 2.0 masks the convolutional features before a Transformer. It quantises the unmasked features into discrete targets with a Gumbel-softmax quantiser. It uses product quantisation over several codebooks and an extra term that encourages all codebook entries to be used. The loss is InfoNCE with cosine similarity, applied only at masked steps, with negatives taken from other masked steps §3.3.2.
• Challenge: how negatives are chosen also sets invariances. Taking negatives only from the same utterance pushes features towards speaker invariance §3.3.3.

Predictive §3.4

These models predict targets computed outside the training graph, usually by a separate model. Because there are no negatives, a trivial solution has to be avoided another way §3.4.1.

• Discrete BERT uses a frozen vq-wav2vec model to turn speech into discrete units. A BERT model then predicts masked units with cross-entropy §3.4.2.
• HuBERT takes its first targets from k-means clusters over MFCC features. After the first pass, it re-clusters the hidden features of the first model to get new targets. The loss is cross-entropy on masked and unmasked frames, weighted by a scalar β §3.4.2. The review says that two iterations are enough to match or beat the previous state of the art for low-resource recognition §3.4.2.
• WavLM is close to HuBERT and adds a gated relative position bias inside attention. It also mixes in speech from other utterances in the batch, and predicts only the targets of the original speaker §3.4.2.
• data2vec uses an exponential moving average (EMA) of its own parameters as the teacher. The targets are the average of the top k teacher layers over the unmasked input. The loss is a smoothed L1 with a transition parameter η §3.4.2.
• Challenge: iterative target building is awkward for large data, and the teacher's cost grows with model size §3.4.3.

Multimodal and acoustic word embeddings §3.5, §3.6

• Multimodal models learn from speech with a paired modality, such as images or lip movements, and are mainly evaluated on cross-modal retrieval §3.5.
• Acoustic word embeddings map a word segment to one vector. Simple pooling of frame-level self-supervised features has not yet worked well, according to the review §3.6.

Pseudocode: predictive masked prediction (HuBERT-style)

This is a generic template built from the description in §3.4.2 and §3.2.2. The review does not give a mask rate, the number of clusters, or β. This is not the official training recipe.

# Masked prediction against offline cluster targets  [§3.4.2]
# Input:  waveform (B, S); targets: cluster id per frame (B, T)
# Output: scalar loss

def masked_prediction_step(waveform, targets):
    # Encode the waveform into frame features  [§3.4.2]
    features = conv_encoder(waveform)              # (B, T, D)
    # Replace spans of consecutive frames by a learned vector  [§3.2.2]
    masked = mask_consecutive_spans(features)      # (B, T, D)
    hidden = transformer(masked)                   # (B, T, D)
    logits = output_head(hidden)                   # (B, T, K)
    # Cross-entropy on masked and on unmasked frames  [§3.4.2]
    loss_masked = cross_entropy(logits[is_masked], targets[is_masked])
    loss_unmasked = cross_entropy(logits[~is_masked], targets[~is_masked])
    # Weight the masked term by beta; the review does not state beta  [§3.4.2]
    return beta * loss_masked + (1 - beta) * loss_unmasked

Contrastive counterpart: the InfoNCE term for one step k in CPC, with the true future frame and sampled negatives in one softmax §3.3.2.

# InfoNCE for one prediction offset k  [§3.3.2]
# Input:  anchor (D,); positive (D,); negatives (N-1, D)
# Output: scalar loss

def info_nce(anchor, positive, negatives):
    # The true future frame sits in the candidate set with the negatives
    # [§3.3.2]
    candidates = concat(positive, negatives)       # (N, D)
    scores = candidates @ anchor                   # (N,)
    # Cross-entropy with the true frame at index 0  [§3.3.2]
    return -log_softmax(scores)[0]

Hyperparameters

Only values the review states are listed. Anything it does not state is marked "not stated". Table and equation numbers are not used; sections are cited.

Results

Author-reported. The review collects these numbers from the original papers. We did not reproduce them. Setup is noted in each row §4.5.

§4.5, state-of-the-art table

• LibriSpeech test-clean WER with 960, 100, 10 and 1 hour and 10 minutes of labels. Wav2vec 2.0-L and HuBERT-L/XL are competitive when all 960 hours are used. With 100 hours of labels, they match or beat IPL, slimIPL and noisy student. With 1 hour or 10 minutes they give lower or comparable WER §4.5, LibriSpeech figure.
• Wav2vec 2.0 is the first model the review credits with single-digit WER on LibriSpeech using only Libri-light subsets for fine-tuning §3.3.2.
• Unsupervised ASR, learning without paired text, reports 18.6% phone error on TIMIT and 5.9% WER on LibriSpeech for wav2vec 2.0 features with adversarial mapping and self-training §6.1, unsupervised ASR table.

Paper vs code

No official code. The review links no code repository. Its only links are project pages for speech GSLM demos §6.2 and dataset licence pages §4.3. The LaTeX source and arXiv metadata contain no repository link. This is a survey, so the search stopped at the paper: we did not search GitHub.

Pitfalls

• This is a survey. The pseudocode above is a generic template for the family, not a recipe from any one paper. An implementer needs the original paper of the model they want, not this review.
• Numbers from different papers are not comparable. The review says results differ in language models, prediction heads, and whether the encoder is frozen or fine-tuned §4.5. The state-of-the-art row mixes SSL with self-training §4.5.
• Licences of the listed datasets vary. Several are non-commercial or no-derivatives: TED-LIUM 2 and 3 (CC BY-NC-ND 3.0), Fluent Speech Commands (CC BY-NC-ND 4.0), Open Mandarin (partly CC BY-NC-ND 4.0), and TUT acoustic scenes (non-commercial). Some are LDC-licensed or not released. Check each one before reuse §4.3, dataset table. The review's licence column is its own statement and we did not verify it.
• Single-frame masking is too easy. Neighbouring speech frames are similar, so a model can interpolate them. Mask consecutive chunks instead §3.2.2.
• Negative sampling changes what is learned. Negatives from the same utterance bias features towards speaker invariance §3.3.3.
• Layer choice matters. For wav2vec 2.0, a phone classifier does best on frozen layers 10 to 21 of 24, and the last layers are worse §5.1. In a 95M-parameter WavLM, speaker information sits in lower layers and phonetic and semantic information in higher ones §5.1. For a 12-layer BERTphone model, language recognition draws most on layer 10 §5.1.
• More and more diverse data helps, but the size law is not measured. The review says the exact relation between data size and performance has not been quantified §5.3.
• Adversarial robustness is poorly understood. The review says it is unclear whether SSL models are more or less robust to attack than supervised ones §7.
• The review is from 2022. Newer models are not covered. The arXiv metadata shows version 3 updated 2022-10-27. The superseding work is not stated.

Lineage

• Surveys this review builds on: earlier reviews of self-supervised learning in general, NLP and vision, and a brief speech overview cited as borgholt_22 §1.
• Methods reviewed, without cards in this wiki yet: wav2vec 2.0, HuBERT, WavLM, data2vec, CPC, APC, DeCoAR, VQ-VAE, Discrete BERT and vq-wav2vec §3.
• Superseded by: not stated in the paper.