data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

In plain words

Self-supervised learning usually needs a different recipe for each kind of data: speech models learn a set of sound units, image models predict pixel values or patch codes, and language models predict words or word pieces. This paper asks whether one recipe can work for all three.

The idea is a teacher and a student that share the same network. The teacher reads the whole input and records what the network computes at each step. The student reads a copy with parts hidden and has to predict the teacher's record at the hidden parts. The teacher is a slowly moving average of the student, so it stays stable while the student learns. Because the record comes from the whole input, the target at each step carries context from everywhere in the input, not just from the local piece.

The authors report that the same method reaches top results on image classification, competitive speech recognition in the low-resource setups, and beats a RoBERTa baseline on the GLUE language benchmark. The input encoders and the masking still differ between the three data types.

TL;DR

data2vec trains one standard Transformer in two modes. An EMA teacher encodes the unmasked input and its top K blocks are normalised and averaged into a target per time-step or patch. A student encodes a masked input and regresses those targets at the masked positions with an L2 or smooth L1 loss §3.3 §3.4. Image, speech and text use the same objective, with modality-specific input encoders and masks §3.1 §3.2. Reported results are author-reported: 84.2 and 86.6 top-1 on ImageNet-1K for single ViT-B and ViT-L tab. 1; lower WER than wav2vec 2.0 and HuBERT in every Base setup and in the Large 10 min to 10 h setups, and close to them for Large at 100 h and 960 h tab. 2; GLUE 82.7 against a RoBERTa baseline at 82.5 tab. 4. Read the Pitfalls section before implementing: the paper and the released code differ in several hyperparameters.

Problem

• Masked prediction methods predict local targets: discrete tokens, pixels or units. Those targets are fixed per data type and carry little context §1 §2.
• Speech has no natural vocabulary of units, so prior systems learn one, for example by quantising (wav2vec 2.0) or by clustering (HuBERT) §2.
• Images have no word-like vocabulary either. Image methods predict visual tokens, raw pixels or engineered features §2.
• Language models predict words, sub-words, characters or bytes, and a single learned embedding must fit every context in which a unit occurs §2 §6.
• Self-distillation methods such as BYOL and DINO regress a momentum encoder, but they regress only the top layer and target whole-image invariance, not masked prediction §2.
• The authors' question is whether a single self-supervised learning method can work across speech, vision and language while keeping modality-specific input encoders §1 §7.

Method

Overview. The model is trained by predicting the teacher's representation of the full input from a partial view of the same input. The student encodes a masked sample; the teacher, which has the same architecture and is an exponentially moving average (EMA) of the student, encodes the unmasked sample §3 fig. 1. The student's output goes through a projection before the loss; the speech code uses one linear layer data2vec_audio.py, and the text config sets head_layers: 2 base.yaml. The paper does not describe this head. Teacher and student share the modality feature encoder and positional encoder §3.1 §3.3.

Architecture §3.1 §4. One standard Transformer for all modalities. Vision: a 224x224 image as 16x16 patches, each linearly projected, giving 196 tokens §4.1. Speech: a multi-layer 1-D convolution maps 16 kHz waveform to 50 Hz features §4.2. Text: BPE sub-word embeddings §4.3.

Masking §3.2. Masked positions get a learned MASK embedding.

• Vision: block-wise masking as in BEiT, blocks of at least 16 patches with a random aspect ratio §4.1.
• Speech: spans of latent speech representations, as in wav2vec 2.0 §3.2 §4.2.
• Text: the BERT scheme, 15 percent of tokens chosen uniformly, of which 80 percent get the mask token, 10 percent stay unchanged and 10 percent get a random token; no next-sentence task §4.3. The GLUE variant uses spans of four BPE tokens with masking probability 0.35 and no unmasked or random tokens §5.3 tab. 4.

Teacher targets §3.3 §3.4.

1. Run the EMA teacher on the unmasked input (same modality-specific encoder).
2. Take the output of each of the top K Transformer blocks. The paper uses the FFN output before the last residual connection (footnote on §3.3).
3. Normalise each block, then average the K blocks per time-step or patch. Speech uses instance normalisation over time per channel without learned parameters; vision and text use parameter-free layer normalisation §3.3.
4. Keep only the masked time-steps as targets §3.3.

Student objective §3.4 eq. 1. The student output at each masked step is regressed onto the target with a smooth L1 loss. The loss is

\mathcal{L}(y_t, f_t(x)) = \tfrac{1}{2}(y_t - f_t(x))^2/\beta \ \text{if} \ |y_t - f_t(x)| \le \beta, \ \text{otherwise} \ |y_t - f_t(x)| - \tfrac{1}{2}\beta eq. 1

with \beta the transition between squared and L1 loss. The paper notes that the loss is less sensitive to outliers and that \beta must be tuned §3.4. Speech uses a plain L2 loss in the paper §4.2 tab. 7.

Teacher update §3.3 §4.2 §4.3. The teacher parameters are an EMA of the student, with decay tau that rises linearly from tau_0 to tau_e over the first tau_n updates and is then held constant. This makes the teacher move fast early on, when the student is random, and slowly later.

# data2vec training step  [§3] [§3.2] [§3.3] [§3.4]
# Input:  sample (image patches, audio frames or text tokens)
# Output: scalar loss over the masked steps

def training_step(sample, student, teacher, K):
    # Teacher sees the unmasked sample and keeps every block output  [§3.3]
    with no_grad():
        block_outputs = teacher.encode_all_blocks(sample)  # L x (T, d)
        top_k = block_outputs[-K:]                         # last K blocks
        # Normalise each block; modality-specific choice  [§3.3]
        top_k = [normalise(block) for block in top_k]
        # Average the K blocks into one target per step  [§3.3]
        targets = sum(top_k) / K                           # (T, d)

    # Student sees a masked copy of the sample  [§3.2]
    masked_sample = apply_mask(sample)
    predictions = student.regress(student.encode(masked_sample))  # (T, d)

    # Regress the masked steps only  [§3.3]
    masked = mask_positions(sample)
    return smooth_l1(predictions[masked], targets[masked])  # [eq. 1]

# EMA teacher update after each optimiser step  [§3.3]
# tau rises linearly from tau_0 to tau_e over tau_n updates  [§4.2] [§4.3]

def update_teacher(teacher, student, tau):
    # Teacher <- tau * Teacher + (1 - tau) * Student  [§3.3]
    for teacher_param, student_param in zip(teacher, student):
        teacher_param = tau * teacher_param + (1 - tau) * student_param

Vision §4.1. Teacher and student receive the same augmented image (random resized crops, horizontal flip, colour jitter) §4.1. The classifier for fine-tuning mean-pools the last block and feeds a softmax classifier §4.1.

Speech §4.2. Feature encoder: seven 1-D convolutions with 512 channels, strides (5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2), giving 50 Hz and a receptive field of 400 samples (25 ms). Input is normalised to zero mean and unit variance §4.2. The EMA covers the Transformer layers; the positional encoder is shared with the student §3.3 data2vec_audio.py.

Text §4.3. The student and teacher share the embedding tables and the position encoder in the EMA setup data2vec_text.py. The paper emphasises that targets are not a closed vocabulary: each target is a continuous representation that depends on the whole sequence §5.3 §6.

Hyperparameters

Values are from the paper unless the Notes column says otherwise. "code" means the official pretraining configs at code_commit. Only the pretraining configs of the three modalities were compared; the fine-tuning configs were not checked against the paper. The Large models have no pretraining config in this repository commit (only Base), so Large numbers below come from the paper.

Shared

Speech (Base, pretraining on LS-960) §4.2

Speech, AudioSet pretraining §5.2 app. A

Vision (ViT-B, ViT-L) §4.1

Text (BERT-style Base, 16 GPUs) §4.3

Results

Author-reported. Each table is the paper's; the setup is given above it.

Vision: ImageNet-1K top-1 validation accuracy, ViT-B and ViT-L tab. 1

Setup: pretrained on ImageNet-1K train images, fine-tuned on the same labels §5.1. ViT-B pretrained 800 epochs and ViT-L 1600 epochs tab. 1 caption.

The paper reports the best single-model numbers for ViT-B and ViT-L; BEiT and PeCo rely on extra visual tokenisers or distilled models tab. 1 §5.1.

Speech: word error rate, test-other, 4-gram LM tab. 2

Setup: pretraining on LS-960 (Base) or LL-60K (Large); fine-tuning on the Libri-light 10 min, 1 h and 10 h labelled setups and on the LibriSpeech 100 h and 960 h subsets tab. 2 caption. Abbreviations from the paper's macros: LS-960 = LibriSpeech 960 h, LL-60K = Libri-light 60K h [macro.tex].

Base-model dev/test detail, including 100 h rows with other LMs, is in appendix tab. 6 below.

Speech: dev and test WER, Base model, 10 min to 100 h tab. 6

Setup: dev and test WER, Libri-light 10 min, 1 h, 10 h and LibriSpeech 100 h labelled setups; the caption says the paper's models use LS-960 as unlabelled audio tab. 6 caption. Rows from other systems use the unlabelled data and LM named in their row.

Audio event classification: mAP on AudioSet eval tab. 3

Setup: fine-tuned on the 20K subset; data2vec pretrained on AudioSet only §5.2 tab. 3 caption.

Language: GLUE development set, single-task fine-tuning tab. 4

Setup: pretrained on BookCorpus and English Wikipedia for 1M updates, batch 256 sequences §5.3. Baseline is RoBERTa re-trained in the same setup tab. 4 caption. Average over five fine-tuning runs §4.3. Metrics as in the caption: MNLI accuracy on both dev sets; MRPC and QQP the mean of accuracy and F1; STS-B the mean of Pearson and Spearman; CoLA Matthews correlation; others accuracy.

Target features and ablations

• Layer averaging (K) fig. 2. Setup: Base models with 12 layers. Speech: 200K updates on LS-960, fine-tuned on 10 h, dev-other without LM. NLP: GLUE validation average. Vision: 300 epochs, ImageNet top-1 §5.4. Values read from the figure's source data:

  K	Speech WER	NLP GLUE avg	Vision top-1
  1	39.39	58.63	83.01
  2	36.04	73.08	83.50
  3	26.37	76.80	84.02
  4	26.55	78.03	83.88
  5	17.43	79.09	83.86
  6	13.73	80.07	83.79
  7	13.90	80.15	83.74
  8	13.43	81.42	83.70
  9	15.44	80.50	83.64
  10	13.74	82.13	83.74
  11	14.01	81.44	83.65
  12	14.08	81.63	83.66

  The authors read this as multi-layer targets beating K=1 (BYOL-style) in all three modalities; the gain is large for speech and NLP and small for vision fig. 2 §5.4.

• Target contextualisation fig. 3. The teacher sees only a fraction of the input. Speech WER falls from 78.6 at 0.3 percent of the context to 14.8 with full context. Vision top-1 rises from 77.0 at 1 percent to 83.8 with full context fig. 3 §5.4.

• Target feature type tab. 5. Setup: reduced Base speech setup, not comparable to the main results tab. 5 caption.

  Target layer	WER
  self-attention output	100.0
  FFN output	13.1
  FFN output + residual	14.8
  End of block	14.5

  The self-attention output gives an unusable model; the authors explain this by its position before the residual connection §5.4 tab. 5.

Appendix ablations

• Loss function, Librispeech dev-other, no LM tab. 7: L2 17.1; L1 17.2; smooth L1 with beta 0.08 17.2, 0.25 16.8, 0.5 16.8, 1 17.3. The authors say the choice has a small effect app. B.
• Speech masking, dev-other, no LM tab. 8: baseline (mask prob 0.65, length 10) 17.1; mask prob 0.8: 17.2; mask prob 0.5: 17.3; length 5: 22.1; length 15: 18.9. The paper says that tuning speech masking gave no significant improvement app. C.

Paper vs code

Code: pytorch/fairseq at code_commit (examples/data2vec/). The paper's link points to that subdirectory (abstract). The repository is the current fairseq master, which also contains a later variant (data2vec2.py, config/v2/); this card covers only the v1 files named below.

• Speech teacher target. The paper uses the FFN output before the last residual §3.3 footnote. Code matches: the teacher's layer_result is the FFN output before the residual add wav2vec2.py.
• Speech loss. Paper: plain L2 §4.2. Code: MSE summed over feature dims and over masked steps, times 1/sqrt(768) when loss_scale is null data2vec_audio.py. The scale is not stated in the paper.
• Speech mask probability. Text: p = 0.065 §4.2. Appendix baseline and config: 0.65 tab. 8 base_librispeech.yaml.
• Speech positional encoder. Paper: shared between teacher and student §3.3. Code: excluded from the EMA data2vec_audio.py. Matches.
• Speech normalisation. Paper: instance norm §3.3. Code: instance_norm_target_layer: true base_librispeech.yaml. Matches.
• Speech updates and K. Paper: 400K updates, K=8, tau_n=30,000 §4.2. Config: 400000, average_top_k_layers: 8, ema_anneal_end_step: 30000 base_librispeech.yaml. Matches.
• Speech LR schedule. Tri-stage 3/90/7 §4.2. Config phase_ratio: [0.03,0.9,0.07] base_librispeech.yaml. Matches.
• AudioSet. Paper: same setup as LibriSpeech with K=12, 200K updates §5.2. Config: K=12, 200K updates, but cosine schedule with 10,000 warm-up updates and ema_layers_only: false audioset.yaml. These differ from the LibriSpeech setup; the paper does not say which it used.
• Text tau_n. Paper: 100,000 §4.3. Config: ema_anneal_end_step: 300000 base.yaml.
• Text learning rate schedule. Paper: tri-stage 5/80/15 §4.3. Config: cosine with 10,000 warm-up updates base.yaml.
• Text masking. Paper main results: BERT masking, 15 percent, 80/10/10 §4.3. Config: spans of 4 with probability 0.35, no random or unmasked tokens base.yaml. The config matches the GLUE row "+ wav2vec 2.0 masking" tab. 4, not the main row.
• Text targets. Paper: average of top K=10 blocks, LN per block, beta 4 §4.3. Code: fc_results (FFN outputs) averaged over the top K, with per-block LN and smooth L1 beta 4 data2vec_text.py. Matches; the code adds loss_scale: 1 base.yaml.
• Vision K and beta. Paper: K=6, beta=2 for ViT-B and ViT-L §4.1. The d2v1 config matches base_imagenet_d2v1.yaml. base_imagenet.yaml sets neither, so it runs the defaults K=8 and beta=0 (L2) data2vec_vision.py.
• Vision target location. Paper footnote: FFN output before the last residual §3.3. Code default end_of_block_targets: True, i.e. the output after the residual data2vec_vision.py, which neither vision config overrides base_imagenet.yaml. Discrepancy.
• Vision masking ratio. Paper: 60 percent of patches §4.1. Code default: num_mask_patches 75 of 196 patches data2vec_vision.py, about 38 percent. Discrepancy.
• Vision EMA. Paper: constant tau = 0.9998 §4.1. d2v1 config: 0.999 to 0.9998 annealed over max_update base_imagenet_d2v1.yaml. Discrepancy.
• Vision drop path. Paper: 0.2 §4.1. Code: 0.1 default data2vec_vision.py; 0.25 in d2v1 base_imagenet_d2v1.yaml. Discrepancy.
• Vision learning rate. Paper: peak 0.002 for ViT-B §4.1. Code: lr 0.0005 in both configs base_imagenet.yaml base_imagenet_d2v1.yaml. Discrepancy.
• Vision batch. Paper: 2048 for ViT-B §4.1. d2v1 config: 128 per GPU, 16 GPUs = 2048 base_imagenet_d2v1.yaml. base_imagenet.yaml: 64 per GPU = 1024 base_imagenet.yaml.
• Vision layer normalisation. Paper: per-block parameter-free layer norm for vision §3.3. Code: layer_norm_target_layer off by default and off in base_imagenet.yaml; d2v1 uses layer_norm_targets on the average instead base_imagenet_d2v1.yaml. Discrepancy.
• Vision optimiser. Adam betas, eps and weight decay are not in the paper; they differ between the two configs (see Hyperparameters).
• Large models. Paper: ViT-L and Speech Large results §5.1 tab. 2. The repository holds only Base pretraining configs for all three modalities; the ViT-L teacher-reset schedule §4.1 has no config here.

No code was checked for the fine-tuning recipes; those are not compared above.

Pitfalls

• Speech masking probability is ambiguous in the paper. The text gives p = 0.065 and about 49 percent masked §4.2, but the appendix baseline and the config use 0.65 tab. 8 base_librispeech.yaml. The config matches the table. Confirm with the authors before using 0.065.
• The vision configs do not reproduce the paper's ViT-B recipe. Neither file matches all of the paper's values: target location, mask ratio, EMA, drop path, learning rate and layer normalisation differ from the paper, and base_imagenet.yaml also leaves K and beta at their defaults (see Paper vs code). Take the values from the paper and check each config key against it.
• Update counts and epochs. The d2v1 config comments "300*1251" for max_update base_imagenet_d2v1.yaml. That count assumes about 1251 updates per epoch, which matches a batch of about 1024, not the 2048 the paper uses. The epoch count of the run is therefore not clear from the config alone.
• Scale of the speech loss. The code multiplies the summed loss by 1/sqrt(768) when loss_scale is null data2vec_audio.py. Neither the paper nor the config states this, so an implementation that averages instead of summing, or that omits the scale, changes the effective learning rate and the optimiser's behaviour.
• Loss is summed over masked steps, not averaged. Both the speech and text code sum over masked positions data2vec_audio.py data2vec_text.py. The paper does not say this, so batch size and padding change the loss scale.
• Collapse guard in the code. After 5,000 updates the speech code raises an exception if target or prediction variance falls below 0.1 or 0.01 data2vec_audio.py. The paper's Discussion lists the causes of collapse (too high a learning rate, too short warm-up, tau too low, correlated targets) §6. Implement the guard and log target_var and pred_var.
• Target normalisation is part of the recipe. The paper says normalising targets prevents collapse and stops high-norm layers from dominating §3.3. Each modality has its own choice (instance norm for speech, parameter-free layer norm for vision and text), so using the wrong one changes the target scale. The paper has no published ablation for removing it.
• Speech targets are sensitive to the Transformer's residual. The self-attention output gives WER 100 (unusable) tab. 5; use the FFN output.
• Stochastic depth and layerdrop are config-only. Speech encoder_layerdrop 0.05, text head_layers 2, and vision drop path are in the configs, not the paper (see Hyperparameters).
• Large models are paper-only. No Large pretraining config is in the repository commit, for any modality.
• Results are author-reported without variance. GLUE numbers are averages over five fine-tuning runs §4.3; the ImageNet, speech and AudioSet tables give no variance or seeds.
• Ablations use a reduced setup. The target-feature table is a reduced setup and is not comparable with the main results tab. 5 caption.

Lineage

• Builds on: BYOL and DINO (regressing an EMA teacher, but on the top layer only and with global invariance targets) §2 §3.3; wav2vec 2.0 (span masking, feature encoder, tri-stage schedule) §3.2 §4.2; BEiT (block masking and the open-source vision implementation) §4.1; BERT and RoBERTa (NLP setup) §4.3; the fairseq toolkit §4.2 §4.3.
• Compared against: MAE, SimMIM, iBOT, MaskFeat, MoCo v3, DINO, BEiT, PeCo (vision) tab. 1; wav2vec 2.0, HuBERT, WavLM, IPL, SlimIPL, Noisy student (speech) tab. 2 tab. 6; SSAST, MaskSpec, MAE-AST (AudioSet) tab. 3; RoBERTa (NLP) tab. 4.
• Superseded by: not established from this wiki's content.