---
kind: concept
slug: self-supervised-learning
title: Self-supervised learning
summary: >-
  Pre-train an encoder on unlabelled data with a pretext task whose targets come from the data itself: rebuild hidden parts (generative), pull matching views together and push others apart (contrastive), or train a generator against a discriminator. Pick the family by data modality, batch and memory budget, and whether the features will be fine-tuned or probed linearly.
tags: [representation-learning, pretraining, self-supervised-learning]
papers: ["1907.11879", "1909.11825", "1910.12240", "2005.13249", "2006.08218", "2010.05113", "2010.14945", "2011.00362", "2011.02578", "2011.05499", "2011.09643", "2012.09311", "2101.01169", "2101.12037", "2102.06810", "2102.07810", "2103.00111", "2103.00113", "2103.12051", "2104.02057", "2104.07713", "2104.14294", "2106.01342", "2106.14112", "2107.14293", "2108.07258", "2110.09348", "2111.06377", "2111.07832", "2111.09296", "2111.09886", "2112.01511", "2112.05814", "2112.12750", "2201.00424", "2201.10005", "2202.03026", "2202.03555", "2203.06173", "2203.08414", "2203.12602", "2204.07141", "2205.09904", "2205.10643", "2206.09113", "2207.08051", "2208.11296", "2209.07522", "2210.06551", "2212.03191", "2212.03860", "2303.16727", "2304.07213", "2306.03881", "2401.06209", "2506.20380", "2609.24410"]
related: [contrastive-learning, image-manipulation-detection, discrete-speech-units, multimodal-language-model, benchmark-validity, masked-image-modeling, masked-language-modeling, vision-transformer, language-model-pretraining, graph-neural-network, data-augmentation, metric-learning, generative-adversarial-network, linear-probing, knowledge-distillation, test-time-training, semantic-segmentation, policy-gradient, time-series-classification, transfer-learning, foundation-model, part-based-features, k-means-clustering, domain-adaptation, one-class-classification, anomaly-detection, node-embedding, attributed-network-embedding, imitation-learning, learning-from-demonstration, nearest-neighbor-search, time-series-forecasting, self-attention, neural-style-transfer, image-to-image-translation, semi-supervised-learning, consistency-regularization, self-training, quantization-aware-training, vision-language-pretraining, image-copy-detection, diffusion-models, tabular-foundation-model, gradient-descent-stability, dense-correspondence, out-of-distribution-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

Self-supervised learning (SSL) gets its training signal from the data itself. Part of the input is hidden or altered, and the network is trained to recover it or to recognise it. The learned encoder is then reused for a real task, usually with few or no labels [arxiv:2006.08218 §1]. The pretext task is only a means: what matters is the representation it leaves behind.

The term is also used for a neighbouring setting. A 2022 survey separates semi-supervised learning, which trains on a few labels and many unlabelled images in one stage, from unsupervised pre-training, which learns from unlabelled images alone and is then fine-tuned [arxiv:2208.11296 §2.1] [arxiv:2208.11296 §3.1]. Its taxonomy groups the methods by the smoothness assumption they depend on: agreement across views, agreement between neighbours, or targets from class boundaries the model has already found [arxiv:2208.11296 §4.1]. This page covers the pre-training side, and points to the [semi-supervised learning](/concepts/semi-supervised-learning) page for the other.

Surveys sort SSL methods by objective. One places them in three families: **generative** (encoder plus decoder, reconstruction or likelihood loss), **contrastive** (encoder plus a small discriminator, similarity loss such as InfoNCE) and **generative-contrastive**, or adversarial (generator plus discriminator) [arxiv:2006.08218 §2]. A second survey treats every contrastive method as one framework: a similarity distribution that supplies positive and negative keys, an encoder, an optional transform head and a loss [arxiv:2010.05113 §III.A]. A 2020 survey of contrastive methods for vision and NLP lists four sources of negatives: the current batch, a memory bank, a momentum-encoder queue and cluster assignments [arxiv:2011.00362 §3]. For graphs, one encoder-decoder formulation covers the surveyed methods: the encoder is trained on a pretext loss and a downstream decoder is then trained on labels [arxiv:2103.00111 §3.1] [arxiv:2103.00111 eq. 1–2].

The same objectives now appear outside images and graphs. Contrastive pre-training uses patient-level positives for ECG [arxiv:2005.13249 §4.2], paired web text and code for text embeddings [arxiv:2201.10005 §2.2], and a masked-image encoder for robot control [arxiv:2203.06173 §2.1]. Temporal prediction with contrast is used for unlabelled sensor and EEG series [arxiv:2106.14112 §3], masked contrast for multilingual speech [arxiv:2111.09296 §2], and one teacher-student regression recipe is tested on speech, images and text at once [arxiv:2202.03555 §1]. Video and satellite data extend the same objectives: a masked video autoencoder with tube masking [arxiv:2203.12602 §3.3], a billion-parameter masked video autoencoder [arxiv:2303.16727 §4.1], masked autoencoders over satellite time series and spectral bands [arxiv:2207.08051 §4.1], and a Barlow Twins model for per-pixel satellite embeddings [arxiv:2506.20380 §3.3]. Tabular rows are another case: SAINT applies a contrastive loss and a denoising loss to two views of each row [arxiv:2106.01342 §4]. Long biosignals and human motion follow the same pattern: masked-span contrast over EEG windows [arxiv:2101.12037 §2.3] [arxiv:2101.12037 eq. 2], and masked 2D-to-3D lifting over skeleton sequences [arxiv:2210.06551 §3.3].

Speech has its own review. It sorts self-supervised speech models into generative methods that rebuild masked or future audio, contrastive methods that pick the true next segment out of distractors, and predictive methods that regress targets from a helper model. It names wav2vec 2.0 and HuBERT as the usual choices [arxiv:2205.10643 §3] [arxiv:2205.10643 §7]. The review proposes no method of its own, and warns that numbers from different papers are hard to compare because each uses its own fine-tuning recipe [arxiv:2205.10643 §4.5]. Pretext tasks that work for text do not carry straight over to speech, because the signal is long, continuous and has no fixed unit inventory [arxiv:2205.10643 §3]. The discrete-unit side is covered on the [discrete speech units](/concepts/discrete-speech-units) page.

The foundation-model report describes the same pattern at scale: pretrain on a surrogate task, then adapt to the downstream task. It names compute, task data and gradient access as the factors to weigh when choosing an adaptation method [arxiv:2108.07258 §1] [arxiv:2108.07258 §4.3]. It has no experiments of its own.

For vision transformers, a 2021 survey lists MoCo v3 and DINO as the self-supervised recipes applied to ViTs [arxiv:2101.01169 §3.2.4]. It also reports that a ViT-L trained on ImageNet alone is weaker by about 13% top-1 than one pre-trained on JFT-300M [arxiv:2101.01169 §2.2]. The contrastive ViT study that follows finds the training unstable: the losses often do not diverge but can cost accuracy without a visible failure [arxiv:2104.02057 §1].

The pretext choice decides what the representation ignores. If two crops and colour changes of one image count as a match, the model learns to ignore crop position and colour [arxiv:2010.05113 §VI.A]. Pretext tasks are chosen heuristically: the authors of the first survey list automatic pretext design as an open problem [arxiv:2006.08218 §7]. A lower pre-training loss does not mean better features. In SimMIM, heavier heads reach a lower loss yet transfer worse, so checkpoints should be selected by downstream fine-tuning [arxiv:2111.09886 §4.1.3].

The pre-training stage can be combined with the final task in three ways. The graph survey calls them pre-train then fine-tune, joint learning with a trade-off weight α, and unsupervised representation learning with a frozen encoder [arxiv:2103.00111 §3.3] [arxiv:2103.00111 eq. 6–8].

The same pretext losses also appear in two other roles. They can be added to the supervised loss as an auxiliary term and trained jointly [arxiv:2011.09643 §4.3], or run on each test input before that input is predicted [arxiv:2209.07522 §3].

## Variants

### Generative: predict the hidden part

**Auto-regressive.** The likelihood factorises into left-to-right conditionals, as in GPT-style models [arxiv:2006.08218 §3.1] [arxiv:2006.08218 eq. 1]. Each position sees one direction only, which is the stated weakness [arxiv:2006.08218 §3.1].

**Masked prediction.** BERT-style masked language modelling is treated as a denoising auto-encoder. Its stated weakness is that masked tokens are assumed independent given the unmasked ones [arxiv:2006.08218 §3.3.3]. The vision counterpart is masked image modelling. SimMIM uses 32×32 masked patches at ratio 0.6, a learnable mask token, a single linear layer as the head and an ℓ1 loss on masked pixels only [arxiv:2111.09886 §3.2–3.4] [arxiv:2111.09886 eq. 1]. Masks of 4 or 8 pixels score lower than 32 at the same ratio [arxiv:2111.09886 tab. 1]. Computing the loss on visible pixels too drops fine-tuning from 82.8 to 81.7 [arxiv:2111.09886 tab. 4]. The method is described in [SimMIM](/papers/2111.09886), and the vision-side variants are on the [masked image modelling](/concepts/masked-image-modeling) page.

**Masked prediction with a learned teacher.** iBOT keeps the masked-patch objective but replaces pixel targets with the output of a slowly updated copy of the network. The student sees masked views, the teacher sees the unmasked ones, and a cross-entropy between the teacher's centred and sharpened patch distribution and the student's distribution is summed over masked positions [arxiv:2111.07832 §3.1] [arxiv:2111.07832 eq. 3]. The targets are learned during training, so no separate tokenizer is pre-trained [arxiv:2111.07832 §1]. In the paper's reduced ablation, the full method reaches 74.2 linear and 81.5 fine-tuned accuracy, while a frozen DINO teacher in the same slot reaches 60.0 linear and 81.7 fine-tuned [arxiv:2111.07832 tab. 9]. The cost is higher per step, since a second network runs on the unmasked views [arxiv:2111.07832 §3.1].

**Masked autoencoder at scale (MAE).** The encoder is a ViT that sees only the visible patches. A random 75% of patches is hidden, and a small decoder with one shared mask token per gap reconstructs the hidden pixels, with the loss on masked patches only [arxiv:2111.06377 §3] [arxiv:2111.06377 tab. 1 caption]. Keeping mask tokens out of the encoder matters: with them, linear probing falls from 73.5 to 59.6 and the encoder costs 3.3× the FLOPs in the paper's ablation [arxiv:2111.06377 tab. 1c]. Fine-tuned on ImageNet-1K data only, ViT-H reaches 86.9 at 224 px and 87.8 at 448 px [arxiv:2111.06377 tab. 3]. Its linear probes are weaker: 68.0, 75.8 and 76.6 for ViT-B, L and H, against 77.6 for MoCo v3 with ViT-L [arxiv:2111.06377 app. B]. The paper argues that linear probing is a poor proxy for transfer [arxiv:2111.06377 §4.3]. Pre-training is cheaper per epoch: the encoder runs on a quarter of the patches [arxiv:2111.06377 §4.1]. The card is [the MAE paper](/papers/2111.06377).

**Masked prediction with an encoder-side regressor (CAE-style).** The encoder sees only the visible patches. A cross-attention regressor works on representations, not pixels, and predicts the encoder's representations of the hidden patches. A separate decoder maps those predictions to discrete tokens [arxiv:2202.03026 §3.1] [arxiv:2202.03026 fig. 1]. The loss is a token cross-entropy plus twice a mean-squared error between predicted and target representations [arxiv:2202.03026 eq. 1] [arxiv:2202.03026 §5.2]. The targets come from a no-gradient copy of the encoder on the hidden patches [arxiv:2202.03026 §3.2]. Half the patches are hidden, in random blocks [arxiv:2202.03026 §3.2]. In the 300-epoch ablation, adding the alignment term to the decoder model raises linear probing from 63.1 to 64.1 [arxiv:2202.03026 tab. 5]. Only the encoder is kept. Author-reported ViT-B results at 1600 epochs are 83.9 fine-tuned, 71.4 linear and 77.4 attentive, against 83.6, 67.8 and 74.2 for MAE [arxiv:2202.03026 tab. 1]. The [CAE card](/papers/2202.03026) notes that the reported rows use the d-VAE tokenizer, while the released code defaults to DALL-E.

**Masked autoencoder for video with tube masking (VideoMAE-style).** A 16-frame clip is cut into 2×16×16 cubes, and 90% of them are hidden. The same spatial cells are hidden at every time step, so a masked patch cannot be copied from a neighbouring frame [arxiv:2203.12602 §1] [arxiv:2203.12602 §3.3]. A plain ViT-B encodes only the visible cubes, and a 4-block decoder of width 384 reconstructs the hidden ones with an MSE loss on normalised cube pixels [arxiv:2203.12602 §4.2] [arxiv:2203.12602 app. Architectures]. On SSv2, tube masking at 90% scores 69.6 top-1, against 68.3 for independent random cube masking and 61.5 for hiding whole frames [arxiv:2203.12602 tab. 1]. Pre-training on each target's own unlabelled split, with no labels and no image weights, gives 91.3% on UCF101 and 62.6% on HMDB51. MoCo v3 reaches 81.7 and 39.2 on the same splits, and training from scratch reaches 51.4 and 18.0 [arxiv:2203.12602 tab. 2]. Pre-training on SSv2 takes 19.5 hours on 64 V100 GPUs, against 61.7 for MoCo v3 [arxiv:2203.12602 tab. 3]. The numbers are author-reported. The [VideoMAE card](/papers/2203.12602) records that the released pre-training scripts use a batch size that differs from the paper's table.

**Masked autoencoder for video with dual masking (VideoMAE V2-style).** The encoder keeps the tube masking of VideoMAE, with 90% of cubes hidden. The decoder receives a second mask that keeps about half the cubes, and the loss covers only cubes that the encoder hid and the decoder kept [arxiv:2303.16727 §3.2] [arxiv:2303.16727 tab. 10] [arxiv:2303.16727 eq. 2]. For ViT-B on Something-Something V2 at 800 epochs, dual masking reaches 70.15 top-1 against 70.28 for encoder-only masking, with FLOPs cut from 35.48G to 25.87G [arxiv:2303.16727 tab. 1]. For ViT-g on the unlabelled hybrid set, memory falls from 1753M to 1050M, a 1.48× speed-up [arxiv:2303.16727 tab. 2]. The scaled model has 1,011M parameters [arxiv:2303.16727 §4.1]. Training runs in three stages: masked pre-training on 1.35M unlabelled clips, supervised post-pre-training on the labelled Kinetics mix K710, then fine-tuning on each target [arxiv:2303.16727 §3.3]. Post-pre-training helps on Kinetics-400 but is worse on Something-Something V2 [arxiv:2303.16727 §4.2]. Author-reported: 90.0% top-1 on Kinetics-400 with ViT-g [arxiv:2303.16727 tab. 6]. The [VideoMAE V2 card](/papers/2303.16727) notes that the released scripts use a different schedule from the paper's table.

**Masked video encoder inside a fused model (InternVideo).** One branch trains a ViT-H to reconstruct video with 90% tube masking, for 1200 epochs at base learning rate 2.5e-4 in the paper's table [arxiv:2212.03191 §3.1] [arxiv:2212.03191 §4.2]. The second branch is a CLIP-initialised video-text encoder, trained with a contrastive loss and a caption decoder [arxiv:2212.03191 §3.1]. A cross-model attention stage then combines the two frozen encoders through a learned linear score combination [arxiv:2212.03191 §3.3]. The paper reports 91.1% top-1 on Kinetics-400 [arxiv:2212.03191 tab. 2] and 77.2% on Something-Something V2 [arxiv:2212.03191 tab. 3], both author-reported. The table does not say which variant gives the Something-Something figure. The [InternVideo card](/papers/2212.03191) records that the released pre-training script uses a different schedule and that the fusion stage is not in the released code.

**Masked autoencoder for satellite time series and bands (SatMAE-style).** Each image of a time series is patchified with one shared embedding. Each token gets a positional code and a temporal code built from the year, month and hour of its image [arxiv:2207.08051 §4.1] [arxiv:2207.08051 §4.1.1] [arxiv:2207.08051 eq. 2]. Masking is drawn per image. In consistent masking, by contrast, the same locations are hidden in every frame, as in VideoMAE [arxiv:2207.08051 §4.1.2]. For multi-band input, the bands are split into groups, each with its own patch embedding and a group code, so G groups give G·L tokens [arxiv:2207.08051 §4.2] [arxiv:2207.08051 eq. 3]. The paper uses three groups over the ten Sentinel-2 bands it keeps [arxiv:2207.08051 §5.4]. With a ViT-L backbone, the temporal model reaches 81.49% top-1 on fMoW temporal, against 78.90% for an MAE that receives three copies of the image [arxiv:2207.08051 §5.3]. That figure uses test-time augmentation over nine sequences [arxiv:2207.08051 app. A.3.2]. On the fMoW-Sentinel validation set, the grouped model with independent masking reaches 59.30% top-1 without extra augmentation [arxiv:2207.08051 §5.4]. On transfer tasks the method is not always best: SpaceNet v1 mIoU is 78.07 against 78.51 for GASSL [arxiv:2207.08051 §5.5]. All numbers are author-reported. The [SatMAE card](/papers/2207.08051) lists where the released code differs from the paper.

**Masked prediction in feature space (MSN).** No pixels are reconstructed. A randomly masked view is trained to match the prototype assignments that an EMA teacher gives the unmasked view, with cross-entropy and an entropy term that spreads use across prototypes [arxiv:2204.07141 §3] [arxiv:2204.07141 eq. 1]. The default recipe pairs one masked global view with ten small unmasked focal views [arxiv:2204.07141 §5]. In its low-label tables, ViT-B/4 reaches 72.4% top-1 with 5 labels per class and 75.7% with 1% of labels [arxiv:2204.07141 tab. 1] [arxiv:2204.07141 tab. 2]. These are the authors' numbers on ImageNet-1K.

**Regression of averaged teacher layers (data2vec-style).** One recipe covers speech, images and text. An EMA teacher encodes the unmasked input. For each modality, the outputs of its top K blocks are normalised and averaged, and the student regresses that average at the masked steps [arxiv:2202.03555 §3.3] [arxiv:2202.03555 §3.4]. Speech uses K = 8 with an L2 loss, vision K = 6 with a smooth L1 loss, and text K = 10 [arxiv:2202.03555 §4.1] [arxiv:2202.03555 §4.2] [arxiv:2202.03555 §4.3]. Using several blocks rather than the last one is reported to matter most for speech and text [arxiv:2202.03555 §5.4] [arxiv:2202.03555 fig. 2]. Reported single-model ImageNet-1K top-1 is 84.2 for ViT-B and 86.6 for ViT-L [arxiv:2202.03555 tab. 1]. On Libri-light with 10 min of labels, the Base model reaches 12.3 test-other WER against 15.6 for wav2vec 2.0 Base, both with a 4-gram LM [arxiv:2202.03555 tab. 2]. On GLUE dev it averages 82.7 against 82.5 for the RoBERTa baseline the authors retrained [arxiv:2202.03555 tab. 4]. The speech masking probability is inconsistent in the paper: the text gives 0.065 and the appendix and config use 0.65 [arxiv:2202.03555 §4.2] [arxiv:2202.03555 tab. 8]. The [data2vec card](/papers/2202.03555) lists the other config mismatches.

**Forecasting as the pretext (sparse clinical series).** STraTS represents each observation as a (time, variable, value) triplet. Its pretext task predicts, from the pooled representation, which variables appear in a later two-hour window, with squared error only on observed entries [arxiv:2107.14293 §3.2.6] [arxiv:2107.14293 eq. 9–10]. The authors chose forecasting over masking as the pretext [arxiv:2107.14293 §3.2.6]. The model is pre-trained on this task and then fine-tuned for in-hospital mortality [arxiv:2107.14293 §3.2.6]. Its table gives PR-AUC 0.577 with self-supervision against 0.546 without on MIMIC-III, and 0.498 against 0.467 on PhysioNet-2012 [arxiv:2107.14293 tab. 5]. The [STraTS card](/papers/2107.14293) records that the released code changes several settings, including the hidden size and the pre-training patience.

**Flow and VAE.** Flows learn an invertible map with a known prior and a tractable Jacobian [arxiv:2006.08218 §3.2] [arxiv:2006.08218 eq. 3]. VAEs maximise the ELBO [arxiv:2006.08218 eq. 5]. VQ-VAE replaces the continuous latent with a codebook lookup [arxiv:2006.08218 eq. 6].

**Pros and cons.** Generative methods need no task-specific assumptions. They are reported as less competitive than contrastive methods on some computer-vision classification benchmarks, and their point-wise likelihood objective is sensitive to rare samples [arxiv:2006.08218 §3.5].

### Contrastive: match views, reject others

The standard loss is InfoNCE, a softmax over one positive and K negatives [arxiv:2006.08218 eq. 9] [arxiv:2010.05113 eq. 1]. Positives can come from augmentations of one input, from several sensors, from neighbouring frames or sentences, or from patches of one image [arxiv:2010.05113 §III.B]. Many contrastive methods do poorly without crop and colour distortion, even with the same encoder and loss [arxiv:2010.05113 §II.C]. The 2020 contrastive survey's ImageNet table, whose numbers are copied from the original papers and not re-run, puts SwAV at 75.3 top-1 with a frozen encoder, just under supervised ResNet-50 at 76.5 [arxiv:2011.00362 tab. 1].

**In-batch negatives (SimCLR-style).** Both encoders receive gradients. From N images, 2N views are made and each positive pair has 2(N−1) negatives, scored with cosine similarity [arxiv:2006.08218 §4.2.2] [arxiv:2006.08218 eq. 13–14]. The authors credit a learnable projection head, longer training and deeper networks [arxiv:2006.08218 §4.2.2]. All activations live on the GPU, so the batch size caps the number of negatives [arxiv:2010.05113 §III.C.1].

**Memory bank.** Each sample's detached representation is stored and read back as a key. Entries are refreshed once per epoch, so keys can be up to an epoch old [arxiv:2010.05113 §III.C.2]. This removes the batch-size limit at the price of stale keys.

**Momentum encoder with a queue (MoCo-style).** The key encoder is an exponentially weighted average of the query encoder, and the newest keys replace the oldest in a queue. The original queue holds up to 65536 keys [arxiv:2006.08218 §4.2.2] [arxiv:2006.08218 eq. 12]. The momentum coefficient is not stated in either survey [arxiv:2006.08218 Hyperparameters] [arxiv:2010.05113 §III.C.2]. MoCo v3 keeps the momentum key encoder and drops the queue, because the queue's gain shrinks at large batch sizes [arxiv:2104.02057 §3]. In its ViT-B ablation a momentum of 0.99 scores best, at 76.5% linear accuracy, against 74.3% with m = 0 and 75.0% with 0.999 [arxiv:2104.02057 §6.2].

**Contrastive ViT pre-training (MoCo v3-style).** The momentum-encoder contrast with a prediction head on the query side is applied to ViTs. The authors report that training is unstable. The losses often do not diverge, but they can cost 1 to 3% accuracy, and sparse kNN checks can miss the dips [arxiv:2104.02057 §1] [arxiv:2104.02057 §4.1]. The one change aimed at stability is to freeze the first patch projection at its random starting values, with a stop-gradient after it [arxiv:2104.02057 §4.2]. In the 100-epoch ablation at base learning rate 1.5e-4, the random projection reaches 73.4% linear accuracy and the learned one 71.7% [arxiv:2104.02057 fig. 5]. With the random projection, training can still become unstable at a high learning rate [arxiv:2104.02057 §4.2]. Author-reported ImageNet-1K results are 76.7% linear and 83.2% fine-tuned for ViT-B/16, and 77.6% linear and 84.1% fine-tuned for ViT-L [arxiv:2104.02057 tab. 1]. The [MoCo v3 card](/papers/2104.02057) lists where the released ViT commands use a different learning rate, weight decay and momentum schedule from the paper's text.

**Negative-free (BYOL, SimSiam).** An online network with a predictor regresses an EMA target network [arxiv:2006.08218 §4.2.2] [arxiv:2006.08218 eq. 15]. The survey reports that SimSiam identifies the stop-gradient as the critical part [arxiv:2006.08218 §4.2.2]. Without negatives, a constant encoder also minimises the positive-pair distance, so collapse must be prevented by architecture, such as the predictor, the EMA target, whitening or batch-wise codes [arxiv:2010.05113 §III.E] [arxiv:2010.05113 §V.B]. The DINO-style [CLS] loss in iBOT is also negative-free. It matches the teacher's centred, sharpened output across two views, and the teacher is a moving average of the student [arxiv:2111.07832 §2.2] [arxiv:2111.07832 eq. 2].

A 2021 analysis explains the mechanism for a linear model. The predictor, the stop-gradient, the EMA target and weight decay together keep the online weights growing alongside the predictor [arxiv:2102.06810 §2] [arxiv:2102.06810 §3]. The analysis assumes a bias-free two-layer linear network, isotropic data and augmentation noise, and a symmetric predictor. The non-symmetric case is left open [arxiv:2102.06810 §3] [arxiv:2102.06810 §3.1]. Its DirectPred variant sets the linear predictor from the eigendecomposition of a running correlation estimate, with no gradient steps for the predictor [arxiv:2102.06810 §4] [arxiv:2102.06810 eq. 18–19]. At 300 epochs on ImageNet it reaches 72.4 top-1 against 69.9 for a gradient-trained linear predictor, and 72.5 for BYOL's default two-layer predictor [arxiv:2102.06810 tab. 9].

**Self-distillation with centring and sharpening (DINO-style).** A student ViT sees several crops: two global views at 224² and several local views at 96². It is trained to match the softmax output of an EMA teacher that sees only the global views [arxiv:2104.14294 §3.1] [arxiv:2104.14294 eq. 3]. Collapse is prevented by centring the teacher output and sharpening it with a low temperature. There is no predictor, no negative pair and no batch normalisation in the ViT case [arxiv:2104.14294 §3.1]. Centring alone drives the output to uniform and sharpening alone drives it to one dimension, so both are needed [arxiv:2104.14294 §5.3]. The teacher must be an EMA of the student: copying the student, or using its previous iterate, does not converge [arxiv:2104.14294 §5.2]. In the ViT-S/16 ablation at 300 epochs, removing multi-crop lowers k-NN top-1 from 72.8 to 67.9 [arxiv:2104.14294 tab. 7]. The same multi-crop recipe lowers BYOL's scores in the authors' runs [arxiv:2104.14294 app. E]. Frozen features reach 77.0 linear and 74.5 k-NN with ViT-S/16, and 80.1 linear with ViT-B/8 [arxiv:2104.14294 tab. 2]. Those are author-reported ImageNet-1K numbers. The paper warms the teacher temperature from 0.04 to 0.07 over 30 epochs [arxiv:2104.14294 §3.2], while the released code defaults to a constant 0.04. The [DINO card](/papers/2104.14294) lists this and other differences.

**Cluster-based (SwAV, DeepCluster).** Views of one image must receive the same cluster assignment, which is a swapped prediction over prototypes and needs no instance negatives [arxiv:2006.08218 §4.2.1].

**Alignment and uniformity.** The contrastive loss splits into an alignment term that pulls positives together and a uniformity term that spreads embeddings out. Both are needed, and an over-weighted uniformity term causes collapse [arxiv:2006.08218 §6.3.2].

**Projector and dimensional collapse (DirectCLR).** Contrastive learning avoids complete collapse, but its embeddings can still span only a low-dimensional subspace, with several covariance singular values near zero [arxiv:2110.09348 §1] [arxiv:2110.09348 §3]. The paper ties this to strong augmentation in a linear model, and to implicit low-rank bias in deep linear networks when augmentation is weak [arxiv:2110.09348 §4.2] [arxiv:2110.09348 §5.3]. DirectCLR computes InfoNCE on the first d0 channels of the encoder output, with no projector. The text does not state d0 [arxiv:2110.09348 §6.2]. On ImageNet with ResNet-50 for 100 epochs, it reaches 62.7% linear-probe top-1, against 61.1% for SimCLR with a one-layer projector and 66.5% for SimCLR with a two-layer projector [arxiv:2110.09348 tab. 1]. The paper cannot explain why a nonlinear projector helps [arxiv:2110.09348 §6.2], and its collapse theory covers linear networks only [arxiv:2110.09348 §4.2]. The released code is non-commercial, so this page draws only on the paper.

**Cross-correlation redundancy reduction, no negatives (Barlow Twins-style).** Two views of one input are embedded, and the batch-normalised outputs are compared through their cross-correlation matrix. The loss pushes the diagonal towards 1 and the off-diagonal entries towards 0, with the off-diagonal weight set by a coefficient [arxiv:2506.20380 §3.3] [arxiv:2506.20380 alg. 1]. TESSERA applies this to per-pixel Sentinel-1 and Sentinel-2 time series. Each view samples 40 valid dates per sensor, the projector is 16,384 wide, and the coefficient is 5e-3 [arxiv:2506.20380 §3.2] [arxiv:2506.20380 §3.3]. A mixup term, which mixes one view with a shuffled one, is added to keep the cross-correlations close to linear behaviour [arxiv:2506.20380 §3.3]. In its ablation on Austrian Crop, removing mixup drops validation F1 from 77.3 to 66.2, and removing global shuffling drops it to 68.1 [arxiv:2506.20380 tab. 3]. Pre-training uses one epoch, since the paper calls the task compute-bound rather than data-bound [arxiv:2506.20380 §3.3]. The reported cost is about 6,200 GPU hours [arxiv:2506.20380 app. Pre-training Details].

**Domain-defined positives (CLOCS-style).** For ECG, positives are segments that share a patient: adjacent non-overlapping segments, different leads, or both. Because the loss covers every same-patient pair in a mini-batch, the positive set depends on which patients happen to repeat in the batch [arxiv:2005.13249 §4.2] [arxiv:2005.13249 §4.3]. The authors report linear AUC on Chapman with 50% of labels of 0.896 for the segment variant against 0.738 for SimCLR, at 4 leads and embedding size 128 [arxiv:2005.13249 tab. 1]. These are author-reported AUC points, not relative gains.

**Paired web and code data at very large batch.** For text, two neighbouring pieces of web text are a positive pair. For code, a function's docstring and its body are a pair. Every other pair in the batch is a negative, and the encoders start from GPT-family or Codex models [arxiv:2201.10005 §2.2] [arxiv:2201.10005 §3]. Batch size is the main lever. For a 300M text model, NQ MRR@10 is 71.4 at batch 1536 and 84.7 at 12288 [arxiv:2201.10005 tab. 9]. The paper does not state the data size or the optimiser [arxiv:2201.10005 §3].

**Temporal prediction plus contrast (TS-TCC-style).** Two augmented views of a series are encoded by a small 1-D convolutional encoder. A Transformer over one view's latent steps up to a cut point must predict the next K latent steps of the other view, scored against the other samples in the batch [arxiv:2106.14112 §3.2] [arxiv:2106.14112 eq. 1–2]. A second, SimCLR-style loss contrasts the Transformer contexts of the two views [arxiv:2106.14112 §3.3] [arxiv:2106.14112 eq. 5]. A linear layer on the frozen encoder reaches 90.37% accuracy on HAR, 83.00% on Sleep-EDF and 97.23% on Epilepsy, against 90.14%, 83.41% and 96.66% for supervised training [arxiv:2106.14112 tab. 2]. The ablation attributes more than 5 points on HAR to the cross-view prediction task [arxiv:2106.14112 §5.4] [arxiv:2106.14112 tab. 4]. All three are single-dataset results over five seeds. The [TS-TCC card](/papers/2106.14112) lists where the public code differs from the paper, including the scaling factor, the number of segments and the loss normalisation.

**Masked-span contrast on EEG (BENDR-style).** A convolutional encoder turns 60-second, 20-channel EEG windows at 256 Hz into 512-dimensional vectors at about 2.67 Hz [arxiv:2101.12037 §2.2] [arxiv:2101.12037 §2.3]. Spans of these vectors are masked, and an 8-layer Transformer reads the sequence. At each masked position the model must pick the true vector out of 20 distractors drawn from the same recording, with temperature 0.1 [arxiv:2101.12037 §2.4.1] [arxiv:2101.12037 eq. 2]. Pre-training uses TUEG, about 1.5 TB of EDF before preprocessing [arxiv:2101.12037 §2.1]. The paper does not state the pre-training learning rate, batch size or epoch count [arxiv:2101.12037 Hyperparameters]. The author-reported downstream results are balanced accuracy of 86.7 on motor imagery and 0.72 on sleep staging, and AUROC of 0.65 on error-related negativity and 0.72 on P300 [arxiv:2101.12037 tab. 2]. The paper gives no numbers for its comparison with a prior sleep-staging self-supervision method [arxiv:2101.12037 §3.2]. The [BENDR card](/papers/2101.12037) lists where the released code differs, including the pre-training batch size and the P300 window.

**Multilingual masked contrast (XLS-R).** wav2vec 2.0 pre-training is scaled to 436K hours of unlabelled speech in 128 languages, with 0.3B, 1B and 2B parameter models [arxiv:2111.09296 §3.1] [arxiv:2111.09296 tab. 2]. Spans of 10 latent steps are masked, and the model picks the true quantised latent among K = 100 distractors [arxiv:2111.09296 §2]. Languages and corpora are sampled with an exponent of 0.5, which boosts low-resource data [arxiv:2111.09296 §4.1]. Against the prior best on CoVoST-2 X-to-English, the 2B model averages 22.1 BLEU over 21 directions against 14.7 [arxiv:2111.09296 tab. 3]. On English-to-X over 15 directions, the 2B model averages 27.8 against 26.6 for an English-only 720M model [arxiv:2111.09296 tab. 4]. The gain is not uniform. On LibriSpeech with 10 min of labels, the 0.3B model is behind the English-only 0.3B model, 34.1 against 32.1 test-clean, while the 1B model is ahead at 29.1 [arxiv:2111.09296 tab. 11]. All results are author-reported. The [XLS-R card](/papers/2111.09296) notes that the sampling formula is not normalised in the paper.

**Pre-training, then noisy-student rounds on untranscribed audio (Jordanian Arabic).** A wav2vec 2.0-style encoder is pre-trained on unlabelled audio and fine-tuned on the transcribed part as the first teacher [arxiv:2609.24410 §7.2] [arxiv:2609.24410 §8.1]. Each later round writes pseudo-labels for batches of unlabelled audio and trains a larger student on those labels, the human-checked data and augmented copies. The student becomes the next teacher. Five rounds take the word error rate on the authors' evaluation set from 0.568 to 0.515 [arxiv:2609.24410 tab. 4] [arxiv:2609.24410 fig. 11]. The data is about 97,000 recordings, 52 h of them transcribed [arxiv:2609.24410 §6.4]. The paper gives one run, no optimiser, learning rate, batch size or augmentation settings [arxiv:2609.24410 Hyperparameters], and no official code was found.

**Distribution augmentation for one-class detection.** A detector trained on inlier images only can struggle when contrastive features spread evenly over the sphere, which leaves little room for an outlier to stand out [arxiv:2011.02578 §1] [arxiv:2011.02578 §3.1.3]. The fix adds the 90°, 180° and 270° rotations of each training image as separate distributions, which makes the training set four times larger [arxiv:2011.02578 §3.1.3]. The features then feed a kernel density estimator or a one-class SVM. The authors report a mean AUC of 89.9 with OC-SVM, against 86.9 for vanilla contrastive learning and 83.1 for an end-to-end rotation classifier from prior work [arxiv:2011.02578 tab. 2]. These are author-reported means over five settings. Hyperparameters were picked on CIFAR-10 using outlier labels, which the paper acknowledges [arxiv:2011.02578 app. A.4]. Among the batch sizes tried, 32 scored best on the CIFAR-10 validation set [arxiv:2011.02578 §5.1]. The [one-class card](/papers/2011.02578) lists where the released code differs, including the denominator of the contrastive loss.

**Contrastive features with Mahalanobis scoring (SSD-style).** The encoder is trained with SimCLR's NT-Xent loss on unlabelled in-distribution images only, with no outlier data [arxiv:2103.12051 §3] [arxiv:2103.12051 eq. 1]. At test time a single Gaussian is fitted to the training features, and an input is scored by its Mahalanobis distance from that cluster [arxiv:2103.12051 §3.1] [arxiv:2103.12051 eq. 2]. The paper uses 500 epochs, batch 512, SGD at learning rate 0.5 and temperature 0.5, with 0.1 for its anomaly experiment [arxiv:2103.12051 §4.1]. Averaged over four in-distribution and out-of-distribution pairs, the AUROC is 88.7 against 80.9 for the rotation-loss baseline, and 92.5 when five outlier samples per class are used [arxiv:2103.12051 tab. 1]. The score is sensitive to temperature: 66.7 at 0.001 against 89.5 at 0.5 [arxiv:2103.12051 tab. 5]. All figures are author-reported. The [SSD card](/papers/2103.12051) notes that the released code standardises each feature dimension and uses a pseudo-inverse, and that its checkpoint is chosen on CIFAR-10 test data.

**Per-pixel contrast with known correspondences.** VADeR applies the contrastive loss to pixels instead of whole images. Two views of one image are sampled so that they share at least 32 pixels, and each matched pixel pair is a positive. Negatives are pixels from other images, held in a MoCo queue of 65,536 entries [arxiv:2011.05499 §3.2]. The encoder starts from MoCo v2 [arxiv:2011.05499 §3.2]. The method needs known pixel correspondences between views, which the augmentations provide [arxiv:2011.05499 §1] [arxiv:2011.05499 §3.1]. With frozen features and a 1×1 convolution, the paper reports 56.7 mIoU on VOC, against 43.0 for MoCo and 54.4 for ImageNet-supervised ResNet-50 [arxiv:2011.05499 tab. 1]. When pixel pairs are not matched, frozen VOC segmentation falls to 32.4 [arxiv:2011.05499 tab. 3]. The [VADeR card](/papers/2011.05499) finds no official code, and notes that the queue update rule and learning-rate schedule are not stated.

**Image self-supervision added to language contrast (SLIP-style).** The image encoder is trained on CLIP's image-text loss and on a SimCLR loss over two augmented views, summed with weight 1.0 [arxiv:2112.12750 §3.3]. On YFCC15M for 25 epochs with ViT-B/16, the combined model reaches 72.1% linear ImageNet accuracy and 42.8% zero-shot, against 66.5% and 37.6% for the authors' CLIP run [arxiv:2112.12750 tab. 2]. The CLIP baseline is the authors' own re-implementation [arxiv:2112.12750 §4]. The fine-tuned gain is smaller, 82.6% against 80.5% [arxiv:2112.12750 tab. 2]. The cost is higher: 30.5 hours against 22.3 on 64 V100 GPUs for ViT-B/16 [arxiv:2112.12750 §7].

**Distribution matching for strong views (CLSA-style).** The standard contrastive loss is kept on gentle views. A strongly augmented query is trained so that its softmax over the key queue matches that of its weakly augmented twin, which acts as a teacher with gradients stopped [arxiv:2104.07713 §3.2] [arxiv:2104.07713 eq. 5–6]. The two terms are summed with weight 1.0 [arxiv:2104.07713 eq. 7]. With ResNet-50 at 200 epochs, the single-crop variant reaches 69.4% ImageNet linear top-1 against 67.5% for MoCo v2 [arxiv:2104.07713 tab. 2]. Multi-crop CLSA* reaches 76.2% at 800 epochs, against 76.5% for supervised training [arxiv:2104.07713 tab. 3]. Putting strong views into the contrastive loss directly does not help: a strong query alone scores 67.7 and a strong query with a strong key scores 67.0 [arxiv:2104.07713 tab. 5]. The cost is one extra forward pass per strong view. All results are author-reported. The [CLSA card](/papers/2104.07713) lists the open licence question for its code.

**Pros and cons.** Encoder-only and lightweight, and strong on discriminative tasks. Open questions are negative sampling (tricky, biased, slow) and the lack of theory for why augmentations help, which hinders discrete domains such as text and graphs [arxiv:2006.08218 §4.4].

### Adversarial: generator against discriminator

These methods keep a decoder, so the representation must reconstruct as well as discriminate. The GAN objective is a min-max game [arxiv:2006.08218 eq. 16]. The survey calls them more expressive but harder to optimise and less stable than contrastive learning [arxiv:2006.08218 §5]. ELECTRA is the text case. A small masked-LM generator fills masked positions, and a discriminator labels each token as original or replaced [arxiv:2006.08218 §5.3] [arxiv:2006.08218 eq. 19]. It is not trained as a GAN because discrete tokens block gradients [arxiv:2006.08218 §5.3]. The reported result is that ELECTRA surpasses BERT at equal compute, with no number given in the survey [arxiv:2006.08218 §5.3].

### Frozen encoder with a task head

Pre-train a large encoder without labels, freeze it, and train a head for the task. The encoder stays fixed in each case below, unless the paragraph says otherwise.

**Transform prediction on unlabelled motion windows (multi-task HAR).** A 1-D CNN with eight heads is trained to say which of eight signal transforms was applied to an accelerometer window [arxiv:1907.11879 §3.1] [arxiv:1907.11879 §3.2]. The convolutional trunk is then reused for activity classification. Kappa is 0.76 to 0.88 with the trunk frozen and 0.76 to 0.89 with its last conv layer fine-tuned, across six smartphone datasets [arxiv:1907.11879 tab. 2]. Frozen features beat autoencoder features on all six, but trail supervised training on three of them [arxiv:1907.11879 tab. 2]. The transform parameters, task-loss weights and batch size are not stated [arxiv:1907.11879 alg. 1], and no official code was found. The [multi-task HAR card](/papers/1907.11879) lists the other gaps.

**Dense output.** For canopy height, a DINOv2 ViT-H is pre-trained on 18 million unlabelled satellite tiles, frozen, and a DPT decoder is trained from scratch on aerial lidar heights. The decoder predicts a histogram over 256 height bins per pixel, and the output is the expected height [arxiv:2304.07213 §2.3.2] [arxiv:2304.07213 §3.1] [arxiv:2304.07213 §3.2]. The pre-training is reported at about 16,896 GPU-hours [arxiv:2304.07213 app. D]. Author-reported NEON test MAE with GEDI correction is 2.7 m, against 3.0 m for a ResUNet trained from labels [arxiv:2304.07213 tab. 2]. That baseline has no self-supervised pre-training, so the table does not isolate what pre-training contributes. The card is [the canopy-height paper](/papers/2304.07213).

**Policy head.** A masked autoencoder is pre-trained on about 700k images of hands and objects from video and image collections, then frozen. A small PPO controller reads the classification-token feature and the robot's joint state [arxiv:2203.06173 §2.1] [arxiv:2203.06173 §2.2]. The authors report that the controller beats a supervised ImageNet encoder on 7 of 8 pixel-based tasks, with the per-task values shown only in a figure [arxiv:2203.06173 §5.1] [arxiv:2203.06173 fig. 5]. Each image-based run takes about 5 hours on one 2080 Ti GPU for about 8 million environment steps [arxiv:2203.06173 §3.2].

**Skeleton lifting pretext (MotionBERT-style).** A spatio-temporal Transformer (DSTformer) is pre-trained to lift corrupted 2D skeleton sequences to 3D. Motion-capture data supplies 3D targets, and in-the-wild video, which has no 3D labels, supplies a 2D reprojection loss [arxiv:2210.06551 §3.2] [arxiv:2210.06551 §3.3]. The encoder is then fine-tuned with a small head for 3D pose, action recognition or mesh recovery [arxiv:2210.06551 §3.4]. Clips are 243 frames for mocap, 81 for InstaVariety and 30 for PoseTrack, and training runs 90 epochs, the first 30 with the 3D loss only [arxiv:2210.06551 §4.1] [arxiv:2210.06551 §4.2]. Author-reported Human3.6M MPJPE, in mm, is 39.2 from scratch and 37.5 after pre-training, with detected 2D poses [arxiv:2210.06551 tab. 1]. On NTU-RGB+D, cross-subject and cross-view accuracy is 87.7 and 94.1 from scratch, against 93.0 and 97.2 after pre-training [arxiv:2210.06551 tab. 2]. The [MotionBERT card](/papers/2210.06551) notes that the released configs use different epoch counts and an extra scale loss.

**Frozen dense features, no backbone training.** Keys from the last attention block of a DINO ViT serve as patch descriptors. K-means over pooled keys, a saliency vote and GrabCut give zero-shot co-segmentation, and best-buddies matching gives point correspondences [arxiv:2112.05814 §4]. The authors report 79.5 J_m on PASCAL-CO, above the best unsupervised method in their table at 75.8 [arxiv:2112.05814 tab. 2]. Keys beat tokens, queries and values in their facet ablation [arxiv:2112.05814 tab. 3]. STEGO keeps the DINO ViT frozen and trains a small head whose 70-dimensional pixel codes keep the feature correlations of the backbone. The codes are then clustered and refined with a CRF [arxiv:2203.08414 §3.2] [arxiv:2203.08414 §3.3]. On CocoStuff-27 it reports 28.2 unsupervised mIoU, against 14.4 for PiCIE+H [arxiv:2203.08414 §4.2].

**Frozen denoiser features for correspondence (DIFT-style).** No backbone is trained. A real image is noised to a chosen level t and passed once through a pretrained diffusion U-Net. Features are read from one intermediate block, and several noise draws are averaged [arxiv:2306.03881 §4.2] [arxiv:2306.03881 app. B] [arxiv:2306.03881 app. C]. Matching is nearest neighbour by cosine similarity [arxiv:2306.03881 §3]. On SPair-71k, Stable Diffusion features reach 52.9 PCK per image, against 33.9 for DINO, 38.4 for OpenCLIP and 55.3 for the supervised SCorrSAN [arxiv:2306.03881 tab. 1]. On PF-WILLOW the method scores 81.2 against 80.0 for SCorrSAN [arxiv:2306.03881 tab. 3]. The noise level and block index are grid-searched on SPair-71k, which is also the reported benchmark [arxiv:2306.03881 §5.2]. All numbers are author-reported. The [DIFT card](/papers/2306.03881) lists where the released code differs from the paper.

**Retrieval for copy detection (DCR).** Each generated image is scored against every training image with frozen features, and a high score means one shared object [arxiv:2212.03860 §3] [arxiv:2212.03860 §4]. Among ten extractors, DINO ViT-B/16 with token-wise split-product similarity ties Swin-B for the best average rank, 4.1 [arxiv:2212.03860 tab. 1]. Small diffusion models trained on 300 or 3,000 Celeb-A images produce visible copies. A class-conditional latent diffusion model on ImageNet never exceeds 0.65 similarity to its training data [arxiv:2212.03860 §5] [arxiv:2212.03860 §6]. Stable Diffusion v1.4 has about 1.88% of 9,000 generations with a top-1 match above 0.5, but the search covered only a 12M-image subset of its training data, and the authors call the rate an underestimate [arxiv:2212.03860 §7] [arxiv:2212.03860 §9]. The [DCR card](/papers/2212.03860) lists where the released code normalises features differently from the paper.

**Masked reconstruction of a long history, frozen, then a forecaster (STEP-style).** A small Transformer is pre-trained once per dataset on one to two weeks of traffic history. It hides 75% of the 12-step patches and rebuilds them [arxiv:2206.09113 §3.1]. The pre-trained model is then frozen. Its patch states, and a kNN-based regulariser for a learned sparse graph, feed a Graph WaveNet forecaster [arxiv:2206.09113 §3.2]. The long history is therefore paid for in advance, and the forecaster keeps a short input window. The authors report the best MAE, RMSE and MAPE in nearly all cells on METR-LA, PEMS-BAY and PEMS04 against 12 baselines [arxiv:2206.09113 tab. 2]. The released code differs from the paper in ways that change the learned graph, so the card's Pitfalls section should be read before reimplementing [arxiv:2206.09113 Pitfalls]. The [STEP card](/papers/2206.09113) lists them.

### Vision-only tokens beside a CLIP encoder (interleaved MoF-style)

A multimodal LLM with only a CLIP image encoder answers simple detail questions poorly. The paper builds a benchmark from image pairs that CLIP embeds almost identically (cosine above 0.95) while a vision-only encoder, DINOv2, separates them (cosine below 0.6) [arxiv:2401.06209 §2.1]. LLaVA with CLIP alone scores 5.5 on that benchmark, where chance is 25% [arxiv:2401.06209 tab. 2] [arxiv:2401.06209 §2.3]. Two ways of adding DINOv2 are tested. A linear blend of the two feature sets, with DINOv2 share α, reaches 18.7 at α = 0.75, but LLaVA-bench falls from 81.8 to 75.8 at that share [arxiv:2401.06209 tab. 2] [arxiv:2401.06209 §4.2]. Interleaving the two token sequences, each through its own adapter, reaches 16.7 at 224 px and keeps LLaVA-bench at 82.8 [arxiv:2401.06209 tab. 3] [arxiv:2401.06209 §4.3]. The interleaved model feeds 512 visual tokens to the LLM where LLaVA feeds 256, and the paper's control, a plain resolution increase, gives no gain [arxiv:2401.06209 tab. 3] [arxiv:2401.06209 §4.3]. Replacing DINOv2 in the same slot with MAE or MoCo v3 gives 27.3 and 26.7 against 28.0, so the gain is not specific to DINOv2 [arxiv:2401.06209 tab. 6]. All figures are author-reported. The [MMVP card](/papers/2401.06209) records that the released scripts cover only the LLaVA-1 setting.

### Frozen encoder as a loss network

Splice trains a small U-Net generator on one pair of images, and uses a frozen DINO ViT-B/8 only to compute losses. No task head is trained [arxiv:2201.00424 §3] [arxiv:2201.00424 app. A.2]. The appearance term matches the CLS token of the deepest layer to that of the appearance image. The structure term matches the deepest-layer key self-similarity to that of the structure image, and an identity term keeps the appearance image's keys when the generator is fed that image [arxiv:2201.00424 §3.2] [arxiv:2201.00424 §3.3]. The paper weights the structure and identity terms at 0.1 each [arxiv:2201.00424 §3.3]. In a two-alternative forced-choice study on Wild-Pairs, outputs were preferred over STROTSS in 79.0% of judgements, and over WCT² in 83.1% [arxiv:2201.00424 tab. 1]. The method trains a new generator for each pair, which takes about 20 minutes at 512 × 512 on one GPU [arxiv:2201.00424 app. A.3]. The [Splice card](/papers/2201.00424) notes that the released code uses different loss weights and norms.

### Nearest-neighbour head on a self-supervised encoder

VINN fine-tunes an ImageNet-initialised ResNet-50 with BYOL on the demonstration frames, without action labels [arxiv:2112.01511 §3.1] [arxiv:2112.01511 §4.3]. It then embeds every demonstration frame, and predicts an action as a softmin-weighted average of the actions of the k nearest frames [arxiv:2112.01511 §3.2]. The paper states no temperature or bandwidth for those weights [arxiv:2112.01511 §3.2]. On a real door-opening task, VINN succeeded in 80% of 30 rollouts. A behaviour-cloning policy trained on the representations reached 53.3%, and end-to-end behaviour cloning reached 0% [arxiv:2112.01511 tab. I]. With the sign and handle occluded, VINN kept 70% success [arxiv:2112.01511 tab. II]. The best pre-training objective depends on the task. In offline test MSE, the VICReg-trained lookup beats the BYOL one on Pushing (1.50 against 2.43), while BYOL is better on Door Opening (0.92 against 1.05) [arxiv:2112.01511 tab. III]. Each query is compared with every demonstration embedding, so lookup cost grows with the number of frames [arxiv:2112.01511 §4.8]. The [VINN card](/papers/2112.01511) records that the gripper output is under-specified.

### Test-time adaptation on each input

The pretext loss can also run on a single test input before the prediction. TTT-MAE starts from a pretrained masked autoencoder with a ViT-B probe head, makes 128 copies of one image, gives each a random 75% mask, and takes 20 SGD steps on the reconstruction loss. It then predicts on the unmasked image and resets the weights for the next input [arxiv:2209.07522 §3] [arxiv:2209.07522 §4.1]. No label is used in the adaptation step, and labels are used only to score the output [arxiv:2209.07522 §4.1]. On ImageNet-C at severity 5, the paper reports a gain over the fixed ViT-probing baseline on all 15 corruptions, for example JPEG from 51.2% to 63.0% [arxiv:2209.07522 tab. 2]. In its rotation-invariant check, rotation-based test-time training hurts, while the MAE variant still helps on the four corruptions shown [arxiv:2209.07522 tab. 1]. Each input costs 20 optimiser steps over 128 copies, and the paper reports no per-image timing [arxiv:2209.07522 §4.1].

### Puzzle tasks beside a supervised loss (domain adaptation)

Sun et al. train one shared network on the labelled source task and on label-free puzzles: rotation, flip and patch location. Each puzzle step uses half source and half target images, and the source and target labels are dropped from the puzzle loss [arxiv:1909.11825 §4] [arxiv:1909.11825 app. B]. The main objective is the plain sum of the task loss and the puzzle losses, with no adversarial term [arxiv:1909.11825 eq. 2]. The appendix trains one gradient step per puzzle per batch, and says that a joint step over all losses changes accuracy by usually less than 1% [arxiv:1909.11825 app. B]. Pixel-reconstruction tasks such as inpainting and colourisation were argued to push the two domains apart, and the paper reports they were little better than source-only [arxiv:1909.11825 §3]. Over seven benchmarks, the method is best on four, including 98.9 on MNIST→MNIST-M, tied with DIRT-T, and 96.5 on MNIST→USPS [arxiv:1909.11825 tab. 2]. Rotation gave trivial solutions on SVHN, because the network can read the neighbouring digits at the crop edges, so each puzzle needs testing on the target data [arxiv:1909.11825 §5.1]. The [paper card](/papers/1909.11825) records that the released code picks the checkpoint with target test images and source test labels, not the unlabelled validation sets the paper describes.

### Self-generated forgeries as training pairs (I2G-style)

Here the self-supervised part is the data, not a separate pre-training stage. A generator makes forged face frames from real ones and keeps the pixel mask it used, so the mask labels cost nothing. A detector is then trained on a patch-pair consistency target built from that mask. Real frames get an all-agree target [arxiv:2012.09311 §3.1] [arxiv:2012.09311 §4.2]. Each real training sample has a 50% chance of being replaced by a generated fake [arxiv:2012.09311 §4.2]. Its ablation on the cross-dataset average AUC gives 73.74 with the consistency loss off and 82.83 with it on at the chosen weight [arxiv:2012.09311 tab. 7]. Using the generator in pre-training alone, then fine-tuning on the target data, averages 84.89 against 79.19 without [arxiv:2012.09311 §4.5]. The paper's own training is a supervised detector, so the self-supervised reading is ours. The generator's landmark threshold and blend set are not stated, and the paper's code is not public, so the numbers are author-reported [arxiv:2012.09311 Pitfalls]. The [detector card](/papers/2012.09311) lists these gaps.

### Self-supervised registration with synthetic motion (PRNet-style)

Here the self-supervised task is the target task itself, with no pre-train and reuse step. Two partial views of one shape are made with a known rigid motion between them. That motion is the label, so no hand-labelled keypoints are used [arxiv:1910.12240 §4.1] [arxiv:1910.12240 §4.3]. Each pass keeps the keypoints with the largest feature norms, matches them with a Gumbel-Softmax, and solves the rigid motion in closed form. The source is moved and the pass is repeated, three times in the released code [arxiv:1910.12240 §3.2] [code: main.py#L85-L86]. On ModelNet40 partial-to-partial, the author-reported rotation MSE is 10.235, against 45.005 for DCP-v2 [arxiv:1910.12240 tab. 1]. On unseen categories, FGR has lower rotation MAE, 1.952 against 2.329 [arxiv:1910.12240 tab. 2]. For the Stanford Bunny, the model was fine-tuned at inference time on 100 synthetic examples per scan, and the paper gives only qualitative results [arxiv:1910.12240 §4.2]. The [PRNet card](/papers/1910.12240) notes that test-time matching is sampled at random, and that the temperature clamp is in the code but not in the paper.

### Graph self-supervision

For graphs the pretext tasks fall into four categories [arxiv:2103.00111 §3.2]:

- **Generation:** perturb the input and rebuild features or edges, for example GAE [arxiv:2103.00111 eq. 3].
- **Auxiliary property:** predict cheap properties such as cluster index or node degree [arxiv:2103.00111 eq. 4].
- **Contrast:** two augmented views of a node, subgraph or graph; augmentations are feature masking or shuffling, and edge modification, graph diffusion or subgraph sampling [arxiv:2103.00111 §6.1] [arxiv:2103.00111 eq. 5].
- **Hybrid:** several of the above trained with a weighted sum [arxiv:2103.00111 §7].

**Auxiliary term in joint training.** SimP-GCN adds a self-supervised term to node classification. For sampled pairs of nodes it regresses the cosine similarity of their input features from the difference of their first-layer hidden states, and the loss is the classification loss plus λ times that term [arxiv:2011.09643 §4.2] [arxiv:2011.09643 §4.3]. Its ablation on the disassortative graphs, with the other settings matched, gives 84.05 vs 82.70 on Cornell and 36.20 vs 36.09 on Actor [arxiv:2011.09643 tab. 5]. These are author-reported. The paper gives the grid for λ but not how the value was picked [arxiv:2011.09643 §5.1]. The [SimP-GCN card](/papers/2011.09643) records where the released code differs from the paper.

**Mutual information on multiplex graphs.** HDMI runs one one-layer GCN per relation layer. Each node is trained with a binary discriminator that scores three pairings: the node embedding with the graph summary, with the node's own attributes, and with the attributes of another node [arxiv:2102.07810 §3.1.1]. Negatives come from shuffling the attribute rows while the graph stays fixed [arxiv:2102.07810 §3.1.2]. Semantic attention then fuses the layer embeddings per node, and the fused embedding is trained with the same objective [arxiv:2102.07810 §3.2]. The authors report the best or tied-best Macro-F1 on ACM, IMDB, DBLP and Amazon, at 0.901, 0.650, 0.820 and 0.808 [arxiv:2102.07810 tab. 3]. For clustering and similarity search, a baseline is slightly ahead on ACM [arxiv:2102.07810 tab. 4]. Results come from a single configuration [arxiv:2102.07810 §4.1.4]. Each step encodes the full graph twice, and the DBLP PATAP layer alone has about 57 million edges, so memory is the constraint [arxiv:2102.07810 tab. 2]. The [HDMI card](/papers/2102.07810) notes that the code has no licence file.

**Node against its local subgraph (CoLA-style).** For each node, the positive pair is the node with a sampled neighbourhood of its own. The negative pair is the node with the neighbourhood of another node. A one-layer GCN with a bilinear discriminator is trained to tell them apart [arxiv:2103.00113 §4.1] [arxiv:2103.00113 §4.2]. The anomaly score is the mean, over 256 sampled rounds, of the negative score minus the positive score [arxiv:2103.00113 §4.3] [arxiv:2103.00113 eq. 12]. The authors report the best ROC-AUC on all seven benchmarks, from 0.7513 on Flickr to 0.9512 on Pubmed [arxiv:2103.00113 tab. 3]. Those benchmarks inject synthetic anomalies, so the figures measure recovery of injected anomalies, not real anomaly rates [arxiv:2103.00113 §5.1]. The [CoLA card](/papers/2103.00113) lists where the released code differs from the paper, including how negatives are drawn.

**Adaptive augmentation for node contrast (GCA-style).** Two views of a graph are made by dropping edges and masking feature dimensions. The probability of removing an edge or a feature dimension falls as its centrality rises, so the influential parts survive in both views [arxiv:2010.14945 §1] [arxiv:2010.14945 §3.3.1] [arxiv:2010.14945 §3.3.2]. Centrality is degree, eigenvector or PageRank, and each view has its own strengths [arxiv:2010.14945 §3.3.1] [arxiv:2010.14945 §3.3.2]. Node embeddings are trained with the cross-view InfoNCE-style loss, and a logistic regression is fitted on the frozen encoder [arxiv:2010.14945 §3.2] [arxiv:2010.14945 §4.1]. On five benchmarks the paper reports the best label-free accuracy, and the best GCA variant is above both GCN and GAT on four of them, by 0.07 to 0.92 points over the stronger of the two. On Amazon-Photo, GAT is ahead [arxiv:2010.14945 tab. 3]. In the paper's ablation, each adaptive part helps and both together are best on all five datasets [arxiv:2010.14945 tab. 4]. The [GCA card](/papers/2010.14945) lists where the released code differs from the paper, including the edge-weight rule and the PageRank update.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Vision transformer, fine-tune on the target task, no labels up front | Masked image modelling (SimMIM-style) | SimMIM reports 83.8% ImageNet-1K fine-tune top-1 for ViT-B, against 83.2 for BEiT and MoCo v3 and 82.8 for DINO [arxiv:2111.09886 tab. 6]. Results are author-reported. |
| Frozen features, linear probe, or retrieval embeddings | Contrastive or self-distillation | The same table lists linear-probe top-1 of 78.2 for DINO and 76.7 for MoCo v3 against 56.7 for SimMIM. SimMIM's authors do not optimise for linear evaluation [arxiv:2111.09886 §3.5]. |
| Vision transformer, contrastive pre-training with a large batch, linear probe and fine-tuning both wanted | MoCo v3-style momentum contrast with a frozen random patch projection | 76.7% linear and 83.2% fine-tuned for ViT-B/16 on ImageNet-1K, author-reported [arxiv:2104.02057 tab. 1]. Random projection gives 73.4% against 71.7% for learned at base lr 1.5e-4 [arxiv:2104.02057 fig. 5]. Training can still destabilise at high learning rates [arxiv:2104.02057 §4.2]. |
| Small batch or single GPU | Memory bank or momentum queue, or a negative-free method | End-to-end contrast ties negatives to batch size [arxiv:2010.05113 §III.C.1]. Queues keep keys consistent without large batches [arxiv:2010.05113 §III.C.2]. |
| Large compute budget, batch size can grow | In-batch contrast (SimCLR-style) with strong augmentation | SimCLR's gains come with large batches and long training [arxiv:2006.08218 §4.2.2]. |
| Batch size is a concern | Negative-free (BYOL-style) | Reported ImageNet drop from batch 512 to 4096 is −0.3% for BYOL against −1.4% for SimCLR [arxiv:2006.08218 §4.2.2]. Second-hand figures from one setting. |
| Text | Generative language-model pre-training | The 2020 survey calls contrastive pre-training for NLP unconvincing, with contrastive work mostly in fine-tuning and sentence embeddings [arxiv:2006.08218 §4.4]. A 2022 contrastive text model reports strong embeddings, see Contradictions. |
| Graph, few labels | Contrast with cross-scale views, or a pre-train-then-fine-tune scheme | Contrastive methods are reported to beat generation and auxiliary methods on node classification [arxiv:2103.00111 tab. 5]. Cross-scale tends to beat same-scale on graph classification [arxiv:2103.00111 tab. 7]. |
| Graph, large graphs | Contrast over generation | Generation-based methods can be memory-heavy on large graphs [arxiv:2103.00111 §3.2]. Contrast needs time-consuming tuning of augmentations and losses [arxiv:2103.00111 §3.2]. |
| Graph classification with labels available | Run the supervised baseline first | Graph URL methods score below the supervised GIN baseline on the surveyed graph-classification datasets [arxiv:2103.00111 tab. 7]. |
| Node classification with heterophily and labels available | Joint training with a pair-regression auxiliary term | SimP-GCN's term adds 1.35 points on Cornell and 0.11 on Actor in its own ablation, against a supervised-only variant [arxiv:2011.09643 tab. 5]. Small and author-reported. |
| Vision transformer, strong linear probe and fine-tuning, long training budget | Masked prediction with a learned teacher (iBOT-style) | ViT-B/16 reports 79.5 linear probing and 84.0 fine-tuning on ImageNet-1K at 1600 effective epochs [arxiv:2111.07832 tab. 1] [arxiv:2111.07832 tab. 2]. Each step runs a second network [arxiv:2111.07832 §3.1]. |
| Vision transformer, fine-tune, cheaper pre-training, linear probe not needed | Masked autoencoder (MAE-style) | The encoder sees a quarter of the patches. Dropping mask tokens from the encoder cuts ViT-L pre-training from 42.4 to 15.4 hours on the paper's TPU setup, with 84.2 vs 84.9 fine-tuned [arxiv:2111.06377 tab. 2]. Linear probes are weak [arxiv:2111.06377 app. B]. |
| Few labels (1 to 5 per class), want no pixel reconstruction | Masked prediction in feature space (MSN-style) | Reports 65.5% at 5 labels per class for ViT-B/16 against 61.4% for DINO, and 72.4% for ViT-B/4 [arxiv:2204.07141 tab. 1]. No unmasked anchor view is needed, which saves memory [arxiv:2204.07141 §4]. |
| Distribution shift at deployment, no labels at test time, per-input compute affordable | Test-time reconstruction with a masked autoencoder | Improves over a fixed probe on all 15 ImageNet-C corruptions at severity 5, author-reported [arxiv:2209.07522 tab. 2]. Costs 20 optimiser steps per input [arxiv:2209.07522 §4.1]. |
| Dense per-pixel output, large unlabelled corpus, labels at few sites | Frozen SSL encoder with a trained decoder | Frozen ViT-H features reach 2.7 m NEON MAE with GEDI correction, against 3.0 m for a labelled ResUNet [arxiv:2304.07213 tab. 2]. Pre-training is costly [arxiv:2304.07213 app. D]. |
| Dense labels absent, object classes unknown, no training budget | Frozen DINO keys with clustering (co-segmentation or correspondence) | Keys give 79.5 J_m on PASCAL-CO, author-reported. Correspondence PCK reaches 56.48 at layer 9 with bins, against 61.43 for a supervised method [arxiv:2112.05814 tab. 2] [arxiv:2112.05814 tab. 4]. |
| Unsupervised pixel classes, a cheap head is acceptable | Frozen backbone with a distillation head (STEGO-style) | 28.2 unsupervised mIoU on CocoStuff-27 against 14.4 for PiCIE+H, under two hours on one V100 [arxiv:2203.08414 §4.2] [arxiv:2203.08414 §3.3]. |
| Robot control from pixels, many tasks, frozen features wanted | Frozen masked encoder with a PPO head | Beats a supervised ImageNet encoder on 7 of 8 tasks in the paper's figures [arxiv:2203.06173 fig. 5]. Results are in simulation only. |
| Physiological signals with a natural grouping (patient, lead, time) | Domain-defined positives (CLOCS-style) | Linear AUC 0.896 against 0.738 for SimCLR on Chapman, author-reported [arxiv:2005.13249 tab. 1]. |
| Negative-free method, want to skip tuning a trained predictor | DirectPred predictor | 72.4 top-1 against 69.9 for a trained linear predictor at 300 epochs on ImageNet [arxiv:2102.06810 tab. 9]. Needs one d×d eigendecomposition per update, or fewer with a longer period [arxiv:2102.06810 §4]. |
| Text or code retrieval, one embedding for many tasks | Contrastive text and code encoders on paired data | Strong on search and classification, weak on sentence similarity [arxiv:2201.10005 tab. 4] [arxiv:2201.10005 tab. 5]. Needs very large batches [arxiv:2201.10005 tab. 9]. |
| Sparse, irregular multivariate records (e.g. clinical), labels scarce | Forecasting pretext with a Transformer over (time, variable, value) triplets | PR-AUC 0.577 against 0.546 on MIMIC-III, and 0.498 against 0.467 on PhysioNet-2012, with and without pre-training [arxiv:2107.14293 tab. 5]. Attention cost grows with the number of observations, so the paper caps them [arxiv:2107.14293 §4.4]. |
| Labelled source, unlabelled target from another domain, no adversarial loss wanted | Puzzle tasks trained beside the supervised loss | Best on four of seven benchmarks in the paper [arxiv:1909.11825 tab. 2]. Each puzzle adds one gradient step per batch [arxiv:1909.11825 app. B], and the puzzle must suit the data [arxiv:1909.11825 §5.1]. |
| Only inlier images for training, outliers at test time, no outlier labels | Distribution-augmented contrastive features with OC-SVM or KDE | Mean AUC 89.9 with OC-SVM, author-reported [arxiv:2011.02578 tab. 2]. Hyperparameters were tuned with outlier labels [arxiv:2011.02578 app. A.4]. |
| Image out-of-distribution detection, no outlier labels, one distance score at test time | Contrastive features scored by Mahalanobis distance (SSD-style) | Mean AUROC 88.7 over four in-distribution and out-of-distribution pairs, against 80.9 for the rotation-loss baseline [arxiv:2103.12051 tab. 1]. Sensitive to temperature: 66.7 at 0.001 against 89.5 at 0.5 [arxiv:2103.12051 tab. 5]. |
| Dense per-pixel output, frozen features, views with known pixel correspondences | Per-pixel contrast with a momentum queue | 56.7 mIoU on VOC with frozen features, against 43.0 for MoCo [arxiv:2011.05499 tab. 1]. Needs known correspondences between views [arxiv:2011.05499 §1]. |
| Attributed multiplex graph, no labels, several relation types | Mutual-information node objective with attention fusion | Best or tied-best Macro-F1 on all four datasets [arxiv:2102.07810 tab. 3]. Full-graph training makes memory the limit [arxiv:2102.07810 tab. 2]. |
| Few demonstrations, visual changes expected, no policy training wanted | Fine-tuned encoder with a nearest-neighbour action lookup | 80% door opening against 53.3% for behaviour cloning on representations [arxiv:2112.01511 tab. I]. Lookup cost is linear in the number of demonstration frames [arxiv:2112.01511 §4.8]. |
| One image pair, layout to keep, appearance from another, no training set | Frozen ViT losses on a generator trained per pair | 79.0% preference against STROTSS on Wild-Pairs [arxiv:2201.00424 tab. 1]. About 20 minutes per pair at 512 × 512 [arxiv:2201.00424 app. A.3]. |
| Frozen ViT embeddings for k-NN lookup or retrieval, no pixel targets | Self-distillation with multi-crop (DINO-style) | 74.5 k-NN and 77.0 linear with ViT-S/16 [arxiv:2104.14294 tab. 2]. Multi-crop adds 4.9 k-NN points at 300 epochs [arxiv:2104.14294 tab. 7]. Author-reported. |
| One training recipe for speech, vision and text, with no tokenizer to learn | Regression of averaged teacher layers (data2vec-style) | 84.2 ViT-B top-1 on ImageNet-1K, author-reported [arxiv:2202.03555 tab. 1]. Speech masking probability is unclear in the paper [arxiv:2202.03555 §4.2] [arxiv:2202.03555 tab. 8]. |
| Unlabelled time series, frozen encoder with a linear classifier | Temporal prediction plus contextual contrast (TS-TCC-style) | 90.37% on HAR against 90.14% for supervised training, but 83.00% against 83.41% on Sleep-EDF [arxiv:2106.14112 tab. 2]. Single-dataset, five seeds. |
| EEG or other long biosignals, many unlabelled hours, small labelled sets | Masked-span contrast over convolutional features (BENDR-style) | Author-reported balanced accuracy of 86.7 on motor imagery and AUROC of 0.72 on P300 [arxiv:2101.12037 tab. 2]. Pre-training used about 1.5 TB of EDF [arxiv:2101.12037 §2.1]. The pre-training learning rate and batch size are not stated. |
| Speech in many languages, or low-resource languages, one shared model wanted | Multilingual masked contrast (XLS-R-style), largest model affordable | 27.8 against 26.6 for an English-only 720M model, 15 directions, author-reported [arxiv:2111.09296 tab. 4]. At 0.3B, the English-only model is ahead on English LibriSpeech with 10 min of labels [arxiv:2111.09296 tab. 11]. |
| Video clips, little labelled data, no image-pretrained weights, plain ViT wanted | Masked video autoencoder with tube masking (VideoMAE-style) | 91.3% on UCF101 against 81.7% for MoCo v3 and 51.4% from scratch, and 62.6% against 39.2% on HMDB51, on each dataset's own unlabelled split [arxiv:2203.12602 tab. 2]. SSv2 pre-training takes 19.5 hours on 64 V100s, against 61.7 for MoCo v3 [arxiv:2203.12602 tab. 3]. |
| Video, large model, memory or compute is the limit | Masked video autoencoder with dual masking (VideoMAE V2-style) | Dual masking cuts ViT-g memory from 1753M to 1050M, a 1.48× speed-up in the paper's setting [arxiv:2303.16727 tab. 2]. At ViT-B on SSv2, top-1 is 70.15 against 70.28 for encoder-only masking [arxiv:2303.16727 tab. 1]. Post-pre-training on K710 helps Kinetics-400 but not SSv2 [arxiv:2303.16727 §4.2]. |
| Satellite or other remote-sensing series, several spectral bands, labels scarce | Masked autoencoder with temporal and band-group encodings (SatMAE-style) | 81.49% top-1 on fMoW temporal against 78.90% for MAE with three copies of the image [arxiv:2207.08051 §5.3]. Per-band grouping (10 groups) gave a very large memory footprint and often NaN loss [arxiv:2207.08051 app. A.3.3]. |
| Per-pixel embeddings from irregular satellite series, reused with small heads, few labels | Barlow Twins over sampled date subsets (TESSERA-style) | Small heads on frozen 128-d embeddings are best in eight of the nine settings in the paper's table, the exception being PASTIS-R with all labels [arxiv:2506.20380 tab. 1]. Author-reported. Pre-training cost is about 6,200 GPU hours [arxiv:2506.20380 app. Pre-training Details]. |
| Node embeddings from an unlabelled graph, a simple classifier on top, no graph-specific labels | Contrast of two centrality-weighted graph views (GCA-style) | Best label-free accuracy on all five benchmarks in the paper; the best GCA variant beats the stronger of GCN and GAT on four, by 0.07 to 0.92 points [arxiv:2010.14945 tab. 3]. Author-reported. |
| Multimodal LLM misses fine visual detail with a CLIP-only encoder, and about twice the visual tokens is affordable | Interleave a second vision-only encoder's tokens with the CLIP tokens (interleaved MoF-style) | 16.7 against 5.5 on MMVP for LLaVA at 224 px, with LLaVA-bench at 82.8 against 81.8 for the baseline [arxiv:2401.06209 tab. 2] [arxiv:2401.06209 tab. 3]. Author-reported; the released scripts cover the LLaVA-1 setting only. |
| Choosing how to adapt a pre-trained encoder | Weigh compute, task data and gradient access | Factors named by the foundation-model report. It gives no measurements [arxiv:2108.07258 §4.3]. |
| Attributed graph, no anomaly labels, anomalous nodes to rank | Node-versus-local-subgraph contrast with sampled scores (CoLA-style) | ROC-AUC 0.7513 to 0.9512 across seven benchmarks with injected anomalies, author-reported [arxiv:2103.00113 tab. 3]. Scoring draws 256 sample rounds per node [arxiv:2103.00113 §4.3]. |
| Tabular rows, few labels, neural model wanted | Contrastive plus denoising pre-training on CutMix and mixup views (SAINT-style) | Mean AUROC over 14 binary datasets is 86.91 with pre-training at 50 labels, against 85.78 without and 84.08 for CatBoost [arxiv:2106.01342 tab. 3]. Author-reported. |
| Image-text pairs and unlabelled images both available, linear probe or zero-shot wanted | Image SimCLR loss added to the CLIP loss (SLIP-style) | 72.1% linear and 42.8% zero-shot for ViT-B/16 on YFCC15M, against 66.5% and 37.6% for CLIP [arxiv:2112.12750 tab. 2]. About 30.5 against 22.3 GPU-hours on 64 V100s [arxiv:2112.12750 §7]. |
| Video backbone for action recognition and video-text tasks, large compute budget | Masked video encoder fused with a video-text encoder (InternVideo-style) | 91.1% top-1 on Kinetics-400 [arxiv:2212.03191 tab. 2] and 77.2% on Something-Something V2 [arxiv:2212.03191 tab. 3], author-reported. The fusion stage is not in the released code, per the [InternVideo card](/papers/2212.03191). |
| Check whether generated images copy training images, no labels | Frozen DINO ViT-B/16 keys with token-wise split-product retrieval | Ties Swin-B for the best average rank among ten extractors, 4.1 [arxiv:2212.03860 tab. 1]. About 1.88% of Stable Diffusion v1.4 samples have a top-1 match above 0.5, in a 12M-image subset [arxiv:2212.03860 §7] [arxiv:2212.03860 §9]. |
| Vision transformer for dense tasks (segmentation, detection) after fine-tuning, linear probe not needed | Masked prediction with an encoder-side regressor (CAE-style) | ADE20K mIoU 50.1 for ViT-B at 1600 epochs against 48.1 for MAE, and COCO AP^b 50.2 against 48.4 [arxiv:2202.03026 tab. 2] [arxiv:2202.03026 tab. 3]. Linear probing reaches 71.4, below iBOT's 79.5 [arxiv:2202.03026 tab. 1]. Author-reported. |
| Labelled target is small, a closely related unlabelled dataset much larger than the target exists | Self-supervised pre-training on that dataset, then fine-tune | The survey ranks this third of six source types, after supervised training on a very similar labelled source and on a closely related source with pseudo-labels [arxiv:2205.09904 §4.2]. The ranking rests on the survey's review of cited work, not on its own experiment. |
| Dense correspondence between two photos, no correspondence labels, a pretrained diffusion model available | Frozen diffusion U-Net features with nearest-neighbour matching (DIFT-style) | 52.9 PCK per image on SPair-71k against 33.9 for DINO, and 2.4 points below supervised SCorrSAN at 55.3 [arxiv:2306.03881 tab. 1]. The noise level and block index are set per task [arxiv:2306.03881 §4.2]. |
| Partial point clouds to register, no correspondence labels, synthetic motions can be sampled | Self-supervised keypoint matching with closed-form pose (PRNet-style) | Rotation MSE 10.235 against 45.005 for DCP-v2 on ModelNet40 partial-to-partial, author-reported [arxiv:1910.12240 tab. 1]. FGR has lower rotation MAE on unseen categories [arxiv:1910.12240 tab. 2]. |
| Skeleton sequences for 3D pose or action recognition, mocap plus unlabelled video | Masked 2D-to-3D lifting pre-training (MotionBERT-style) | Human3.6M MPJPE 37.5 after pre-training against 39.2 from scratch, with detected 2D poses [arxiv:2210.06551 tab. 1]. NTU-RGB+D cross-subject 93.0 against 87.7 from scratch [arxiv:2210.06551 tab. 2]. Author-reported. |
| Forgery detection on face frames, no mask labels, real data only for training | Generate forgeries with known masks and train a patch-consistency detector (I2G-style) | Cross-dataset average AUC 82.83 with the consistency loss against 73.74 without it [arxiv:2012.09311 tab. 7]. The released code is absent, and the landmark threshold and blend set are unstated [arxiv:2012.09311 Pitfalls]. |
| Long multivariate history, forecaster with a short window, pre-training budget affordable | Frozen masked Transformer over patches feeding a graph forecaster (STEP-style) | Best MAE, RMSE and MAPE in nearly all cells on three traffic datasets, author-reported [arxiv:2206.09113 tab. 2]. The code changes the learned graph; see the [STEP card](/papers/2206.09113). |
| Speech in a low-resource language or domain, a survey of method families first | Generative, contrastive or predictive speech pre-training (see the review) | The review finds wav2vec 2.0 and HuBERT competitive with few labels, but its numbers mix recipes [arxiv:2205.10643 §4.5] [arxiv:2205.10643 §7]. A survey, so no new measurement. |
| Contrastive pre-training where strong augmentations are wanted, MoCo-style queue available | Distribution matching on strong views (CLSA-style) | 69.4% against 67.5% for MoCo v2 at 200 epochs with ResNet-50, author-reported [arxiv:2104.07713 tab. 2]. A strong query without the matching term scores 67.7 [arxiv:2104.07713 tab. 5]. Each strong view adds a forward pass [arxiv:2104.07713 §3.2]. |
| Smartphone or wearable motion data, no activity labels, a small labelled set for the final classifier | Transform prediction as the pretext, frozen convolutional trunk | Kappa 0.76 to 0.88 across six datasets with the trunk frozen. Frozen features beat autoencoder features on all six, and trail supervised training on three [arxiv:1907.11879 tab. 2]. |
| Low-resource speech with a large untranscribed pool, a small transcribed set, compute for several training rounds | Pre-train, then noisy-student rounds on pseudo-labels | Word error rate 0.568 to 0.515 after five rounds on the authors' own evaluation set, one run [arxiv:2609.24410 tab. 4]. No optimiser or batch settings are given [arxiv:2609.24410 Hyperparameters]. |

## Implementation sketch

The steps shared by generative and contrastive methods are the same. The pretext input, the encoder, the head and the loss differ. The head is discarded after pre-training.

```pseudocode
# Pretext stage, shared by generative and contrastive methods
# Input: unlabelled batch x of shape (B, ...); encoder f; head h
# Output: encoder f for downstream use; head h is discarded

# 1. Build the pretext input from the data itself.
#    Generative: hide parts of x  [arxiv:2111.09886 §3.2]
#    Contrastive: two random views of each x  [arxiv:2010.05113 §II.C]
x_in = mask(x)              # generative
x_a, x_b = aug(x), aug(x)   # contrastive

# 2. Encode. For masking, masked tokens are replaced by a learnable
#    vector before the encoder runs  [arxiv:2111.09886 §3.2]
#    MAE instead drops the masked tokens entirely  [arxiv:2111.06377 §3]
z = f(x_in)

# 3. Head maps the encoder output to the prediction space
#    [arxiv:2010.05113 §III.D]
pred = h(z)

# 4. Score the prediction against the target.
#    Masked pixels only, l1  [arxiv:2111.09886 eq. 1]
#    InfoNCE over one positive and K negatives  [arxiv:2006.08218 eq. 9]
#    Feature-space variant: cross-entropy to sharpened prototype
#    assignments of an EMA teacher  [arxiv:2204.07141 §3]
loss = masked_l1(pred, x) | info_nce(pred_a, pred_b, negatives)

# 5. Update f and h. After pre-training keep f; the standard recipe
#    drops h, though one intermediate head layer may transfer better
#    [arxiv:2010.05113 §VI.C]
```

## Contradictions

- **Generative or contrastive for ViT fine-tuning.** The 2020 survey states that generative methods are less competitive than contrastive ones on some computer-vision classification benchmarks [arxiv:2006.08218 §3.5]. The SimMIM card reports that masked image modelling fine-tunes better than MoCo v3 and DINO on ImageNet-1K ViT-B [arxiv:2111.09886 tab. 6], while its linear-probe score is far lower [arxiv:2111.09886 tab. 6]. The MAE paper reports the same pattern at larger scale: fine-tuning at 86.9 to 87.8 with ImageNet-1K data alone, and linear probing at 75.8 for ViT-L [arxiv:2111.06377 tab. 3] [arxiv:2111.06377 app. B]. The 2020 survey predates SimMIM and MAE, so it cannot cover these comparisons. iBOT, a masked method with a learned teacher, reports 79.5 linear probing for ViT-B [arxiv:2111.07832 tab. 1], so the low SimMIM score does not show that masked modelling gives weak linear features in general.
- **More negatives.** One survey says more negatives help, at the cost of large batches or memory [arxiv:2010.05113 §VI.B]. Another reports that beyond some threshold, more negatives can hurt [arxiv:2006.08218 §6.3.3], and notes that the InfoNCE bound tightens with more negatives without guaranteeing better features [arxiv:2006.08218 §6.2.2].
- **Is masked language modelling contrastive?** One survey files it under denoising auto-encoders [arxiv:2006.08218 §3.3.3]. The other counts it as contrastive, arguing that for word embeddings reconstruction and contrast are equivalent [arxiv:2010.05113 §V.A]. This is a definitional disagreement, not a measured one.
- **Contrastive NLP pre-training.** The 2020 survey calls it unconvincing so far [arxiv:2006.08218 §4.4]. The other survey lists InfoNCE-based language methods in its language table [arxiv:2010.05113 tab. 2], with a citation mix-up noted in its Pitfalls. A 2022 paper reports that contrastive pre-training on paired text and code gives the best unsupervised SentEval linear-probe average, 91.8 against 88.3 for DECLUTR [arxiv:2201.10005 tab. 2]. It also trails SimCSE on sentence similarity, 71.1 against 79.8 [arxiv:2201.10005 tab. 4]. The survey predates it, so the two do not directly conflict, but they disagree on how promising the method is.
- **MAE in the low-label regime.** MSN's table gives MAE ViT-B/16 40.5% at 5 labels per class, against 65.5% for MSN and 61.4% for DINO [arxiv:2204.07141 tab. 1]. MAE's paper favours partial or full fine-tuning and calls linear probing a poor proxy [arxiv:2111.06377 §4.3]. MSN's table notes that MAE uses partial fine-tuning in that comparison [arxiv:2204.07141 tab. 1]. The ranking therefore depends on the protocol, and the two papers do not measure the same thing.
- **Pixel or learned targets.** SimMIM's ablation finds an ℓ1 pixel target at 192² reaches 82.8 top-1 and the BEiT dVAE target reaches 82.7 [arxiv:2111.09886 tab. 5], so the target is not decisive in its setting. MAE's ablation at ViT-L/16 shows the same spread: normalised pixels 85.4 fine-tuned, dVAE tokens 85.3 and unnormalised pixels 84.9, but linear probing is 71.6 for dVAE and 73.5 for pixels [arxiv:2111.06377 tab. 1d]. iBOT argues that pixel regression spends capacity on high-frequency detail and gives weak semantics, and uses teacher distributions instead [arxiv:2111.07832 §1]. data2vec argues that local targets, whether tokens, pixels or units, carry little context, and regresses an average of teacher layers that depends on the whole input [arxiv:2202.03555 §1] [arxiv:2202.03555 §2]. The papers do not compare these targets under one protocol, so the disagreement is about reasoning and reported gains, not one measured head-to-head.
- **ViT-B fine-tune baselines across papers.** SimMIM lists DINO at 82.8 and MoCo v3 at 83.2 [arxiv:2111.09886 tab. 6]. iBOT lists DINO at 83.6 with 1600 effective epochs and BEiT at 83.4 with 800 [arxiv:2111.07832 tab. 2]. data2vec lists DINO at 82.8 and MAE at 83.6 for ViT-B [arxiv:2202.03555 tab. 1]. The recipes and epoch counts differ, so the ranking changes between the tables and should not be read across papers.
- **Reconstruction as a training signal.** The HDMI paper rejects reconstruction-based attribute losses, citing evidence that reconstruction error does not track embedding quality [arxiv:2102.07810 §1]. Image masked autoencoders reconstruct pixels and reach 86.9 fine-tuned top-1 for ViT-H on ImageNet-1K data alone [arxiv:2111.06377 tab. 3]. The two papers work on different data, so their results do not test each other.
- **Rotation as a pretext task.** In the one-class paper, rotation-prediction features with a KDE reach a mean AUC of 88.2, above vanilla contrastive features at 86.8 and below distribution-augmented contrast at 89.8 [arxiv:2011.02578 tab. 2]. The domain-adaptation paper reports that rotation gives trivial solutions on house-number crops [arxiv:1909.11825 §5.1]. The tasks differ, and the second result is a warning that the pretext has to suit the data. The SSD paper's rotation-loss baseline reaches a mean AUROC of 80.9 on its image out-of-distribution pairs, against 88.7 for contrastive features with Mahalanobis scoring [arxiv:2103.12051 tab. 1]. Its benchmark pairs and scoring rule differ from the one-class paper's, so the two rotation numbers are not comparable.
- **Uniformity.** The one-class paper argues that large contrastive batches push features towards uniformity, which hurts outlier detection [arxiv:2011.02578 §5.1]. The 2020 survey treats uniformity as a needed term and reports that an over-weighted uniformity term causes collapse [arxiv:2006.08218 §6.3.2]. The two describe different failure modes on different tasks.
- **Is a predictor needed to avoid collapse?** The linear analysis treats the predictor as one of the ingredients that keep negative-free training from collapsing [arxiv:2102.06810 §2]. DINO's ablation, which uses centring and sharpening, finds that adding a predictor gives 71.8 k-NN against 72.8 without one [arxiv:2104.14294 tab. 7]. MoCo v3, which keeps a momentum target and in-batch negatives, finds the prediction MLP adds about one point on ViT-B: 76.5% with it and 75.5% without [arxiv:2104.02057 §6.2]. The settings differ, with a linear model, a self-distillation ViT and a contrastive ViT as the three sides, so these results do not settle the question for BYOL-style methods.
- **Masking across time.** The VideoMAE V2 paper keeps tube masking in its encoder to avoid leaking information across frames [arxiv:2303.16727 §3.2]. The SatMAE temporal ablation favours per-image masking, 79.90 top-1 against 78.45 for consistent masking, but those two rows also differ in cropping [arxiv:2207.08051 §5.3]. The data differ as well, so neither paper tests the other, yet they disagree on whether the same locations should be hidden across frames. VideoMAE's ablation points the other way: on SSv2, tube masking at 90% scores 69.6 against 68.3 for independent random cube masking [arxiv:2203.12602 tab. 1]. Random cube masking is not per-image masking, so this is a near miss rather than a direct test of the SatMAE result.
- **Strong augmentation in contrastive learning.** The 2020 framework review says many contrastive methods do poorly without crop and colour distortion [arxiv:2010.05113 §II.C]. CLSA reports that feeding strong views straight into the contrastive loss gives 67.0 to 67.7 against 67.5 for the gentle baseline [arxiv:2104.07713 tab. 5]. Its distribution-matching term is what gives the gain, to 69.4 at single crop [arxiv:2104.07713 tab. 2]. The two sources measure different things, so neither contradicts the other. CLSA's own table also lists InfoMin Augmentation at 70.1 for single-crop training, above CLSA's 69.4 [arxiv:2104.07713 tab. 2].

## Papers

- [Self-supervised Learning: Generative or Contrastive](/papers/2006.08218): taxonomy of generative, contrastive and adversarial methods, with the collected losses and theory.
- [Contrastive Representation Learning: A Framework and Review](/papers/2010.05113): unified framework of similarity, encoder, head and loss, with classification tables for language, vision, audio, graphs and multimodal data.
- [Graph Self-Supervised Learning: A Survey](/papers/2103.00111): pretext categories and training schemes for graphs, with compiled node and graph classification accuracies.
- [SimMIM: A Simple Framework for Masked Image Modeling](/papers/2111.09886): masked image modelling with a linear head and masked ℓ1 loss, with ViT and Swin fine-tuning results.
- [iBOT: Image BERT Pre-Training with Online Tokenizer](/papers/2111.07832): masked patch prediction against an online teacher, combined with a [CLS] self-distillation loss.
- [Masked Autoencoders Are Scalable Vision Learners](/papers/2111.06377): asymmetric masked autoencoder for ViTs, 75% masking, encoder without mask tokens, with ImageNet-1K fine-tuning and linear-probe results.
- [Masked Siamese Networks for Label-Efficient Learning](/papers/2204.07141): masked views matched to an EMA teacher's prototype assignments, with low-label ImageNet results.
- [Understanding Self-Supervised Learning Dynamics without Contrastive Pairs](/papers/2102.06810): linear-model analysis of why BYOL and SimSiam avoid collapse, and DirectPred, a gradient-free predictor.
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](/papers/2005.13249): ECG contrastive pre-training with patient-level positives across segments and leads.
- [Text and Code Embeddings by Contrastive Pre-Training](/papers/2201.10005): contrastive pre-training on paired web text and docstring-function pairs, with large in-batch negatives.
- [Deep ViT Features as Dense Visual Descriptors](/papers/2112.05814): DINO ViT keys as zero-shot dense descriptors for co-segmentation and correspondence.
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](/papers/2203.08414): STEGO, a small head on a frozen DINO ViT trained to keep feature correlations, then clustered.
- [Masked Visual Pre-training for Motor Control](/papers/2203.06173): masked-image encoder pre-trained on hand-object images, frozen, and used by a PPO controller on pixel tasks.
- [Test-Time Training with Masked Autoencoders](/papers/2209.07522): per-input masked reconstruction at test time, with ImageNet-C results.
- [Very high resolution canopy height maps from RGB imagery](/papers/2304.07213): frozen self-supervised ViT encoder with a trained dense decoder, with aerial lidar labels.
- [Node Similarity Preserving Graph Convolutional Networks](/papers/2011.09643): SimP-GCN, with a pair-regression self-supervised term added to node classification.
- [Unsupervised Domain Adaptation through Self-Supervision](/papers/1909.11825): rotation, flip and patch-location puzzles trained beside the source task, with no adversarial loss.
- [Learning and Evaluating Representations for Deep One-class Classification](/papers/2011.02578): distribution-augmented contrastive features for one-class detection, with OC-SVM or KDE on top.
- [Unsupervised Learning of Dense Visual Representations](/papers/2011.05499): VADeR, per-pixel contrast with correspondences from known view geometry.
- [HDMI: High-order Deep Multiplex Infomax](/papers/2102.07810): mutual-information node embeddings for multiplex graphs, fused by semantic attention.
- [Transformers in Vision: A Survey](/papers/2101.01169): survey of Transformers in vision, with self-supervised ViTs as one category; numbers are compiled from the cited papers.
- [An Empirical Study of Training Self-Supervised Vision Transformers](/papers/2104.02057): MoCo v3 applied to ViTs, the instability it causes, and a frozen random patch projection that reduces it.
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](/papers/2203.12602): masked video autoencoder with tube masking and a plain ViT, pre-trained without labels or extra data.
- [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](/papers/2107.14293): STraTS, a Transformer over observation triplets, pre-trained by forecasting.
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](/papers/2112.01511): VINN, BYOL-tuned encoder with a softmin nearest-neighbour action lookup.
- [Splicing ViT Features for Semantic Appearance Transfer](/papers/2201.00424): Splice, a per-pair generator trained with losses from a frozen DINO ViT.
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](/papers/2208.11296): survey of semi-supervised and unsupervised visual learning, with a taxonomy by smoothness assumption; no experiments.
- [Emerging Properties in Self-Supervised Vision Transformers](/papers/2104.14294): DINO, self-distillation with centring and sharpening, multi-crop, and k-NN evaluation of ViT features.
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](/papers/2106.14112): TS-TCC, cross-view temporal prediction plus contextual contrast, with linear evaluation on HAR, Sleep-EDF and Epilepsy.
- [On the Opportunities and Risks of Foundation Models](/papers/2108.07258): report defining foundation models as broadly trained, adaptable models; no experiments.
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](/papers/2111.09296): masked-contrast speech pre-training on 436K hours in 128 languages, with 0.3B to 2B models.
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](/papers/2202.03555): one teacher-student regression recipe for speech, images and text, with modality-specific encoders and masks.
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](/papers/2207.08051): masked autoencoder with temporal encodings, per-image masking and band groups for satellite data.
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](/papers/2303.16727): billion-parameter video masked autoencoder with dual encoder and decoder masking, then supervised post-pre-training.
- [TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis](/papers/2506.20380): Barlow Twins over sampled date subsets for per-pixel Sentinel-1 and Sentinel-2 embeddings, with mixup and global shuffling.
- [CoLA: Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning](/papers/2103.00113): node-versus-local-subgraph contrast with sampled scores for unsupervised node anomaly detection.
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](/papers/2106.01342): row and column attention for tables, with optional contrastive and denoising pre-training.
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](/papers/2110.09348): dimensional collapse in contrastive learning, and DirectCLR, a projector-free variant.
- [SLIP: Self-supervision meets Language-Image Pre-training](/papers/2112.12750): one ViT trained on CLIP's image-text loss plus an image-only SimCLR loss.
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](/papers/2212.03191): masked video ViT and video-language encoder, fused on frozen backbones.
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](/papers/2212.03860): frozen DINO features and retrieval to detect diffusion-model copies of training images.
- [Graph Contrastive Learning with Adaptive Augmentation](/papers/2010.14945): two graph views with edges and feature dimensions dropped by centrality, then node-level contrast.
- [A Survey on Contrastive Self-supervised Learning](/papers/2011.00362): survey of contrastive pretext tasks and the four sources of negatives, with reported benchmark tables; no experiments.
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](/papers/2401.06209): MMVP benchmark of CLIP-blind image pairs, and interleaved vision-only tokens added to LLaVA.
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](/papers/1910.12240): self-supervised keypoint matching and closed-form pose for partial point clouds, trained on synthetic motions.
- [Context Autoencoder for Self-Supervised Representation Learning](/papers/2202.03026): masked-image pre-training with an encoder-side regressor and a separate decoder, with ImageNet, ADE20K and COCO results.
- [Deep transfer learning for image classification: a survey](/papers/2205.09904): survey of transfer learning for image classification, with a source ranking that includes self-supervised pre-training; one supervised experiment.
- [Emergent Correspondence from Image Diffusion](/papers/2306.03881): frozen diffusion U-Net features with nearest-neighbour matching for correspondence, with no training.
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](/papers/2101.12037): masked-span contrast over convolutional EEG features with an 8-layer Transformer, pre-trained on TUEG.
- [SSD: A Unified Framework for Self-Supervised Outlier Detection](/papers/2103.12051): SimCLR-style features trained on in-distribution images only, scored by Mahalanobis distance for outlier detection.
- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](/papers/2210.06551): masked 2D-to-3D lifting pre-training on skeleton sequences, with mocap and in-the-wild video, then fine-tuned heads.
- [Learning Self-Consistency for Deepfake Detection](/papers/2012.09311): patch-pair consistency loss with generated face forgeries whose masks come for free.
- [Self-Supervised Speech Representation Learning: A Review](/papers/2205.10643): survey of generative, contrastive and predictive speech pre-training; no new method.
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](/papers/2206.09113): masked patch pre-training of a frozen Transformer that feeds a graph forecaster.
- [Multi-task Self-Supervised Learning for Human Activity Detection](/papers/1907.11879): predicts which signal transform was applied to accelerometer windows, then reuses the conv features for activity classification.
- [Contrastive Learning with Stronger Augmentations](/papers/2104.07713): CLSA, a divergence term that matches strong-view similarity profiles to weak-view ones over a key queue.
- [End-to-end Jordanian dialect speech-to-text self-supervised learning framework](/papers/2609.24410): wav2vec 2.0-style pre-training and five noisy-student rounds on a Jordanian Arabic speech set.