Self-supervised Learning: Generative or Contrastive

In plain words

Labelling data by hand is slow and expensive, yet the internet is full of unlabelled text, images and graphs. Self-supervised learning gets its training signal from the data itself: hide part of the input and ask the model to fill it in, or show two altered copies of the same photo and ask the model to recognise that they belong together. The network learns useful features this way, and those features are later reused for real tasks such as classification with only a few labels.

This survey sorts the many self-supervised methods by what they are trained to do. Generative methods rebuild the input, as in "fill in the missing word" (BERT) or "predict the next word" (GPT). Contrastive methods never rebuild anything. They learn to tell matching pairs from non-matching ones, and the survey places MoCo, SimCLR and BYOL here. Generative-contrastive (adversarial) methods put a generator against a judge that tries to spot fakes, as in GANs and ELECTRA.

For each family it writes out the training losses, says what each one is good and bad at, and covers the theory of why matching pairs should produce good features. It ends with open problems: why augmentations help, what negative examples really contribute, and why features learned for one task sometimes fail to transfer. For an engineer it works as a map of the field around 2020–2021, with the key loss functions in one place.

TL;DR

A taxonomy survey, not a method paper. Every self-supervised method is placed in one of three groups by its objective: generative (encoder + decoder, reconstruction / likelihood loss), contrastive (encoder + small discriminator, similarity loss such as InfoNCE), and generative-contrastive / adversarial (encoder-decoder generator + discriminator, distribution-divergence loss) §2. Contrastive work is split further into context-instance (relative position, mutual-information maximisation) and instance-instance (cluster discrimination, instance discrimination) §4. The useful parts for an implementer are the collected loss equations (AR, flow, VQ-VAE, PLM, NCE/InfoNCE, DIM, MoCo, NT-Xent, BYOL, GAN, AAE, ELECTRA, alignment/uniformity) and tab. 1, which tags 50-odd methods by field, generator type, pretext task, hard negative/positive sampling and negative-sampling strategy. Several equations carry typos (see Pitfalls). arXiv v5 (March 2021). No code, no experiments.

Problem

Supervised deep learning needs costly manual labels and suffers from poor generalisation, spurious correlations and adversarial vulnerability §1. The authors' cost example: a labelling company charging $6.4 per image for segmentation, so a 10k+ image segmentation dataset could cost up to a million dollars §2. Self-supervised learning instead derives "labels" from the data via a semi-automatic process and predicts parts of the data from other parts, where the given part may be incomplete, transformed, distorted or corrupted §1. The survey separates it from unsupervised learning in the narrow sense (clustering, community detection, anomaly detection): self-supervised learning recovers parts of the input and so stays within the supervised paradigm §1 fig. 1. Existing reviews covered pre-trained LMs, GANs, autoencoders or visual contrastive learning separately. This one aims to cover self-supervision itself across CV, NLP and graphs §1.

Method

The "method" of a survey is its taxonomy. Section numbering of the compiled paper: §1 Introduction, §2 Motivation (taxonomy), §3 Generative, §4 Contrastive, §5 Generative-Contrastive (Adversarial), §6 Theory, §7 Discussions and Future Directions, §8 Conclusion. There is no appendix. A "Background" file in the source (word embeddings, ResNet, graph basics) is not included in the compiled paper. The single table tab. 1 is the method overview. Equations are numbered (1)–(37) in order across §3–§6, as listed below.

Taxonomy §2 fig. 3 fig. 4

All three families share a generator (encoder, optionally decoder) and differ in three ways §2:

The authors argue that contrastive objectives model invariance between instances of the same class, which matches classification-style downstream tasks §2.

Generative self-supervised learning §3

Auto-regressive (AR) §3.1. Factorise the joint into left-to-right conditionals:

\max_\theta p_\theta(\mathbf{x}) = \sum_{t=1}^{T} \log p_\theta(x_t \mid \mathbf{x}_{1:t-1}) eq. 1

Covered: GPT/GPT-2 (Transformer decoder; GPT-2 models p(\text{output}\mid\text{input},\text{task}) and drops task-specific fine-tuning), PixelRNN/PixelCNN (pixel by pixel, masked filters, two combined conv nets to remove the blind spot), WaveNet (dilated causal convolutions, gated residual blocks, skip connections), GraphRNN (graph-level RNN creates nodes, edge-level RNN creates edges), MRNN and GCPN (RL with domain rewards for molecules; RNN vs GCN state encoders) §3.1. Strength: models context dependency well. Weakness: each position only sees one direction §3.1.

Flow-based §3.2. Learn an invertible, differentiable z = f_\theta(x) with known prior p_Z:

p_\theta(x) = p(f_\theta(x)) \left| \frac{\partial f_\theta(x)}{\partial x} \right| eq. 2

\max_\theta \sum_i \log p_\theta(x^{(i)}) = \max_\theta \sum_i \log p_Z(f_\theta(x^{(i)})) + \log \left| \frac{\partial f_\theta}{\partial x}(x^{(i)}) \right| eq. 3

x and z must have equal dimension, and the Jacobian determinant must be cheap §3.2. NICE/RealNVP split x=(x_1,x_2) and set z_1=x_1, z_2 = x_2 + m(x_1); Glow adds invertible 1\times1 convolutions §3.2.

Auto-encoding (AE) §3.3: reconstruct (part of) the input from a (corrupted) input.

• Basic AE §3.3.1: h=f_{enc}(x), x'=f_{dec}(h), minimise e.g. MSE between x and x'. A linear AE corresponds to PCA. Overcomplete hidden layers need sparsity constraints. RBM is presented as an undirected precursor.
• Context prediction (CPM) §3.3.2: CBOW (predict the token from its context), Skip-Gram (predict the context from the token), usually with negative sampling. FastText adds subwords. DeepWalk runs Skip-Gram on truncated random walks. LINE predicts neighbours:

O = -\sum_{(i,j)\in E} w_{ij} \log p(v_j \mid v_i) eq. 4

with w_{ij} the edge weight, approximated with negative edge sampling §3.3.2.

• Denoising AE §3.3.3: masked language modelling is treated as denoising AE. BERT masks tokens with [MASK], and to reduce the pre-train/fine-tune mismatch it sometimes keeps the original word or substitutes a random one, each with a small probability (exact rates not given in the survey). SpanBERT masks contiguous spans and predicts them from span-boundary representations. ERNIE (Baidu) masks entities/phrases. ERNIE (Tsinghua) injects KG entities and relations. GPT-GNN masks edges and attributes and asks the GNN to generate them. Weakness: MLM assumes masked tokens are independent given the unmasked ones §3.3.3.
• Variational AE §3.3.4: maximise the ELBO,

\log p(x) \ge -D_{KL}(q(z\mid x)\,\|\,p(z)) + \mathbb{E}_{\sim q(z\mid x)}[\log p(x\mid z)] eq. 5

VAE: Gaussian prior \mathcal N(0,1) and Gaussian posterior, reparameterised as z=\mu+\sigma\epsilon, \epsilon\sim\mathcal N(0,1), with \mu,\sigma from neural nets. VQ-VAE replaces the continuous latent with a nearest-neighbour lookup in a learnable codebook and passes gradients with a straight-through estimator:

\mathcal{L}(x, D(e)) = \|x - D(e)\|_2^2 + \|sg[E(x)] - e\|_2^2 + \beta\,\|sg[e] - E(x)\|_2^2 eq. 6

with sg = stop-gradient and \beta controlling how reluctant the encoder output is to change its code §3.3.4. VQ-VAE-2 adds a PixelCNN prior and a multi-scale hierarchy. Graph variants: VGAE (GCN encoder, reconstructs the adjacency matrix), DVNE (Gaussian node embeddings compared with the 2-Wasserstein distance), vGraph (joint node embedding and community detection) §3.3.4.

Hybrid generative §3.4. MADE masks AE weights to respect an AR ordering §3.4.1. XLNet's permutation LM maximises the expected likelihood over factorisation orders, with \mathcal Z_T the set of permutations of [1..T]:

\max_\theta \mathbb{E}_{\mathbf{z}\sim\mathcal{Z}_T}\left[\sum_{t=1}^{T} \log p_\theta(x_{z_t} \mid \mathbf{x}_{\mathbf{z}_{<t}})\right] eq. 7

plus position-aware reparameterisation, two-stream self-attention, and Transformer-XL's segment recurrence and relative encoding §3.4.1 fig. 6. GraphAF combines flow and AR for molecule generation, with valency checks and dequantisation of discrete node/edge types §3.4.2.

Pros and cons §3.5. Pro: models the data distribution without assumptions about the downstream task, and all generation tasks rely on it. Cons: (1) less competitive than contrastive methods on some classification benchmarks in CV, though NLP still relies on generative LMs for text classification; (2) the point-wise MLE objective \mathcal L_{MLE} = -\sum_x \log p(x\mid c) is sensitive and conservative (the loss explodes as p(x\mid c)\to 0, so rare samples dominate) and low-level (it is modelled at the level of pixels, words or nodes, while classification needs high-level abstraction) §3.5.

Contrastive self-supervised learning §4

Base objectives, with f the encoder, x^+ similar and x^- / x^k dissimilar to x §4:

\mathcal{L} = \mathbb{E}_{x,x^+,x^-}\left[-\log \frac{e^{f(x)^T f(x^+)}}{e^{f(x)^T f(x^+)} + e^{f(x)^T f(x^-)}}\right] eq. 8 (NCE)

\mathcal{L} = \mathbb{E}_{x,x^+,x^k}\left[-\log \frac{e^{f(x)^T f(x^+)}}{e^{f(x)^T f(x^+)} + \sum_{k=1}^{K} e^{f(x)^T f(x^k)}}\right] eq. 9 (InfoNCE)

Context-instance (global-local) contrast §4.1: relate a local feature to its global context.

• Predict relative position (PRP) §4.1.1 fig. 8: relative patch position, jigsaw, rotation prediction. In NLP: BERT's next-sentence prediction (NSP). RoBERTa drops it. ALBERT replaces it with sentence-order prediction (SOP), whose negatives are the same two sentences swapped, because NSP negatives from other documents reduce the task to topic detection. PIRL uses jigsaw only to build positives.
• Maximise mutual information §4.1.2:

\max_{g_1\in\mathcal{G}_1,\, g_2\in\mathcal{G}_1} I(g_1(x_1), g_2(x_2)) eq. 10

In practice a lower bound is maximised with an NCE objective. Deep InfoMax: encode x to f(x)\in\mathbb R^{M\times M\times d}, take a local vector v\in\mathbb R^d, a summary s=g(f(x))\in\mathbb R^d and a negative summary s^-=g(f(x^-)) from another image §4.1.2 fig. 9:

\mathcal{L} = \mathbb{E}_{v,x}\left[-\log \frac{e^{v^T s}}{e^{v^T s} + e^{v^T s^-}}\right] eq. 11

Followers: CPC (audio and later images, several negatives at once), AMDIM (local and context vectors from two different augmented views), InfoWord (sentence vs. n-grams; context built by masking the n-grams; negatives from the corpus), DGI (node vs. 2-hop neighbourhood summary; negatives by permuting node features while keeping structure) fig. 10, InfoGraph (graph-level vs. substructures), contrastive multi-view graph learning (graph diffusion is reported as the best way to make positive pairs), Hu et al.'s graph pre-training strategies (Context Prediction; Attribute Mask), S²GRL (k-hop context subgraphs) §4.1.2.

Instance-instance contrast §4.2. Motivated by Tschannen et al.'s finding that MI-based success is only loosely tied to MI and owes more to the encoder and negative sampling §4.2.

• Cluster discrimination §4.2.1 fig. 11. DeepCluster alternates (1) K-means on encodings to make pseudo-labels and (2) training to predict them. Local Aggregation uses per-example neighbours and a soft-clustering objective. ClusterFit inserts a cluster-prediction fine-tuning stage between pre-training and evaluation. SwAV does online clustering with multi-view swapped prediction: views of one image should get the same prototype assignment ("codes"). SEER scales SwAV to 1.3B parameters on 1B Instagram images. M3S uses DeepCluster-style pseudo-labels and adds only the top-k confident ones per stage in semi-supervised graph learning.
• Instance discrimination §4.2.2. InstDisc is the prototype. CMC treats several views as positives. MoCo uses a query encoder f_q, an asynchronously (momentum) updated key encoder f_k, and a queue of past keys as negatives:

\mathcal{L} = -\log \frac{\exp(q\cdot k_+/\tau)}{\sum_{i=0}^{K}\exp(q\cdot k_i/\tau)} eq. 12

with K the number of negatives, queue size up to 65536, batch shuffling against trivial solutions, and temperature \tau §4.2.2 fig. 12. SimCLR: end-to-end, strong augmentations fig. 13, large batch. From a batch of N, make 2N views. Each positive pair has 2(N-1) negatives. Use cosine similarity:

l_{i,j} = -\log \frac{\exp(\mathrm{sim}(\hat x_i, \hat x_j)/\tau)}{\sum_{k=1}^{2N} \mathbb{I}_{[k\ne i]} \exp(\mathrm{sim}(\hat x_i, \hat x_k)/\tau)} eq. 13

\mathcal{L} = \frac{1}{2N}\sum_{k=1}^{N}\left[l_{2i-1,2i} + l_{2i,2i-1}\right] eq. 14

plus a learnable nonlinear projection before the loss, more training steps, and deeper networks. MoCo v2 shows these SimCLR tricks also help MoCo. InfoMin picks views that share minimal information (ideally only label information). BYOL drops negatives: an online network with an extra predictor q_\theta regresses an EMA target network's projection fig. 14:

\mathcal{L}^{\rm BYOL}_\theta \triangleq \left\|\bar{q}_\theta(z_\theta) - \bar{z}'_\xi\right\|_2^2 = 2 - 2\cdot\frac{\langle q_\theta(z_\theta), z'_\xi\rangle}{\|q_\theta(z_\theta)\|_2 \cdot \|z'_\xi\|_2} eq. 15

(bars denote \ell_2-normalised vectors). SimSiam identifies stop-gradient as the critical component of BYOL. ReLIC adds a KL regulariser between the prediction logits of different views. GCC (graph): two random-walk-with-restart subgraphs per node, top eigenvectors of the normalised Laplacian as initial features, GNN encoder, InfoNCE. GraphCL: edge-perturbation and node-dropping augmentations and their combinations §4.2.2.

Pre-training + semi-supervised self-training §4.3. Self-training: train on the labelled data, pseudo-label the unlabelled data, keep only confident labels, retrain, iterate. Noisy-student-style training (un-noised teacher; student noised with dropout, stochastic depth and RandAugment) on 300M unlabelled images is cited as the supervised ImageNet state of the art at the time. Zoph et al. (as summarised here): MoCo pre-training can hurt under strong augmentation and its value shrinks with more labels, while self-training keeps helping; the two gains are orthogonal and combining them is best. SimCLR v2 recipe §4.3:

SimCLR v2 semi-supervised recipe  [§4.3]
1. Self-supervised pre-training as in SimCLR v1, with minor architecture changes and a deeper ResNet.
2. Fine-tune the last few layers on 1% or 10% of the ImageNet labels.
3. Use the fine-tuned network as teacher to label unlabelled data; train a smaller student ResNet-50 on it.

Pros and cons §4.4. Encoder only, no decoder, so lightweight and strong on discriminative tasks. Open: (1) contrastive pre-training for NLP is unconvincing so far, and NLP contrastive work mostly sits in fine-tuning (sentence embeddings, dense retrieval); (2) negative sampling is tricky, biased and slow, and its role is unclear; (3) why augmentation helps has no clear theory, which hinders discrete domains (NLP, graphs) §4.4.

Generative-contrastive (adversarial) §5

Keeps the decoder, so representations must be "reconstructive" rather than just "distinguishable". This is more expressive, but harder to optimise and less stable than contrastive learning §5.

• Generate with complete input §5.1:

\min_G \max_D\; \mathbb{E}_{x\sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z\sim p_z(z)}[\log(1 - D(G(z)))] eq. 16

GAN's latent is implicit and has to be extracted. AAE replaces the VAE KL term

\mathcal{L}_{\rm VAE} = -\mathbb{E}_{q(z\mid x)}(-\log(p(x\mid z)) + \mathrm{KL}(q(z\mid x)\,\|\,p(z)) eq. 17 (as printed)

with a discriminator loss between encoder outputs and prior samples:

\mathcal{L}_{Disc} = \mathrm{CrossEntropy}(q(z), p(z)) eq. 18

BiGAN/ALI drop reconstruction entirely. The generator G maps prior z (e.g. uniform(-1,1)^d) to x'=G(z), the encoder E maps x to z'=E(x), and D tells [z, G(z)] from [E(x), x]. Training drives E \to G^{-1}. BigBiGAN scales this up §5.1.

• Recover with partial input §5.2 fig. 15: colorization (predict the A,B channels from L; Zhang et al. quantise (A,B) into 313 classes and use a softmax with temperature T), inpainting with a discriminator, and SRGAN super-resolution.
• Pre-trained LMs §5.3 fig. 16. ELECTRA's replaced-token detection: a small MLM generator G fills masked positions, and discriminator D labels each token as replaced or original ("replaced" = differs from the original token). The survey describes two stages:

ELECTRA training as described by the survey  [§5.3]
1. Warm up G with the MLM loss L_MLM(x, θ_G) for some steps.
2. Initialise D's parameters from G's; train D with the discriminative cross-entropy loss
   L_Disc(x, θ_D) while G is frozen.

\min_{\theta_G,\theta_D} \sum_{x\in\mathcal{X}} \mathcal{L}_{\rm MLM}(x,\theta_G) + \lambda\,\mathcal{L}_{\rm Disc}(x,\theta_D) eq. 19

It is not trained as a GAN because discrete tokens block gradients. The survey reports that ELECTRA's policy-gradient variant did slightly worse, and it argues that the binary RTD loss saves compute but may cause early degeneration of the embedding space. WKLM applies replacement detection to entities. REALM is mentioned but uses no discriminative objective §5.3.

• Graphs §5.4: ANE (Skip-gram-style NCE + discriminator against prior samples), GraphGAN (generator selects nodes and is updated by policy gradient; D classifies links), GraphSGAN (generates fake nodes in density gaps between clusters; D has an extra "fake" class that is removed at test time).
• Domain adaptation / multi-modality §5.5: listed by reference only.
• Pros and cons §5.6: weaker on discrete data (NLP, graphs), and GAN text generation trails unidirectional LMs; prone to collapse (spectral norm, WGAN as fixes); contrastive learning outperformed BiGAN/BigBiGAN for feature extraction with fewer parameters.

Theory §6

• GAN as divergence matching §6.1.1. MLE ≈ minimising \mathrm{KL}(P_{data}\|P_G) eq. 20. f-GAN's variational form \mathcal D_f(P\|Q) = \mathbb E_q[f(p/q)] = \max_T(\mathbb E_p[T(x)] - \mathbb E_q[g(T(x))]) eq. 21 with JS divergence gives the min-max GAN objective eq. 22. Different divergences give different GAN variants.
• Disentanglement §6.1.2. InfoGAN maximises a variational lower bound L_I on I(c; G(z,c)) for a latent code c: \min_G\max_D V_I(D,G) = V(D,G) - \lambda L_I(c; G(z,c)) eq. 23. GAN dissection and counterfactual interventions are cited for disentangled object/background units in BigGAN fig. 18.
• ELBO §6.2.1: \mathrm{KL}(q_\phi(z\mid x)\|p_\theta(z\mid x)) = -\mathrm{ELBO} + \log p_\theta(x) eq. 24. The paper's eq. 25 is a second ELBO expression (sign issue, see Pitfalls). Its decomposition into KL regulariser and reconstruction term is eq. 26.
• Mutual information §6.2.2: I(X,Y) = \mathrm{KL}(p(x,y)\|p(x)p(y)) eq. 27. DIM with JS divergence: \max_T \mathbb E_{p(x,y)}[\log T(x,y)] + \mathbb E_{p(x)p(y)}[\log(1-T(x,y))] eq. 28, the same form as the GAN objective. The InfoNCE derivation eq. 29 ends in

I(x,y) \ge \log N - \mathcal{L} §6.2.2

so the bound tightens with more negatives N. The survey immediately notes Arora et al.'s result that more negatives do not necessarily help. MI and ELBO are only loosely tied to downstream performance: looser bounds can give better accuracy, and equal bound values can give very different representations §6.2.2. InfoNCE rewritten as \log k - \mathbb E[\frac1k\sum_i \log(1+\sum_{j\ne i} e^{f(x_i,y_j)-f(x_i,y_i)})] eq. 30. With f(x,y)=\phi(x)^T\phi(y) it equals the expected multi-class k-pair metric-learning loss eq. 31.

• Relation to supervised learning §6.3.1 fig. 19. Newell et al. (synthetic COCO): pre-training helps with few labels, but converges to the from-scratch baseline before accuracy plateaus. Hendrycks et al.: self-supervision improves robustness to adversarial examples, label corruption and input corruptions, and helps near-distribution OOD detection.
• Alignment and uniformity §6.3.2. Wang & Isola's split of the contrastive loss:

\mathcal{L}_{\rm contrast} = \underbrace{\mathbb{E}[-f_x^T f_y/\tau]}_{\rm alignment} + \underbrace{\mathbb{E}\left[\log\left(e^{f_x^T f_y/\tau} + \textstyle\sum_i e^{f_x^T f_{y_i^-}/\tau}\right)\right]}_{\rm uniformity} eq. 32

\mathcal{L}_{\rm align}(f;\alpha) \triangleq \mathbb{E}_{(x,y)\sim p_{\rm pos}}\left[\|f(x)-f(y)\|_2^\alpha\right], \quad \mathcal{L}_{\rm uniform}(f;t) \triangleq \log \mathbb{E}_{x,y\sim p_{\rm data}}\left[e^{-t\|f(x)-f(y)\|_2^2}\right] eq. 33

Optimised as a weighted sum. Reported: direct optimisation beats the contrastive loss across the CNN/RNN, CV/NLP settings tried; both terms are needed; and an overweighted term causes collapse. BYOL is offered as evidence that uniformity can come from other mechanisms (EMA, BN, regularisation, random init) §6.3.2.

• Generalisation (Arora et al.) §6.3.3. Unsupervised loss \mathcal L_{un}(f) = \mathbb E[l(f(x)^T(f(x^+)-f(x^-)))] eq. 34. With one negative and probability \ge 1-\delta: \mathcal L_{sup}(\hat f) \le \mathcal L^\mu_{sup}(\hat f) \le \mathcal L^{\ne}_{un}(f) + \beta s(f) + \eta\,Gen_M eq. 35, with s(f) the intra-class deviation. Under sub-Gaussian class embeddings with max norm R: eq. 36, \gamma = 1 + c' R\sigma\sqrt{\log(R/\epsilon)}. Extension to several negatives eq. 37. Also from Arora et al.: beyond some threshold, more negatives can hurt §6.3.3.

Open problems §7

Theoretical foundations. The pre-training/downstream gap: pretext-task choice is heuristic (BERT's NSP vs ALBERT), and the authors suggest NAS-style automatic pretext design. Transfer across datasets where embedding spaces differ (graphs, MT). Sampling strategies. "Early degeneration": contrastive and RTD objectives may overfit the discriminative pretext task early, which the authors offer as a possible reason for ELECTRA's reportedly weaker generation and entity-extraction results §7.

Overview table tab. 1

FOS = field of study. Type: G generative, C contrastive, G-C generative-contrastive. Hard NS / Hard PS: yes = adopted, no = not adopted, – = not applicable, "no NS" = no negatives. Generator: AR, AE, Flow. Merged cells of the original are repeated per row.

Hyperparameters

The survey has no experiments and no hyperparameters of its own. Values it quotes from the surveyed methods (check each against the original paper before relying on it):

Results

No experiments of its own. Numbers below are reported by the survey from other papers, i.e. second-hand author-reported results:

fig. 7 plots ImageNet top-1 linear-probe accuracy of self-supervised methods as of March 2021 against a supervised ResNet-50. The caption says the gap is closing fast and that every method in the plot except BigBiGAN is contrastive. The individual values appear only in the figure, not in the text.

Paper vs code

No official code.

Pitfalls

• Equation typos. Do not implement the formulas verbatim:
  • eq. 1 writes \max_\theta p_\theta(\mathbf x) = \sum_t \log p_\theta(\cdot); the left side should be a log-likelihood.
  • eq. 10 constrains both encoders to \mathcal G_1, although the text defines a separate class \mathcal G_i per encoder.
  • eq. 12 sums i=0..K in the denominator while K is called "the number of negative samples". So the denominator has K+1 terms and k_0 must be the positive k_+; the survey never says so.
  • eq. 14 sums over k but indexes l_{2i-1,2i} with i. Read it as a sum over i=1..N of both directions of each positive pair.
  • eq. 17 has unbalanced parentheses and a double negative, so as printed the sign of the reconstruction term is ambiguous. Use eq. 5 (maximise ELBO) as the reference.
  • eq. 25 writes ELBO as E_q[\log q] - E_p[\log p(z,x)], the negative of the form consistent with eq. 24 and eq. 26. The KL in eq. 26 also has misplaced subscripts.
  • eq. 29 mixes K (sum limit), N (final bound) and (N-1), and sums from i=0 while adding the positive term separately.
• Flow coupling. The text calls NICE/RealNVP layers "affine coupling" but gives only the additive form z_2 = x_2 + m(x_1), with no scale term §3.2. Do not take this as RealNVP's full layer.
• SimCLR batch size "8196" §4.2.2 is not a power of two and may be a typo. Take the value from the SimCLR paper itself.
• MoCo's positive pair. The survey says MoCo's positives come from the same sample "without any transformation or augmentation" §4.2.2. This is the survey's characterisation. Check MoCo's augmentation pipeline in the original paper before implementing.
• ELECTRA description is internally inconsistent. The two-stage procedure freezes G while D trains and initialises D from G §5.3, yet eq. 19 minimises both losses jointly over \theta_G,\theta_D. The survey also gives no masking rate, generator size or \lambda. Use an ELECTRA source for the actual schedule.
• BERT masking rates (how often a selected token is kept or randomised) are described only as "a small probability" §3.3.3.
• Table 1 labels are the authors' judgement. "Hard NS/PS" and "NS strategy" are categorical tags, not measured properties. Some groupings are debatable: CMC is listed as instance-instance in tab. 1 and §4.2.2, but its graph counterpart (CMC-Graph) appears under context-instance MI maximisation.
• Theory results hold only under narrow assumptions. The Arora et al. bounds eq. 35 eq. 36 assume one negative per pair, latent classes and sub-Gaussian embeddings. The InfoNCE bound eq. 29 suggests more negatives are better, while the survey separately reports that more negatives can hurt past a threshold §6.2.2 §6.3.3. Neither shows that a larger MI bound gives better features: the survey stresses that MI and ELBO correlate only loosely with downstream accuracy §6.2.2.
• Snapshot in time. v5 dates from March 2021 fig. 7. Claims such as "contrastive pre-training is unconvincing in NLP" §4.4 and "state-of-the-art" labels reflect that date.
• Author order differs between the arXiv listing (Wang before Mian, used in this card's frontmatter) and the PDF title block (Mian before Wang).
• Several claims are cited only loosely: the LeCun quotes §1 §5, the Scale.ai price §2, and "the early degeneration" hypothesis §5.3 §7, which is the authors' conjecture with no experiment behind it.

Lineage

• Builds on: earlier single-area reviews of pre-trained LMs, GANs, autoencoders and visual self-supervised learning §1. This survey merges them into one generative / contrastive / adversarial taxonomy.
• Covers, among others: XLNet (permutation LM, §3.4.1), BERT, GPT/GPT-2, MoCo, SimCLR, BYOL, SwAV, DGI, GCC, ELECTRA, BiGAN.
• Compared against: not applicable (survey).
• Superseded by: not known from content in this wiki.