Contrastive Representation Learning: A Framework and Review

In plain words

Labelling data is slow and expensive, but you often know which examples belong together without knowing what they are: two crops of the same photo, a video frame and the sound playing at that moment, a sentence and the sentence after it. Contrastive learning turns that knowledge into a training signal. A network maps each input to a vector, and training pulls the vectors of "matching" pairs together while pushing apart vectors of pairs that don't match.

This review shows that a long list of methods with different names (Siamese networks, triplet loss, word2vec's negative sampling, CPC, MoCo, SimCLR, BYOL and more) are all the same recipe with four interchangeable parts: how you decide what counts as similar, which encoder turns inputs into vectors, an optional small "head" network on top, and the loss that scores the pairs. Once you see the parts, you can read a new paper by asking which part it changed.

The most practical lesson is that the choice of what counts as "similar" decides what the model learns to ignore. If two random crops and colour changes of the same image count as a match, the model learns to ignore crop position and colour. The review has no experiments of its own; it is a map of the field up to late 2020, with tables classifying each method by those four parts.

TL;DR

A review (IEEE Access, v2 October 2020) with no experiments or code. Its contribution is the Contrastive Representation Learning (CRL) framework §III.A fig. 3: (1) a similarity distribution p^+(\mathbf q,\mathbf k) and dissimilarity distribution p^-(\mathbf q,\mathbf k) that produce positive and negative keys for a query, (2) encoders, (3) transform heads, (4) a contrastive loss (scoring function + loss form). Each component gets a taxonomy §III.B–§III.E, and roughly 50 methods are classified along it in tab. 2–tab. 6 (language, vision, audio, graphs, multimodal). Read it for the unified notation, the derivation chain from parametric softmax to NCE to InfoNCE/NT-Xent eq. 4–eq. 11, and the method tables; go to the primary papers for any training recipe, since the review gives no hyperparameter values. Beware a sign error in the binary-NCE sigmoid and citation mix-ups in tab. 2 (see Pitfalls).

Problem

Supervised deep learning needs labelled pairs (x, y), which are expensive, can raise privacy issues when labelling is outsourced, and can carry annotator bias §II.A.2. Generative representation learning models p(\mathbf x) in input space, which the authors call computationally expensive and wasteful when only an encoder is needed §II.A.1. Contrastive methods are discriminative but need only a definition of similar and dissimilar pairs, not a label per sample §II.B. The authors argue the field spans metric learning, NLP, vision, audio and RL with no common framework, which makes the novelty of new methods hard to judge §I, and that contrastive learning is wrongly seen as just one more self-supervised technique §I.

Method

The "method" of a review is its framework and taxonomy. Section numbering in the compiled paper uses Roman numerals with lettered subsections ("III-B2"); this card writes them as §III.B.2. Sections: I Introduction; II What is contrastive learning (A representation learning, B contrastive representation learning, C instance discrimination example); III Taxonomy (A framework, B similarity, C encoders, D transform heads, E losses); IV Development; V Applications (A language, B vision, C audio, D graphs, E multimodal, F others); VI Discussion; VII Conclusion. No appendix. Equations 1–11 are numbered in order. Tables: tab. 1 history, tab. 2 language, tab. 3 vision, tab. 4 audio, tab. 5 graphs, tab. 6 multimodal. No algorithm blocks.

Worked example: instance discrimination §II.C fig. 2

Instance discrimination (SimCLR-style, as described in the review)  [§II.C]
Input: unlabelled images; transformation family T (random crop, resize, blur,
       colour distortion, perspective distortion, ...)
1. For each image x in the minibatch, sample t, t' ~ T independently.
   x_q = t(x), x_k = t'(x)                 # same image -> positive pair
   views of different images                 -> negative pairs
2. v = e(x)          # encoder; usually a ResNet, v = output of average pooling, v in R^d
3. z = h(v)          # projection head: small MLP (can be one layer + non-linearity),
                     # z in R^{d'}, d' < d
4. normalise every z to unit length
5. loss over the batch of pairs {(z_i, z'_i)}: eq. 1
6. after training: discard h, use e as feature extractor, add a classifier,
   fine-tune on the target task

\mathcal{L}_i = -\log \frac{\exp(\mathbf z_i^\top \mathbf z'_i/\tau)}{\sum_{j=0}^{K}\exp(\mathbf z_i \cdot \mathbf z'_j/\tau)} eq. 1

\tau is a temperature; the denominator runs over one positive and K negatives from the same minibatch, which makes the loss a non-parametric (K+1)-way softmax classification of \mathbf z_i to \mathbf z'_i §II.C. Citing SimCLR, the review stresses that many contrastive methods do very poorly without proper augmentation (random crop and colour distortion), even with the same architecture and loss §II.C.

The CRL framework §III.A fig. 3

Definitions, using MoCo's query/key dictionary metaphor §III.A:

1. Query, key (Def. 1): views of an input \mathbf x\in\mathcal X; (\mathbf q,\mathbf k) is a positive or negative pair. Symbols \mathbf x, \mathbf v, \mathbf z get superscripts ^q, ^k when needed.
2. Similarity distribution (Def. 2): a joint distribution p^+(\mathbf q,\mathbf k) over pairs; negatives come from p^-(\mathbf q,\mathbf k), which may be implicit ("any pair not drawn from p^+"). In practice sample \mathbf q\sim p(\mathbf x), then \mathbf k^+\sim p^+(\cdot|\mathbf q) and \mathbf k^-\sim p^-(\cdot|\mathbf q). For instance discrimination, \mathbf q,\mathbf k\sim p^+ iff \mathbf q=t(\mathbf x), \mathbf k=t'(\mathbf x).
3. Model (Def. 3): f(x;\theta):\mathcal X\to\mathbb R^{|\mathcal Z|}, decomposed into
4. Encoder (Def. 4): e(\mathbf x;\theta_e):\mathcal X\to\mathcal V, \mathbf v\in\mathbb R^d; this is what is reused downstream.
5. Transform head (Def. 5): h(\mathbf v;\theta_h):\mathcal V\to\mathcal Z, \mathbf z\in\mathbb R^{d'}.
6. Contrastive loss (Def. 6): operates on \{(z,z^+),(z,z^-)\}; small positive distance makes the representation invariant to the differences inside positive pairs, large negative distance makes it covariant with what separates negatives.

fig. 4 frames this as information: what \mathbf q and \mathbf k^+ do not share becomes invariant; what \mathbf k^- does not share with them must be kept as covariant signal.

Taxonomy of similarity §III.B

Whether a method is supervised or self-supervised depends only on whether labels enter p^+ (e.g. k\sim p(\cdot|q,y)) §III.B. Five ways to build positive pairs:

1. Multisensory signals §III.B.1 fig. 5: the same scene recorded by several sensors, same modality (two cameras) or different (audio + video). Example: Time-Contrastive Network pulls together simultaneous views from different cameras and pushes apart frames of the same video far apart in time.
2. Data transformation §III.B.2 fig. 6: hand-crafted, meaning-preserving transforms. Images: lighting/colour distortion, crop and pad, noise and blur, rotation, perspective. Text: back-translation through two MT models (CERT). Code: compiler transforms such as variable renaming, identifier mangling, reformatting, beautification, compression, dead-code insertion/elimination (ContraCode). Audio: warping, frequency and temporal masking of the mel spectrogram.
3. Context-instance relationship §III.B.3 fig. 7: local parts vs a global summary. DIM contrasts a global image vector with spatial patches of the same image (positives) and of other images (negatives); CPC summarises past segments into a context and contrasts it with local features of a future step.
4. Sequential coherence and consistency §III.B.4 fig. 8: by the slowness assumption, nearby elements of a sequence are positives and far-apart elements (or other sequences) negatives; multi-frame TCN uses a time window around the query frame.
5. Natural clustering §III.B.5 fig. 9: contrastive learning shapes the manifold bottom-up through local smoothness, clustering top-down; PCL and SwAV combine them; class labels can serve as clusters (supervised contrastive learning).

Taxonomy of encoders §III.C

Classified by how the contrastive gradient updates them:

1. End-to-end §III.C.1: both query and key encoders get gradients; when modalities match they share weights so one forward pass encodes both. All activations live in GPU memory, which caps the batch size and therefore the number of negatives.
2. Online-offline §III.C.2: keys come from an offline encoder not updated by gradient, so its activations are not stored. Two variants:
   • Past checkpoint / memory bank (Wu et al.): store a detached representation for every sample in the dataset; sample keys from the bank; write the current queries back to it for use as keys in the next epoch. Effectively the offline encoder is the previous epoch's online encoder.
   • Momentum encoder + queue (MoCo): the offline encoder's parameters are an exponentially weighted average of the online encoder's; each iteration, push the newest batch of keys into a queue and drop the oldest. The queue's keys are more consistent than memory-bank entries, which are refreshed only once per epoch.
3. Pre-trained §III.C.3: a frozen encoder, typical in cross-modal learning (e.g. a pre-trained BERT for ASR tokens) and in distillation, where a frozen teacher encodes keys and a small student encodes queries (Contrastive Representation Distillation).

Taxonomy of transform heads §III.D

Separating encoder and head keeps the similarity pretext task from forcing the representation to discard useful information §III.D.

1. Projection §III.D.1: linear or non-linear MLP from \mathbf v to a smaller \mathbf z. Ye et al. used a linear head, SimCLR a non-linear 2-layer MLP. BYOL adds a "prediction" network mapping the online network's embedding to the offline network's.
2. Contextualisation §III.D.2: aggregates several feature vectors: a GRU over past time steps in CPC (an ordered autoregressive head), convolution + pooling in DIM, a graph summary in InfoGraph. The contextual \mathbf z may be used instead of, or together with, \mathbf v downstream.
3. Quantisation §III.D.3: maps many representations to one: Gumbel-softmax into a codebook in wav2vec 2.0; Sinkhorn-Knopp soft cluster assignment in SwAV.

Taxonomy of losses §III.E

A loss = scoring function + form. The review counts as "contrastive" every method that contrasts a query with positive keys, including BYOL-style methods that avoid collapse through architecture instead of explicit negatives §III.E.

Scoring functions §III.E.1:

• Distance: L1 or L2, D(\mathbf q,\mathbf k)=\lVert\mathbf q-\mathbf k\rVert_2, used mainly in margin losses.
• Dot product S(\mathbf q,\mathbf k)=\mathbf q^\top\mathbf k: unbounded and magnitude-dependent; one remedy is a \lVert\mathbf z\rVert^2 normalisation term in the loss (Wu et al.).
• Cosine S(\mathbf q,\mathbf k)=\frac{\mathbf q^\top\mathbf k}{\lVert\mathbf q\rVert\lVert\mathbf k\rVert}\in[-1,1], used in NT-Xent.
• Bilinear S(\mathbf q,\mathbf k)=\mathbf q^\top\mathbf A\mathbf k with learned \mathbf A, used in the original InfoNCE.
• Learned discriminator network: possible, but costs compute, can compensate for a weak encoder, and gives no usable distance; the review focuses on simple scores.

Energy-based margin losses §III.E.2. Pair loss (Chopra, Hadsell, LeCun):

\mathcal L_{\text{pair}} = \begin{cases} D(\mathbf q,\mathbf k)^2, & \text{if } \mathbf k\sim p^+(\cdot|\mathbf q)\\ \max(0,\, m - D(\mathbf q,\mathbf k)^2), & \text{if } k\sim p^-(\cdot|\mathbf q)\end{cases} eq. 2

with margin m>0; only negatives inside the margin contribute. Triplet loss:

\mathcal L(\mathbf q,\mathbf k^+,\mathbf k^-) = \max\!\left(0,\, D(\mathbf q,\mathbf k^+)^2 - D(\mathbf q,\mathbf k^-)^2 + m\right) eq. 3

Both converge slowly because each query sees one comparison (pair) or one positive and one negative (triplet); hard-negative mining is essential; Lifted Embedding loss and its generalisation use many positives and negatives per query in a batch §III.E.2.

Probabilistic NCE-based losses §III.E.3. The derivation chain:

p(i|\mathbf q) = \frac{\exp(\mathbf q^\top\mathbf w_i)}{\sum_{j=1}^{n}\exp(\mathbf q^\top\mathbf w_j)} eq. 4 (parametric softmax; \mathbf w_i is a class prototype, so no comparison between representations)

p(\mathbf k^+|\mathbf q) = \frac{\exp(\mathbf q^\top\mathbf k^+)}{\sum_{\mathbf k\in\mathcal K}\exp(\mathbf q^\top\mathbf k)} = \frac{\exp(\mathbf q^\top\mathbf k^+)}{Z(\mathbf q)} eq. 5 (non-parametric; \mathcal K holds all negatives and one positive)

\mathcal L(\mathbf q,\mathcal K) = -\log p(\mathbf k^+|\mathbf q) eq. 6

Z(\mathbf q) sums over every negative in the dataset, which is expensive; NCE replaces it with a binary task, data vs noise. With uniform noise p^-(\cdot|\mathbf q)=1/n and m noise keys per positive:

p(D=1|\mathbf q,\mathbf k) = \frac{p(\mathbf k^+|\mathbf q)}{p(\mathbf k^+|\mathbf q) + m\cdot p(\mathbf k^-|\mathbf q)} eq. 7

The review then parametrises p(D=1|\mathbf q,\mathbf k)=\frac{1}{1+\exp(S(\mathbf q,\mathbf k))} (sign issue, see Pitfalls) and writes

\mathcal L_{\text{NCE-binary}}(\mathbf q,\mathcal K) = -\mathbb E_{p^+}[\log p(D=1|\mathbf q,\mathbf k)] - \mathbb E_{p^-}[\log(1-p(D=1|\mathbf q,\mathbf k))] eq. 8

used for language models and word embeddings; Negative Sampling (NEG) is a variant aimed only at good embeddings. The global/ranking view (Jozefowicz et al.) asks which of \mathcal K=\{\mathbf k^+,\mathbf k_1^-,\dots,\mathbf k_n^-\} is the positive:

p(i|\mathbf q,\mathcal K) = \frac{p^+(\mathbf k_i|\mathbf q)\prod_{j\ne i}p^-(\mathbf k_j|\mathbf q)}{\sum_{n=1}^{N}p^+(\mathbf k_n|\mathbf q)\prod_{j\ne n}p^-(\mathbf k_j|\mathbf q)} eq. 9

With p(i|\mathbf q,\mathcal K)=\frac{\exp(S(\mathbf q,\mathbf k_i))}{\sum_j^N\exp(S(\mathbf q,\mathbf k_j))}:

\mathcal L_{\text{NCE-global}}(q,\mathcal K) = \mathbb E_{P(i|q,\mathcal K)}\left[-\log\frac{\exp(S(q,k^+))}{\sum_{k\in\mathcal K}\exp S(q,k)}\right] eq. 10

Sohn's multi-class N-pair loss has the same form, using in-batch samples as negatives. NT-Xent adds a temperature to the cosine score:

\mathcal L_{\text{NT-Xent}}(\mathbf q,\mathcal K) = -\log\frac{\exp\!\left(\frac{\mathbf q^\top\mathbf k^+}{\lVert\mathbf q\rVert\lVert\mathbf k^+\rVert\tau}\right)}{\sum_{\mathbf k\in\mathcal K}\exp\!\left(\frac{\mathbf q^\top\mathbf k}{\lVert\mathbf q\rVert\lVert\mathbf k\rVert\tau}\right)} eq. 11

\tau controls the attraction-repulsion radius around the query, playing the role of the margin m in eq. 2 §III.E.3.

Mutual-information-based losses §III.E.4: van den Oord et al. showed minimising InfoNCE maximises a lower bound on MI between input and representation. InfoNCE has the N-pair / NT-Xent form with a bilinear score, and the review calls it the most popular form. DIM evaluated Donsker-Varadhan, Jensen-Shannon and InfoNCE estimators. Tschannen et al. argue MI maximisation alone does not explain why contrastive losses work §III.E.4 §VI.E.

History §IV tab. 1

Becker & Hinton (1992, MI between views) and Bromley et al. (1993, Siamese networks) → Chopra/Hadsell/LeCun pair loss (2005/2006) → triplet loss (Weinberger et al.; Collobert & Weston for LMs; Chechik et al. for image retrieval; later deep fine-grained similarity) → lifted structure and N-pair losses for more comparisons per query; sampling quality (Wu et al., Hermans et al.) → NCE (Gutmann & Hyvärinen 2010) for LMs and word embeddings → CPC and DIM (2018) link contrastive loss to an MI lower bound → instance discrimination with memory bank (Wu et al.) → Ye et al., PIRL showing contrastive pre-training can beat supervised-only training → MoCo (momentum encoder, queue) to avoid needing very large batches → SimCLR (explicit projection head) and SimCLRv2 → Local Aggregation (clustering) → BYOL, whitening MSE and Wang & Isola questioning whether negatives are needed §IV.

Applications §V

Language §V.A tab. 2, vision §V.B tab. 3, audio §V.C tab. 4, graphs §V.D tab. 5, multimodal §V.E tab. 6; tables reproduced under Results. Points from the text not in the tables:

• BERT/XLNet: for word embeddings (a lookup table) the authors argue reconstructing and contrasting are equivalent, so masked LM counts as contrastive; Kong et al. show BERT/XLNet maximise global-local MI and that next-sentence prediction builds similarity pairs from sequential coherence §V.A.
• InfoXML (Chi et al.): monolingual and cross-lingual token-sequence MI plus cross-lingual sentence-sentence MI; CLS token with linear projection; momentum encoder for the query §V.A.
• DPC uses three kinds of negatives: easy (other videos), spatial (same video, other location) and hard (same location, other time) §V.B.
• Alignment and uniformity (Wang & Isola): uniformity explains why negatives prevent collapse §V.B.
• Supervised contrastive learning was found more robust to corruption than cross-entropy with the labels alone §V.B.
• Cross-AVID's Cross-modal Agreement adds as positives other videos that agree in both audio and visual features §V.E.
• Others §V.F: CRD (distillation), CPC|Action (belief states), CURL (MoCo-style instance discrimination for pixel-based RL, same augmentation across a frame stack), ATC (augmented observations vs future observations, momentum encoder), patch-wise InfoNCE for unpaired image-to-image translation (Park et al.), and greedy layer-wise training with InfoNCE (GIM, LoCo).

Discussion §VI

• Augmentations set the learned invariances; ImageNet's centred single-object images plus aggressive cropping give occlusion invariance that may misbehave on scene-rich datasets §VI.A.
• More negatives help but cost large clusters and long training; memory tricks or negative-free methods are the two responses; negative quality (hard negatives) matters too; the authors suggest architectural constraints early and hard negatives later as an open idea §VI.B.
• Head design is empirical: SimCLRv2 found the second layer of a 3-layer MLP head transfers better than the ResNet output; BYOL's prediction MLP is crucial §VI.C. LooC learns several embedding sub-spaces, each invariant to all but one transform §VI.C.
• Open: asymmetric, non-transitive scoring functions §VI.D; new loss forms §VI.E; optimising a task directly with a contrastive loss without a separate representation stage §VI.F.
• Practitioner checklist §VII: watch dataset biases (single vs multiple objects, centred objects), decide which invariances and covariances downstream tasks need, and design positive/negative construction to deliver them.

Hyperparameters

The review trains nothing and states almost no values. Everything below is structural.

Results

The review runs no experiments and reports no benchmark numbers. Its results are the classification tables below. Text-only reproduction: in the paper each cell is coloured by the taxonomy (similarity: multisensory / data transformation / context-instance / sequential coherence / clustering; encoder: end-to-end / online-offline / pre-trained; head: projection / contextualisation / quantisation); the colours are not reproduced, but most categories follow from the cell text and §III.

History tab. 1 (chronological by first disclosure):

Language tab. 2:

Vision tab. 3:

Audio tab. 4:

Graphs tab. 5:

Multimodal tab. 6:

Bracketed numbers are the review's reference numbers. Qualitative claims the review repeats from cited work, with no numbers: contrastive instance discrimination was the first self-supervised approach to outperform supervised ImageNet pre-training on several downstream vision tasks §V.B; DIM found InfoNCE the most effective MI estimator, especially with many negatives §V.B; Wav2vec features improved a supervised ASR system §V.C.

Paper vs code

No official code.

Pitfalls

• Sign error in the binary-NCE sigmoid. The review sets p(D=1|\mathbf q,\mathbf k)=1/(1+\exp(S(\mathbf q,\mathbf k))) §III.E.3, which decreases as similarity grows, so eq. 8 would push positives apart. For the loss to pull positives together the probability must increase with S, i.e. use 1/(1+\exp(-S)).
• Eq. 8 drops the noise ratio. Eq. 7 weights noise by m keys per positive, but eq. 8's noise expectation carries no m and p(D=1) is reparametrised as a plain sigmoid of S without it §III.E.3. Take the exact NCE objective from Gutmann & Hyvärinen or the NEG paper, not from this summary.
• m means two things: the margin in eq. 2–3 and the number of noise samples in eq. 7. n is both the number of classes in eq. 4 and the size of the uniform noise support; eq. 9 uses N for the number of candidate keys while \mathcal K was defined with n negatives.
• Eq. 2 puts the margin on squared distance. The hinge is \max(0, m - D^2), so in distance units negatives are pushed only out to radius \sqrt m, although the text calls m the radius §III.E.2. Eq. 3 likewise compares squared distances. Check which form the method you implement uses.
• The denominator includes the positive. In eq. 1 the sum runs j=0..K over one positive plus K negatives, and in eq. 5, 10 and 11 \mathcal K contains \mathbf k^+ §II.C §III.E.3. Excluding it changes the loss.
• No values anywhere. Temperature, batch size, number of negatives, MoCo momentum, head widths, augmentation strengths: none are given §II.C §III.C.2. The review only says results depend heavily on augmentation and on many negatives §II.C §VI.B.
• Collapse. Minimising positive-pair distance alone can be solved by a constant f §III.E. You need explicit negatives, or the architectural constraints that negative-free methods rely on (BYOL's online/momentum networks with a prediction MLP, whitening, SwAV's batch-wise codes) §V.B §VI.C. The review describes these mechanisms only qualitatively.
• End-to-end memory cost. With end-to-end encoders, batch size equals the number of in-batch negatives and is capped by GPU memory §III.C.1; memory banks are stale by up to one epoch, which MoCo's queue + momentum encoder mitigates §III.C.2.
• Dot-product scores are unbounded and scale with vector norms; normalise (cosine) or regularise \lVert\mathbf z\rVert^2 §III.E.1.
• Augmentation bias transfers. Invariances learned from object-centric ImageNet crops may be wrong for scene-rich data §VI.A; design p^+ for the invariances your downstream task needs §VII.
• Discard the head. The standard recipe drops h after pre-training and uses \mathbf v §II.C, but SimCLRv2 found an intermediate head layer transfers better §VI.C; which layer to keep is an empirical choice.
• Citation mix-ups in tab. 2. Bert-NCE and InfoWord are attributed to [18], which in the reference list is InfoXLM (Chi et al.); the text attributes both to Kong et al. [61] §V.A. The row "word2vec [70]" points to Mnih & Kavukcuoglu's NCE word-embedding paper, not to word2vec. NEG is credited to Mikolov et al. [67] in the text §V.A and to [68] in tab. 1. Verify references before citing them.
• "Contrastive" is used broadly. The review counts BYOL, whitening-MSE and masked language modelling as contrastive §III.E §V.A; other sources may not.
• Snapshot in time: v2 dates from October 2020; later methods are not covered.

Lineage

• Builds on: the framework extends SimCLR's description of instance discrimination (arXiv:2002.05709) beyond images and self-supervision §III.A, and borrows the query/key metaphor from MoCo (arXiv:1911.05722). Neither has a card here.
• Compared against: no methods of its own; it classifies CPC (arXiv:1807.03748), DIM (arXiv:1808.06670), BYOL (arXiv:2006.07733), SwAV, PCL, wav2vec 2.0, DGI, CMC and others tab. 2–tab. 6.
• Superseded by: not known from content in this wiki.