A Survey on Contrastive Self-supervised Learning

In plain words

Labeling large image collections by hand is expensive. Self-supervised learning gets around this by letting the data supply its own training signal: the model is asked a question whose answer is known from the data itself, and whatever it learns along the way is reused for other jobs.

Contrastive methods ask the model to make two views of the same thing look alike in its internal representation, and to keep views of different things apart. A view is usually a cropped, recoloured or otherwise altered copy of an image. The other images in the batch serve as the "different" examples. The survey explains the choices involved: which question to ask (the pretext task), how to keep enough "different" examples around, which network to use, and how to check the result on real tasks such as classification and detection.

The survey's main point is that these methods have closed much of the gap with supervised training on ImageNet, while the open problems are mostly about theory, choosing the right pretext task and augmentations, and handling hard negatives.

TL;DR

• A survey, not a new method. It reviews contrastive self-supervised learning in vision and NLP and compares reported results §1 §7.
• Four architectures differ mainly in how they supply negative samples: end-to-end, memory bank, momentum encoder, clustering §3.
• The losses are cosine similarity inside an NCE or InfoNCE softmax, with a temperature τ §5.
• On ImageNet linear evaluation the survey's table puts SwAV at 75.3 top-1, just under the supervised ResNet-50 at 76.5 tab. 1.
• Use it for orientation and for the benchmark numbers. Do not implement from it: most hyperparameters are not stated.

Problem

Supervised learning needs large annotated datasets, and annotation is costly §1. Generative self-supervised models such as GANs were hard to train, with non-convergence and a discriminator that overpowers the generator §1. The survey proposes contrastive, discriminative pre-training as the alternative §1. It assumes that for each training sample a positive view is available and that many negatives exist in the batch or dataset §1 §3.

Method

The survey gives no single algorithm. It describes a generic pipeline, and each step lists the options it reviews.

1. Choose a pretext task that creates labels from the data. Options are colour transformations, geometric transformations, context tasks (jigsaw, frame order, future prediction), and view prediction §2.1–§2.4. The survey stresses that the choice matters and that augmentations add invariances that can help or hurt a given downstream task §2.5 §9.2.
2. Build the pairs. The anchor is the original sample, the positive is an augmented or otherwise related view, and the negatives are the remaining samples in the batch or dataset §1 §2.
3. Encode both views. Most works use a ResNet, most often ResNet-50 §4. Features are average-pooled, and some methods add a small projection head §4. Video methods typically use a 3D-ResNet §4.
4. Decide where negatives come from. The four architectures are:
   • End-to-end §3.1: negatives are the other items in the current batch, so large batches are preferred.
   • Memory bank §3.2: a stored feature per dataset sample, refreshed when that sample is next seen; this allows negatives beyond the batch but is costly to keep current.
   • Momentum encoder §3.3: a queue of keys from a slowly updated key encoder, replacing the memory bank. The key encoder follows the query encoder by the update in eq. 1.
   • Clustering §3.4: instead of pairwise instance discrimination, features are grouped into clusters (SwAV-style).
5. Score and compare. Similarity is cosine similarity, the inner product of unit-length vectors eq. 2. Training uses NCE with one negative eq. 3, or InfoNCE with a sum over several negatives eq. 4, both divided by a temperature τ §5.
6. Optimize. SGD with momentum is common §5. Adam is used in a few works §5. For very large batches the survey describes LARS with a cosine learning-rate schedule §5.
7. Evaluate. Use a frozen encoder with a linear classifier, or fine-tune with 1% or 10% of ImageNet labels tab. 1. The survey also suggests qualitative checks: kernel and attention-map visualisation and nearest-neighbour retrieval §6.1 §6.2 fig. 18.

Momentum update of the key encoder §3.3, with m in [0, 1):

\theta_k \leftarrow m\,\theta_k + (1 - m)\,\theta_q eq. 1

Cosine similarity §5:

\mathrm{cos\_sim}(A, B) = \frac{A \cdot B}{\lVert A \rVert\,\lVert B \rVert} eq. 2

NCE loss with one positive k₊ and one negative k₋ §5:

L_{NCE} = -\log \frac{\exp(\mathrm{sim}(q, k_+)/\tau)}{\exp(\mathrm{sim}(q, k_+)/\tau) + \exp(\mathrm{sim}(q, k_-)/\tau)} eq. 3

InfoNCE loss with negatives k_i for i = 0..K §5:

L_{InfoNCE} = -\log \frac{\exp(\mathrm{sim}(q, k_+)/\tau)}{\exp(\mathrm{sim}(q, k_+)/\tau) + \sum_{i=0}^{K} \exp(\mathrm{sim}(q, k_i)/\tau)} eq. 4

A training step for the momentum-encoder design, written from §3.1, §3.3, eq. 1 and eq. 4. The paper gives no algorithm box, so this is a condensed restatement, not a quotation:

# Momentum-encoder contrastive step  [§3.1] [§3.3] [eq. 1] [eq. 4]
# m and tau have no value in the paper  [§3.3] [§5]
# Input:  images (B, 3, H, W); queue of past keys (K, d)
# Output: scalar loss; updated key encoder and queue

def training_step(images, queue, tau, m):
    # Query encoder sees the original sample  [§3.1]
    queries = normalize(query_encoder(images))         # (B, d)
    # Key encoder sees an augmented sample; no backprop  [§3.1] [§3.3]
    keys = normalize(key_encoder(augment(images)))     # (B, d)
    # Positive logit: each query against its own key  [eq. 2] [eq. 4]
    positive_logit = sum(queries * keys, axis=1)       # (B,)
    # Negative logits: each query against the queued keys  [§3.3] [eq. 4]
    negative_logits = queries @ queue.T                # (B, K)
    # InfoNCE over one positive and K negatives, temperature tau  [eq. 4]
    loss = mean(
        -log(
            exp(positive_logit / tau)
            / (exp(positive_logit / tau)
               + sum(exp(negative_logits / tau), axis=1))
        )
    )
    # Momentum update of the key encoder from the query encoder  [eq. 1]
    key_params = m * key_params + (1 - m) * query_params
    # Enqueue the newest keys and drop the oldest mini-batch  [§3.3]
    queue = enqueue_dequeue(queue, keys)
    return loss

Hyperparameters

Only a few values are given in the text. Everything else in this table is either not stated or named without a value.

Results

The survey does not run experiments. It collects numbers from the original papers on the downstream tasks they report §7, so these are author-reported and not reproduced here. Setup per table is given in its caption.

Table 1 (ImageNet). Top-1 linear evaluation with frozen encoders; the last two columns are top-5 accuracy after fine-tuning on 1% and 10% of labels tab. 1.

Table 2 (Places). Linear classifier on frozen features, pretrained on ImageNet and evaluated on Places; parameters as listed in the table tab. 2.

Table 3 (PASCAL VOC). (1) Linear classification top-1 on frozen features; (2) object detection with fine-tuned features on VOC7+12 using Faster R-CNN tab. 3. The text says the linear models were pretrained on VOC7 §7; the caption does not.

Table 4 (video classification). Linear model on frozen features, except where marked. The caption says every proposed method was pretrained with its own contrastive approach. "†" marks a model trained on another dataset and then fine-tuned, and K is Kinetics tab. 4. The "*" on IIC is not explained in the caption.

NLP. Two tables tab. 5 tab. 6. The numbers use different datasets and metrics and must not be compared across rows §8.

tab. 6 lists the same methods with their datasets and application areas: Distributed representations (Google internal, skip-gram training); Contrastive unsupervised (Wiki-3029, unsupervised representation learning); CONPONO (RTE, COPA, ReCoRD, fine-grained sentence ordering); INFOXLM (XNLI and MLQA, cross-lingual representations); CERT (GLUE, sentence-level semantics); DeCLUTR (OpenWebText, universal sentence representations).

Headline reading, as the survey states it: SwAV is the strongest ImageNet entry in the table and is described as comparable to supervised training §7 tab. 1. It also beats the supervised baselines on Places and VOC in the tables above tab. 2 tab. 3.

Paper vs code

No official code. This is a survey, and its LaTeX source contains no repository, project-page or code link. I searched the source files for GitHub and other URLs and found none. No GitHub search was run, because a survey has no code of its own. There is nothing to implement from code, and no code field is set.

Pitfalls

• This is not a method. Do not implement "the survey". The pipeline in Method is a summary of options, and the numbers are from the original papers. For an implementation, go to the cited original method.
• Numbers are not comparable across tables. The protocols differ: frozen linear probes, fine-tuning on 1% or 10% labels, pretraining on ImageNet or VOC or Kinetics, and different backbones and parameter counts tab. 1–tab. 4. The survey does not re-run any method §7.
• Wrong table reference for Places. The Places text points to the PASCAL table, table:pascal §7, while the Places numbers are in Table 2 tab. 2. Check which table a claim refers to.
• Pretraining data for Table 3 is ambiguous. The text says the linear classifiers were pretrained on VOC7 §7; the caption gives no pretraining dataset. Confirm in the original papers.
• Table 4 caption overstates. It says all proposed methods use contrastive pretraining tab. 4, but the rows include ImageNet-inflated, jigsaw, OPN and others whose pretraining the survey does not describe. The "*" on IIC has no explanation.
• Citation mismatch in the video table. The "inflated SimCLR" row cites the same key as CVRL tab. 4. Check the original before using that number.
• Table 5 mixes metrics. Rows use AVG-5, AVG-10 and unlabelled accuracy on different datasets tab. 5.
• Inconsistent SimCLR citation. SimCLR is cited as chen2020simple in §2.1, §3.1 and Table 1, but as chen2019selfsupervised in §4 (for a 1-hidden-layer MLP head) and §9.2. Check which paper that entry is.
• Priority claims are the survey's framing. It says contrastive learning "was first introduced" by Mikolov et al. in 2013 §8, and that Word2Vec "first introduced self-supervised methods" §2.6.1. The card does not verify these claims.
• Encoder weight sharing is unresolved. §3.1 says the query and key encoders "can be different" and are trained end-to-end. §3.4 says the clustering pair "share parameters". The survey does not say which applies to end-to-end methods such as SimCLR.
• Equation 4 index. The sum runs over i = 0..K, which is K+1 terms, and K is not defined as a count of negatives eq. 4. Decide the negative count when implementing.
• Missing key values. No value is given for τ, the momentum m, the queue or bank size, the batch size of most methods, or any learning rate or schedule §5 §3.3. The only numbers are SimCLR's batch size 4096 for 100 epochs §3.1. Fig. 12 shows more runs, but its values appear only as bars fig. 12.
• Uncited claims. "SGD with momentum is used in most deep learning approaches" §5 and "ResNet-50 ... most widely used" §4 have no citation.
• Vague training-optimizer claims. LARS is said to stabilise large-batch training §5. The survey gives no comparison numbers for it.
• Memory bank cost. The survey notes that keeping a memory bank current is costly, but gives no size or update cost §3.2.
• Hard negatives. The survey cites hard-negative mixing as adding many dataset-specific hyperparameters §9.3. None are listed.

Lineage

• Builds on: none (survey).
• Compared against: the methods in tab. 1–tab. 6, including SimCLR, MoCo, PIRL, SwAV, CPC and CVRL.
• Superseded by: not stated in the paper.