SLIP: Self-supervision meets Language-Image Pre-training

In plain words

A model that learns from photos with captions, such as CLIP, learns to match a picture with the text that describes it. A separate family of methods learns from pictures alone, by checking that two altered copies of the same picture map to similar internal codes, and it needs no labels at all. This paper asks whether the two signals help each other when they are used together.

The answer it gives is yes. The authors train one image network on both objectives at once, so the same network learns from captions and from the pictures themselves. Measured on new images, the combined model improves on the caption-only model when the labels are given as text, and on the picture-only model when a simple linear layer is trained on top of frozen features. The training data is a subset of a public photo collection of about 15 million images. The idea is to add a second objective and keep the rest of the recipe the same.

TL;DR

• SLIP trains one Vision Transformer with CLIP's image-text contrastive loss plus a SimCLR loss on augmented images, summed with weight 1.0 §3.3 alg. 1.
• On YFCC15M for 25 epochs with ViT-B/16, SLIP reaches 42.8% zero-shot and 72.1% linear ImageNet accuracy. The authors' CLIP run gets 37.6% and 66.5% tab. 2.
• The gain holds for ViT-S, B and L, and for 100-epoch training, on ImageNet and on 26 benchmarks including ImageNet §5.1 §5.2 tab. 4 tab. 11.
• Fine-tuning gains are small, +0.1 to +0.4 over the stronger baseline tab. 2. The cost is about 3× more activations per step §3.3, and 30.5 hours against 22.3 hours for ViT-B/16 on 64 V100 GPUs §7.

Problem

The standard self-supervised evaluation on ImageNet classification from frozen features heavily biases results against models not pre-trained on ImageNet §1. CLIP-style language supervision avoids labels and scales well §1. The paper asks whether image self-supervision can aid that language supervision §1. The authors note that the two objectives could interfere, because each asks the encoder to keep different information about the image §1.

Method

SLIP keeps CLIP's two-branch setup and adds an image-only branch. The paper has no numbered equations. Its loss appears in prose §3.3 and as code in alg. 1.

1. Views. Each image gives three inputs: one crop for the CLIP branch, and two augmented copies for the self-supervised branch §3.3 alg. 1. The CLIP crop covers 50% to 100% of the image §4.1.
2. Encoders. The text goes through a text Transformer. All image inputs go through one shared image encoder §3.3 alg. 1.
3. CLIP term. Image and caption embeddings are projected into a shared 512-dimensional space by separate learned linear maps, normalised, and scored with a symmetric InfoNCE loss over the batch §3.1 §4.1 alg. 1.
4. Self-supervised term. The two augmented views are passed through a 3-layer projection head, then trained with a SimCLR loss. Matching views are positives, and all other views in the batch are negatives §3.2 §4.1 alg. 1.
5. Sum. The total loss is the CLIP term plus the self-supervised term times 1.0 §3.3. The paper says the balance can be changed by rescaling the self-supervised term, and that 1.0 works well with SimCLR §3.3.

The self-supervised branch is an adaptation of SimCLR, and the paper says other self-supervised objectives can be swapped in §3.2 §5.5.

# One SLIP training step  [§3.3] [alg. 1]
# Inputs: N images and N captions from the same pairs
# Output: one scalar loss

def slip_loss(images, captions):
    # CLIP view: a crop covering 50-100% of the image  [§4.1]
    clip_view = random_crop(images)         # (N, 3, 224, 224)
    # Two augmented views for the self-supervised branch  [§4.1]
    view_1 = augment(images)                # (N, 3, 224, 224)
    view_2 = augment(images)                # (N, 3, 224, 224)
    tokens = tokenize(captions)             # (N, 77)

    # One shared image encoder for all image views  [§3.3] [alg. 1]
    clip_feats = image_encoder(clip_view)
    feats_1 = image_encoder(view_1)
    feats_2 = image_encoder(view_2)
    text_feats = text_encoder(tokens)

    # Projections: 512-d for CLIP, 3-layer MLP to 256-d for SSL  [§4.1]
    img_emb = clip_projection(clip_feats)   # (N, 512)
    txt_emb = text_projection(text_feats)   # (N, 512)
    z_1 = ssl_head(feats_1)                 # (N, 256)
    z_2 = ssl_head(feats_2)                 # (N, 256)

    # Sum with weight 1.0 on the self-supervised term  [§3.3]
    clip = clip_loss(img_emb, txt_emb)
    return clip + 1.0 * simclr_loss(z_1, z_2)

# Symmetric InfoNCE over the batch  [§3.1] [alg. 1]
# Matching image and caption sit on the diagonal

def clip_loss(img_emb, txt_emb):
    # Normalise both embeddings  [code: losses.py#L32-L33]
    img_emb = normalize(img_emb)
    txt_emb = normalize(txt_emb)
    # Scale cosine similarities by a learned logit scale  [code: models.py#L102]
    logits = exp(log_scale) * img_emb @ txt_emb.T     # (N, N)
    labels = range(N)
    # Image-to-text and text-to-image cross-entropy, averaged  [alg. 1]
    loss_image = cross_entropy(logits, labels)
    loss_text = cross_entropy(logits.T, labels)
    return (loss_image + loss_text) / 2

# SimCLR loss over the two augmented views  [§3.2] [alg. 1]
# tau is the softmax temperature; the paper does not state its value

def simclr_loss(z_1, z_2):
    # Normalise both views  [code: losses.py#L77-L78]
    z_1 = normalize(z_1)
    z_2 = normalize(z_2)
    # Cross-view similarities, scaled by 1 / tau  [code: losses.py#L96-L97]
    cross = z_1 @ z_2.T / tau               # (N, N)
    # Same-view similarities, self-pairs masked  [code: losses.py#L89]
    self_mask = eye(N) * 1e9
    same_1 = z_1 @ z_1.T / tau - self_mask  # (N, N)
    same_2 = z_2 @ z_2.T / tau - self_mask  # (N, N)
    # Each row ranks its positive view against all other views  [alg. 1]
    logits_1 = concat(cross, same_1, dim=1)
    logits_2 = concat(cross.T, same_2, dim=1)
    labels = range(N)
    # Average of the two directions  [code: losses.py#L99-L101]
    loss_1 = cross_entropy(logits_1, labels)
    loss_2 = cross_entropy(logits_2, labels)
    return (loss_1 + loss_2) / 2

Shapes assume one device. The code gathers embeddings across devices before computing the negatives losses.py losses.py.

Hyperparameters

Pre-training unless marked "fine-tune" or "linear". Paper values come first, then code values where the paper is silent or differs.

Results

Author-reported. Setup: pre-training on YFCC15M for 25 epochs unless noted, then evaluation on ImageNet-1K. The CLIP baseline is the authors' own re-implementation, trained with their adjusted augmentation recipe. Their ResNet-50 variant reaches 34.6% zero-shot against 31.3% for the original CLIP result §4. Results are not comparable with methods pre-trained on ImageNet-1K §5.1.

ImageNet, three model sizes, YFCC15M, 25 epochs tab. 2

The printed zero-shot gain for ViT-L is +4.8. The numbers in this table give +5.8. See Pitfalls.

Scaling, ViT-B/16 on YFCC15M tab. 3 tab. 10

Model size, 100 epochs, ViT-S, B and L tab. 3

For ViT-L, linear accuracy falls from 76.0 at 25 epochs to 75.8 at 50 and 75.1 at 100 tab. 10. The paper leaves this unexplained app. B.

Other pre-training datasets, ViT-B/16, 0-shot / linear / finetuned tab. 5

The YFCC15M row is taken from tab. 2, because tab. 5 prints the linear values swapped. See Pitfalls. The CC3M and CC12M sizes are 3.1M and 11.0M images, not the public totals, because assets were removed app. A.

Averages over 26 benchmarks, zero-shot tab. 4 and linear tab. 11

The per-dataset numbers are in tab. 4 and tab. 11. The paper notes that several datasets with little overlap with YFCC15M are near chance. These include Rendered SST2, KITTI distance and PatchCamelyon §5.3.

Self-supervised framework, ViT-B/16, 25 epochs tab. 6

Controls, ViT-B/16, 25 epochs

• Finetuning CLIP from SimCLR-pre-trained weights reaches 41.1 / 68.2 / 82.3, below SLIP's 42.8 / 72.1 / 82.6 tab. 7.
• CLIP with colour and blur augmentation reaches 38.4 / 68.5 / 81.5. Adding resize-crop and flip hurts tab. 8.
• Decoupled SLIP, with a disjoint set of 15M images for the self-supervised term, reaches 42.7 / 72.0 / 82.8 tab. 9.

Paper vs code

The repository linked on the paper's first page is facebookresearch/SLIP. It is MIT-licensed LICENSE and was checked at commit c6faf5d03cbfa7d529d210779f859cd3dddec09a. Each difference below was checked against both the paper and the code.

• Self-supervised augmentation. The paper says the self-supervised branch uses the MoCo v3 augmentation and cites the SimCLR reference §4.1. The code uses a SimCLR-style augmentation: random resized crop from 8% to 100% of the image, colour jitter with probability 0.8, grayscale with probability 0.2, Gaussian blur with probability 0.5, and horizontal flip datasets.py. The code's values are the ones used.
• Linear probe. The paper uses 100 epochs and LR 0.01 on ImageNet §4.1. The code defaults to 90 epochs and LR 0.1, scaled by the batch size divided by 256 main_linear.py main_linear.py. The README command for ImageNet sets neither flag README.md. The batch size for the paper's probe is not stated.
• ViT-L fine-tune. The appendix says drop path is 0 for ViT-L app. A. The README uses drop path 0.1 and layer decay 0.75 README.md. The paper does not state a layer decay for ViT-L.
• Temperature and mask order. The paper's pseudocode applies the self-pair mask before dividing by τ alg. 1. The code divides first and then applies the mask losses.py. The mask is 1e9, so both orders give the same softmax.
• Pre-training learning rate and weight decay. The paper states weight decay 0.5 for CLIP and 0.1 for SLIP §4.1, which matches the README README.md README.md. It states no learning rate, and the README's CLIP rate (5e-4) differs from SLIP's (3e-3). Any comparison that depends on the learning rate is therefore not controlled by the paper.

No other discrepancies found. The self-supervised loss weight, the projection sizes, the 4096 batch, the CLIP and SLIP weight decay values, and the ViT-S and ViT-B fine-tune settings match.

Pitfalls

• Table 5 swaps the YFCC15M linear values. tab. 5 prints CLIP 72.1 and SLIP 66.5 for YFCC15M, but tab. 2, tab. 6 and tab. 7 give CLIP 66.5 and SLIP 72.1. The text says SLIP wins in every ImageNet setting §5.4. Use tab. 2.
• The ViT-L zero-shot gain does not match its numbers. tab. 2 prints +4.8 for ViT-L zero-shot, and §5.1 and §7 repeat "+5.6% to +4.8%". But 46.2 − 40.4 = 5.8. Use the raw numbers.
• The CLIP baseline is the authors' re-implementation. It uses the authors' adjusted augmentation recipe, which the paper says mainly changes data augmentation §4. Numbers from the original CLIP report are not comparable.
• Pre-training LR and temperature are missing from the paper. Take them from the README, and note that the CLIP and SLIP learning rates differ README.md README.md. The temperature is 0.1 in code main.py.
• The logit scale is clamped. It starts at 1/0.07 and is clamped to [0, ln 100] after each optimizer step models.py main.py. The paper says only that the scale is learnable.
• Weight decay excludes some parameters. Biases, norm parameters, BN parameters and anything with fewer than two dimensions get no weight decay main.py. The paper does not say this.
• BatchNorm inside the self-supervised head. The projection head uses SyncBatchNorm between its layers models.py. With gradient accumulation, the batch statistics change. The README warns about this README.md. The paper uses gradient accumulation for ViT-L app. A.
• Negatives span the whole batch. Under distributed training the CLIP loss gathers embeddings from all devices losses.py. The SimCLR loss uses a gather that keeps the graph connected losses.py utils.py. Batch size 4096 is the total across devices.
• Compute is higher than the abstract suggests. SLIP runs about three times as many image activations per step §3.3. That is 30.5 hours against 22.3 hours for ViT-B/16 on 64 V100 GPUs §7.
• ViT-S head count. The code uses 12 heads for ViT-S models.py. The paper cites DeiT for ViT-S and does not state a head count. The README header refers to a 12-versus-6-head comparison README.md.
• Zero-shot template normalisation. The code normalises each template embedding before averaging eval_zeroshot.py. The paper says only "averaging the caption embeddings".
• Fine-tune details missing from the paper. Weight decay (0.05), mixup, cutmix, LR warmup and the total number of GPUs are not in the paper. The README sets mixup 0.8, cutmix 1 and warmup 20 epochs for 100-epoch runs README.md.
• Table and figure numbers. I numbered tables by their order in the LaTeX source, because no compiled PDF was available. Check them against the PDF before citing a number.
• Linear accuracy can fall with longer training. For ViT-L, it drops as epochs rise app. B. The authors do not show a cause.
• Rankings across datasets are unstable. The paper says zero-shot rankings change between runs on datasets with little overlap with YFCC15M §5.3.

Lineage

• Builds on: CLIP (Radford et al., "Learning Transferable Visual Models From Natural Language Supervision", cited as Radford2021LearningTV). The paper's own description is that CLIP uses separate encoders for images and text, projects them into a shared space, normalises them and applies InfoNCE §3.1. No arXiv id is given in the source.
• Builds on: SimCLR ("A Simple Framework for Contrastive Learning of Visual Representations", cited as Chen2020ASF). It supplies the self-supervised loss §3.2. No arXiv id is given in the source.
• Builds on: MoCo v3 ("An Empirical Study of Training Self-Supervised Vision Transformers", cited as Chen2021AnES). The paper follows its settings for training self-supervised ViTs §3.2.
• Compared against: MoCo v3, BYOL and BEiT as alternative self-supervised objectives §5.5. Also the CLIP baseline and the fine-tune-from-SSL alternative §6.
• Superseded by: not known from this wiki.