Unsupervised Learning of Dense Visual Representations

In plain words

Most self-supervised image models learn one summary vector per picture. That works for "what is in this photo", but tasks such as outlining objects, guessing depth or following a point through a video need an answer for every pixel.

The authors take two randomly altered copies of the same picture, such as a crop, a colour change or a blur. Because the crop and resize are known, they know which pixel in one copy shows the same spot as which pixel in the other. The network is trained so that these matching pixels get similar descriptions, and other pixels get different ones. No person labels anything.

The result is a network that gives each pixel a description useful for dense tasks. The authors report that its frozen features do better than those of comparable self-supervised models on several such tasks.

TL;DR

VADeR is a contrastive method applied per pixel instead of per image. Pixel correspondences come for free from the view generation. The encoder is a ResNet-50 feature pyramid with a decoder that outputs a 128-channel map at 1/4 resolution, initialised from MoCo v2 for the encoder. With frozen features it beats MoCo and ImageNet-supervised ResNet-50 on VOC segmentation and NYU depth, and it beats MoCo on all COCO Mask R-CNN metrics reported. All results are author-reported. No official code was found.

Problem

Contrastive methods that compare one global vector per image are efficient, but the global vector is invariant to pixel-level changes §1. The authors argue this is enough for some tasks, such as classification, but not for dense prediction such as segmentation, depth, optical flow, keypoints and correspondence §1. Dense methods that did exist usually needed labelled data or specialised architectures §2. The paper assumes that augmentations give known pixel correspondences between views §1 §3.1 §3.2.

Method

VADeR learns an embedding per pixel such that the same scene location seen under two views has a similar embedding and different locations do not §3.1.

1. Pixel representation. A pixel u of image x is the tuple (x, u). A view v maps it to (vx, vu). Views mix appearance changes (noise, illumination, saturation, blur, hue) and geometric changes (affine, homography, non-parametric warps) §3.1. The implementation uses Gaussian blur, colour jitter and/or greyscale, plus random crop and resize to 224×224, the same transforms as MoCo §3.2.
2. Invariance goal. Two encoders f and g map each pixel to a d-dimensional embedding. Their parameters may be shared, partly shared or separate §3.1. The ideal constraint is that a pixel's embedding under one view is more compatible with the same pixel under another view than with any other pixel, for all pixels and view pairs eq. 1 §3.1.
3. Loss. The loss is a noise-contrastive (NCE) loss over pixels. For a pixel u, the positive is the same pixel under a second view. The N negatives are random pixels u' from other images, and the loss is the negative log of the positive's softmax share eq. 2 §3.1.
4. Compatibility. c is a cosine similarity divided by temperature τ, with no learned parameters §3.1. The paper sets τ = 0.07 §3.2.
5. Positive pairs. Each training image gives a random pair of views. The two views must share at least 32 pixels. The correspondence map between them gives matching pixel pairs, and each matching pair is a positive sample §3.2. For each training sample, 32 pairs are drawn at random from all matches §3.2.
6. Negatives. Negatives come from other images, held in a MoCo queue of 65,536 entries with momentum 0.999 §3.2. The authors tried negatives from the same image and could not make them work in preliminary experiments §3.2.
7. Encoder and decoder. Both are the same FPN-based network: a ResNet-50 backbone, an FPN with four pyramid levels at 1/32 to 1/4 resolution and 256 channels, and per-level upsampling blocks that bring every level to 128 channels at 1/4 resolution. Blocks are summed and passed through a 1×1 convolution. For a 224×224 input the output is 128×56×56 §3.2.
8. Update rule. f is trained with SGD. g is a moving average of the weights of f §3.2. The decoder is trained from random initialisation, while the encoder starts from MoCo §3.2.

Pseudocode for one training step, reconstructed from the text. The paper gives no algorithm box, so the loop order and the queue update are reconstructed.

# One VADeR training step  [§3.1] [§3.2] (reconstructed from text)
# Input:  image (3, 224, 224); queue of negative pixel keys (65536, 128)
# Output: scalar loss; updates f (SGD) and g (moving average)

def training_step(image, queue):
    # Sample two views; resample until >= 32 pixels are in both  [§3.2]
    view_1, view_2, match = sample_views_min_overlap(image, 32)
    # Draw 32 matched pixel pairs at random as positives  [§3.2]
    pairs = random_subset(match, 32)
    # Queries from f on view 1; keys from g on view 2  [§3.1] [§3.2]
    queries = f(view_1)                       # (128, 56, 56)
    keys = g(view_2)                          # (128, 56, 56)
    # Cosine similarity divided by temperature 0.07  [§3.1] [§3.2]
    positive = cosine(queries[pairs.q], keys[pairs.k]) / 0.07
    # (32,)
    negative = cosine(queries[pairs.q], queue) / 0.07
    # (32, 65536)
    # NCE over one positive and the queue, scaled by 10  [eq. 2] [§3.2]
    loss = -10 * log(exp(positive) / (exp(positive) + sum(exp(negative))))
    # Encoder f: SGD (lr 3e-7); decoder: SGD (lr 3e-3)  [§3.2]
    f = sgd_step(f, loss, learning_rate=3e-7)
    # g: moving average of f with momentum 0.999  [§3.2]
    g = momentum_update(g, f, 0.999)
    # Not stated which keys enter the queue; the sampled keys are assumed
    queue = enqueue(queue, keys[pairs.k], max_size=65536)
    return loss

The paper states "SGD with weight decay 0.0001 and momentum 0.9" for f §3.2; the pseudocode above omits these. The per-group learning rates are also stated only as values, not as a schedule §3.2.

Key equations:

c(f_{v_1 u},\, g_{v_2 u}) > c(f_{v_1 u},\, g_{u'}) \quad \forall\, u, u', v_1, v_2 eq. 1

\mathcal{L}(u, \mathcal{U}^-) = -\mathbb{E}_{(v_1, v_2)\sim\mathcal{V}_u} \left[ \log \frac{\exp(c(f_{v_1 u}, g_{v_2 u}))}{\exp(c(f_{v_1 u}, g_{v_2 u})) + \sum_{u' \in \mathcal{U}^-} \exp(c(f_{v_1 u}, g_{u'}))} \right] eq. 2

Hyperparameters

Pretraining is on ImageNet-1K train (about 1.28M images) §3.2. Feature extraction and fine-tuning settings are in the appendix app. A.1 app. A.2.

Results

Author-reported. Setup: VADeR pretrained on ImageNet-1K. Frozen-feature experiments use a linear 1×1 convolution on the 128-channel output §4.1. Baselines are ResNet-50 variants adapted to the same 1/4 resolution, with dilated convolutions replacing strided ones §4.1. Table numbers below follow the order in which the paper's captions appear.

Frozen features, dense prediction (mIoU, higher is better; RMSE, lower is better; 5 trials averaged) tab. 1:

Video instance propagation, DAVIS-2017 val (J = region similarity, F = contour accuracy; mean; 320 / 480 px input) tab. 2:

View sampling ablation (VOC mIoU, DAVIS J; "fix." = frozen features, "f.t." = fine-tuned) tab. 3:

Mask R-CNN on COCO val2017 (fine-tuned; 5 trials averaged) tab. 4. The source header for the mask AP50 column reads AP^mk_55; read here as AP50.

Fine-tuning with varying labelled data (mean of 5 trials) app. A.3:

The text says VADeR matches MoCo at 100% of labels with no statistical difference, and beats supervised pretraining at that level §4.2. The fine-tuning figure itself, with standard deviations, is fig. 4.

Qualitative results: the paper shows pixel retrieval fig. 2, task examples fig. 3 and PCA-to-RGB feature maps where MoCo is shown beside VADeR fig. 5. The paper describes these qualitatively and reports no numbers for them.

Paper vs code

No official code. Searched the paper's TeX source, the arXiv abstract page (no code or project links; the only repository links in the source are TimeCycle and Detectron2, which are third-party), and GitHub: the query VADeR, the query view-agnostic dense representations, the query VADeR org:ElementAI (zero results), a direct check of github.com/ElementAI/VADeR (HTTP 404), and a keyword query combining "pixel-level contrastive dense correspondence" with the first author's name (zero results). The GitHub search for VADeR returned repositories with other meanings only. The web search tool was not permitted in this run, so the search of the open web beyond GitHub is incomplete.

Two differences between text and tables, not code:

• Paper text: "consistently outperform MoCo (and the supervised baseline)" on Mask R-CNN §4.3. Table 4 has supervised ImageNet at 59.8 AP50^bb against VADeR's 59.7 tab. 4.
• Paper text: VADeR's video results come from a 2.4M-iteration model §4.1, while pretraining is stated as about 6M iterations §3.2. The paper does not say which checkpoint the other tables use.

Pitfalls

• Pixel correspondence is not fully specified. The paper says correspondences come from known view geometry §3.2. It does not say how a 224×224 input's correspondences map to the 56×56 output grid, nor how the 1/4-stride cells are matched. An implementer has to choose a rounding convention.
• Which keys enter the queue is not stated. The text says the dictionary is updated §3.2. It is not clear whether all positives' keys, all pixels, or a subsample enter the MoCo queue, or how many per image.
• Learning-rate schedule and warm-up are not stated. Only the peak values are given for the encoder (3e-7) and decoder (3e-3) §3.2. The learning-rate gap of four orders of magnitude is not explained.
• Loss scale of 10 interacts with the learning rate. The paper multiplies the loss by 10 for stability in small-label fine-tuning §3.2. The text sits inside the pretraining paragraph. Whether the scale is used during pretraining, fine-tuning or both is ambiguous. Treat it as a pretraining-time factor unless the implementer confirms otherwise.
• Batch size and GPU count are ambiguous. "Batch size 128" on 4 GPUs is not said to be per GPU or in total §3.2. This changes the effective number of negatives and matches per step.
• Cost is hidden. Only 32 positive pairs per image are used per step §3.2. Each step still computes dense features for both views. The paper gives no wall-clock time or memory use.
• Dependence on MoCo. The encoder starts from MoCo v2 §3.2, and the baselines are MoCo v2 §3.2, footnote 1. Reported gains are measured on top of a MoCo initialisation. Behaviour from random encoder initialisation is not reported.
• Baselines are adapted. To match resolution, the baselines drop the last average pooling and use dilated convolutions §4.1. This is the authors' choice. The baselines' own designs would give different output resolutions.
• Detection uses a different FPN. The paper describes its FPN, but the detection experiments use Detectron2's default FPN, which the footnote states §4.3. The defaults were tuned for ImageNet-supervised pretraining §4.3. Both choices may favour the supervised baseline.
• Result reporting. Tables report means over 5 trials without standard deviations §4.1 §4.2 §4.3. The fine-tuning plot has standard deviations fig. 4. Claims of "no statistical difference" at 100% labels §4.2 are therefore not checkable from the tables alone.
• Small gap at full labels. At 100% NYU data, MoCo's RMSE (0.640) is marginally lower than VADeR's (0.641) app. A.3.
• Table header typo. The Mask R-CNN mask-AP50 column header reads AP^mk_55 in the source tab. 4.
• Figure, table and appendix numbers are counted from the LaTeX source. Table and figure numbers follow caption order; appendix labels A.1 to A.3 follow the section structure. Check all of them against the compiled PDF before publication.
• Privacy and misuse. The broader-impact statement names privacy and freedom-of-expression risks for surveillance-type uses of self-supervised image models §6.

Lineage

• Builds on: MoCo [he2019momentum] for the momentum encoder, the queue of negatives and the encoder initialisation (MoCo v2 is the one used) §3.2. Feature pyramid networks [lin2017feature] for the encoder-decoder §3.2. Noise-contrastive estimation [gutmann2010noise] and [oord2018representation] for the loss §3.1.
• Compared against: random initialisation, supervised ImageNet ResNet-50, MoCo v2 §4.1 §4.2 §4.3. For video propagation: SIFT Flow, Video Colorization, TimeCycle, CorrFlow and UVC tab. 2.
• Superseded by: not stated in this card's sources.