wikillm

Paper · 2019-02-17 · cs.CV

Exploiting Unlabeled Data in CNNs by Self-supervised Learning to Rank

Trains CNN regressors with a ranking task on unlabeled data: images are ordered by known distortion level (IQA) or by nested crops (crowd counting), and a Siamese hinge loss is trained jointly with the regression loss. A pair-all-in-batch backprop makes Siamese training cheaper; ranking uncertainty drives active labeling. Author-reported: SROCC 0.806 on TID2013, MAE 279.6 on UCF_CC_50.

Learning-to-rank losses over candidate setsSelf-supervised learningSiamese networkActive learningcomputer-visionself-supervisedranking

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Labeling images for quality scores or head counts is slow. This paper notices that some comparisons are free. If you blur a photo more, its quality is lower, so you know the order of two blurred versions without anyone rating them. If a crop sits inside a larger crop of a crowd photo, the larger one cannot contain fewer people. Training a network to get such orderings right uses unlabeled images, and it helps the main task.

The paper trains one network on two goals at once: give the right number for labeled images, and put unlabeled images in the right order. It also shows a cheaper way to compute the order loss, by using every pair in a batch from one forward pass. Finally, the network's uncertainty about the order tells you which unlabeled images are worth labeling first.

TL;DR

Ranking can serve as a self-supervised proxy task for regression. The paper applies it to no-reference image quality assessment (IQA) and to crowd counting, trains regression and ranking jointly with a weighted sum, and uses a single forward pass per batch for all Siamese pairs. The same uncertainty score drives active learning. Results are author-reported.

Problem

Regression from images needs many labels, and labels are costly. For IQA, each image needs several human opinions §1. For crowd counting, every person must be annotated §5. Fine-tuning from a ranking pre-training was found inferior to joint multi-task training in the authors' earlier work §3.2. Siamese ranking training repeats computation for each image in several pairs, and hard-pair mining adds cost §3.3.

Method

Ranking proxy task §3.1. Let a regression target be a function f of image \mathbf{x}. Suppose a function h(\cdot, \varphi) exists whose parameter is ordered and such that a larger parameter never increases the true score. Then pairs of parameters give pairs of images with a known order, and no label is needed. Two instances are used in the paper:

  • IQA: \varphi is a distortion level. More distortion means lower quality §4.2.
  • Counting: \varphi is a rectangle. A crop contained in another crop has no more people than the larger one §5.2.

Training uses a hinge loss on the ordered pair (lower parameter should score lower by at least a margin \varepsilon). The paper gives no numeric value for \varepsilon §3.1 (see Paper vs code).

Multi-task objective §3.2. The labeled regression loss is a squared error over the labeled set. The total loss adds the ranking hinge over the unlabeled set, weighted by \lambda. The regressor and both Siamese branches share one network and one set of weights §3.2. The paper considers this joint form only, not pre-training followed by fine-tuning §3.2.

Efficient Siamese backpropagation §3.3. A standard Siamese implementation passes both images of each pair through the network, so an image appears in several pairs and is computed several times. The proposed method passes each image through the network once per mini-batch. It then forms all comparable pairs in the loss layer. With n ranked images from one reference, the standard approach needs n^2 - n forward passes and this one needs n; the stated speed-up is n - 1, which is of the order of the batch size. The paper expects the convergence gain to be smaller because pairs in a batch are highly correlated §3.3.

Gradient structure §3.3: the gradient is a sum over pairs, and for the hinge it can be written as one matrix product in which entry (i, j) is the orientation l_{ij} when pair (i, j) is active and zero otherwise. Pairs are comparable only if they come from the same source image §3.3.

Pseudocode
# Pairwise ranking hinge over one mini-batch  [§3.3] [eq. ranking]
# Input:  scores (M,)  one scalar per image from one forward pass
#         phis (M,)    proxy parameter of each image
#         source (M,)  id of the source image each sample came from
#         margin       hinge margin, called epsilon in the paper
# Output: loss (scalar)

def ranking_loss(scores, phis, source, margin):
    # Visit each unordered pair of images once  [§3.3]
    loss = 0
    for i in range(M):
        for j in range(i + 1, M):
            # Only parameters on the same source image are comparable  [§3.3]
            if source[i] != source[j] or phis[i] == phis[j]:
                continue
            # Put the lower-parameter image first  [§3.1] [§3.3] [eq. l_ij]
            if phis[i] < phis[j]:
                low, high = i, j
            else:
                low, high = j, i
            # The higher-parameter image must score above by margin  [§3.1]
            loss += max(0, scores[low] - scores[high] + margin)
    return loss

Note: the paper's loss in §3.1 is written as a sum, and its efficient form in §3.3 is the same sum over all comparable pairs. The loop above is the reference definition; a vectorised version gives the same value.

Active learning §3.4, alg. 1. A certainty score C(\mathbf{z}) is the fraction of K sampled parameter pairs for which the network orders the two transformed images correctly (strict inequality, so ties count as errors) §3.4 eq. ranked. The loop trains on the labeled set, scores the unlabeled pool, and labels the S least-certain images, repeated for T cycles alg. 1.

Pseudocode
# Active learning loop  [alg. 1] [§3.4]
# Input:  labeled set D, unlabeled pool E, cycles T, batch size S, theta
# Output: trained theta

def active_learning(D, E, T, S, theta):
    for t in range(T):
        # Train on the current labeled set  [alg. 1]
        theta = train(D, theta)
        # Certainty = fraction of K sampled pairs ordered correctly  [§3.4]
        certainty = {z: certainty_score(z, theta, K) for z in E}
        # Label the S least-certain images first  [§3.4] [alg. 1]
        chosen = sorted(E, key=lambda z: certainty[z])[:S]
        D = D + chosen
        E = E - chosen
    return theta

IQA instantiation §4.

  • Backbone: VGG-16 pre-trained on ImageNet. The head is replaced by one fully connected layer with a single output §4.3 fig. 2.
  • Regression loss: mean squared error on labeled images §4.3 eq. IQA loss.
  • Crops: random square sub-images, 224 \times 224, taken from the original images, which are 300 to 700 pixels wide. Cropping rather than scaling avoids interpolation artefacts §4.3.
  • Ranked data: the Waterloo dataset (4,744 images) with 4 distortion types (Gaussian blur, Gaussian noise, JPEG, JPEG2000) at five levels for LIVE (20 distortions per image). For TID2013, 17 of 24 types at five levels (85 per image) §4.2 app. A.

Counting instantiation §5.

  • Baseline density network: VGG-16 pre-trained on ImageNet. Its three fully connected layers and pool5 are removed, and one 3 \times 3 \times 512 convolution with stride 1 regresses the density map §5.3.
  • Density targets: a Gaussian of standard deviation 15 pixels per head, summed over heads §5.1.
  • Counting loss: mean squared error on density maps §5.3 eq. L_c.
  • Multi-scale training: square patches of 56 to 448 pixels §5.3.
  • Ranking head: a sum-pooling layer over the density map gives a scalar count for each crop §5.3 fig. 4.
  • Ranked crop sets §5.2 alg. 2:
    1. Pick an anchor point at random inside an anchor region.
    2. Take the largest square patch around the anchor that fits the image.
    3. Take k - 1 further square patches around the same anchor, each smaller by factor s.
    4. Resize all k crops to the network input size. The output is a list ordered by person count.
  • Data: keyword-query images (1,180 crowd images, §5.2) and query-by-example images (256, 2229 and 3819 images for UCF_CC_50, ShanghaiTech Part A and B). No head annotations were added to these sets §5.2.

Crowd-count ranking pseudocode alg. 2.

Pseudocode
# Ranked crop set from one unlabeled crowd image  [alg. 2] [§5.2]
# Input:  image, k (crops), s (scale factor), r (anchor region divisor)
# Output: k crops ordered by person count (smallest first)

def ranked_crops(image, k, s, r):
    # Anchor lies in the central region of size 1/r of the image  [alg. 2]
    anchor = random_point(central_region(image, 1 / r))
    # Largest square around the anchor that fits in the image  [alg. 2]
    size = largest_square_fit(image, anchor)
    crops = []
    for n in range(k):
        # Each crop is smaller by factor s, same centre  [alg. 2]
        crops.append(square_crop(image, anchor, size * s ** n))
    # Resize every crop to the network input  [alg. 2]
    return [resize(c, input_size) for c in reversed(crops)]

Hyperparameters for k, s, r are given in §6.2 (see the table below). The order in alg. 2 is the order the paper uses for the ranking target. The paper does not say explicitly which end of the list is "smallest"; the output description implies that the list is ordered by person count, so smaller crops come first.

Hyperparameters

All values are as stated in the paper. "Code" entries are from the IQA repository, not the paper; see Paper vs code.

NameValueWhereNotes
optimizerSGD§6.1 §6.2momentum, not stated in the paper
weight decay5 \times 10^{-4} (L2)§6.1 §6.2both training phases
IQA learning rate, Siamese training10^{-4}§6.1
IQA learning rate, fine-tuning10^{-6}§6.1
IQA multi-task learning rate10^{-6} LIVE, 10^{-5} TID2013§6.1
IQA schedule×0.1 every 10,000 iterations, 50,000 total§6.1
IQA \lambda (multi-task)1§6.1authors say cross-validation would likely help
IQA sub-image size224 \times 224§4.3 §6.1random crop from the original
IQA mini-batch (multi-task)72 images§6.1equal compute to the comparison methods
IQA random pair baseline36 pairs per mini-batch§6.1
IQA test-time crops30 random sub-images, averaged§6.1
IQA sub-images per epoch1 per training image§6.1
Counting batch size25 (ranking and counting), 50 multi-task§6.2
Counting learning rate10^{-6} (ranking + fine-tune)§6.2
Counting \lambda (multi-task)100§6.2same on all datasets
Counting schedule×0.1 every 5,000 iterations, 10,000 total§6.2
Ranked crops k5§6.2 alg. 2
Scale factor s0.75§6.2 alg. 2
Anchor region r8§6.2 alg. 2"1/r the size of the image"
Density Gaussian sigma15 px§5.1
Multi-scale patch size56 to 448 px§5.3
Active learning, K100§6.1 §6.2sampled pairs for certainty
Active learning, initial labeled fraction10%§6.1 §6.2
Active learning, cycles T9§6.1 §6.2
Active learning, batch S10% of the training set per cycle§6.1 §6.2
Ranking margin \varepsilonnot stated§3.1code uses 10, see Paper vs code
Ranking loss normalisationnot stated§3.1 §3.3code divides by batch size
Optimizer momentum, LR warmupnot stated—
FrameworkCaffe§6

Results

Author-reported. Setups differ between tables; each row keeps its own protocol.

IQA, TID2013, ablation (80/20 split by reference image, averaged over 10 runs) tab. 1:

MethodLCCSROCC
Baseline (fine-tune VGG-16)0.6630.612
RankIQA (ranking only)0.5660.623
RankIQA+FT (random pairs)0.7750.738
RankIQA+FT (hard pairs)0.7820.748
RankIQA+FT (ours)0.7990.780
MT-RankIQA (random pairs)0.8020.770
MT-RankIQA (hard pairs)0.8100.779
MT-RankIQA (ours, efficient Siamese)0.8270.806

IQA, TID2013, SROCC per protocol tab. 2: ALL distortions: MT-RankIQA 0.806; RankIQA+FT 0.780; HOSA 0.728; CORNIA-10K 0.651; BRISQUE 0.562; BLIINDS-II 0.550. The paper reports MT-RankIQA as the best on ALL. It also states that HOSA is better on 6 of 24 distortions §6.1.

Generalisation to unseen distortions tab. 3: overall LCC rises from 0.687 (baseline) to 0.829 (all distortions). Gains on unseen distortions grow from +0.016 (7 distortions) to +0.075 (all).

IQA, LIVE, ALL distortions, NR-IQA, 80/20 split, averaged over 10 runs tab. 4: MT-RankIQA LCC 0.976 and SROCC 0.973. The paper states that the best prior NR-IQA results on ALL are SOM and DNN §6.1.

Crowd counting, UCF_CC_50 ablation (five-fold, average MAE) tab. 5:

MethodAve MAE
Basic CNN454.45
+ ImageNet pre-training328.54
+ multi-scale308.06
Ranking + FT325.73
Multi-task (random pairs)292.67
Multi-task (hard pairs)287.96
Multi-task (ours)279.60

UCF_CC_50 MAE/MSE tab. 6: Multi-task (keyword set) MAE 279.6, MSE 408.1; Multi-task (query-by-example) MAE 291.5, MSE 397.6. The strongest listed prior methods are ic-CNN (MAE 260.9) and CSRNet (MAE 266.1) tab. 6.

ShanghaiTech tab. 7:

SettingPart A MAEPart A MSEPart B MAEPart B MSE
Multi-task (query-by-example)72.0106.614.423.8
Multi-task (keyword)73.6112.013.721.4
CSRNet (prior)68.2115.010.616.0

WorldExpo'10, MAE per scene tab. 8: Baseline average 12.5; ours average 10.5 (outperforms the baseline on all five scenes). ACSCP has the best average (7.5).

UCSD tab. 9: Baseline MAE 1.60, MSE 2.13; ours MAE 1.17, MSE 1.55; ACSCP MAE 1.04 (best).

UCF-QNRF tab. 10: Baseline MAE 137, MSE 228; ours MAE 124, MSE 196. The prior CompositionLoss gives MAE 132 and MSE 191 tab. 10.

Transfer, ShanghaiTech Part A to UCF_CC_50 tab. 11: counting only MAE 349.5, MSE 475.7; multi-task MAE 337.6, MSE 434.3.

Active learning. On TID2013, the paper reports that 20% labeled data with its selection matches 50% labeled data with random selection §6.1, which the paper describes as a 75% labeling-cost reduction. On ShanghaiTech Part A it reports that 20% with its selection matches 40% at random, a 50% reduction §6.2 fig. 5 fig. 7.

Paper vs code

The paper's "Code" footnotes are commented out in the LaTeX source §6.1. The owner xialeiliu is the first author. The repository https://github.com/xialeiliu/RankIQA is the author's IQA code, under MIT (LICENSE: "MIT License, Copyright (c) 2017 xialeiliu"). Its README describes it as the code for the earlier ICCV 2017 RankIQA paper, and it also cites this paper's TPAMI version. We cloned 22ca65cd0156b5b428cecd55ed939366fb64d2e5.

The repository's scope is ranking-only and fine-tuning for IQA. It has no multi-task (MT-RankIQA) training code. Comparison below is against the IQA parts of the paper §4 §6.1.

  • Ranking learning rate, LIVE. Paper: 10^{-4} for Siamese training §6.1. Code: base_lr: 0.00001 solver_vgg.prototxt. The TID2013 ranking solver uses 10^{-4} solver_vgg.prototxt, which matches the paper.
  • Ranking schedule. Paper: ×0.1 every 10,000, 50,000 total §6.1. Code LIVE ranking: stepsize 10,000, gamma 0.1, max_iter 50,000 solver_vgg.prototxt. Matches.
  • Fine-tuning learning rate and schedule. Paper: 10^{-6}, ×0.1 every 10,000, 50,000 total §6.1. Code LIVE fine-tune: base_lr: 0.000001, gamma 0.5, stepsize 5,000, max_iter 20,000 solver_live.prototxt. Code TID2013 fine-tune: base_lr: 0.00001, gamma 0.5, stepsize 5,000, max_iter 10,000 solver_vgg.prototxt. Both differ from the paper's stated schedule, and the TID2013 learning rate is ten times the paper's 10^{-6} for fine-tuning.
  • Weight decay. Paper 5 \times 10^{-4} §6.1; code 5 \times 10^{-4} in all four solvers solver_vgg.prototxt. Match.
  • Margin. The paper gives no value §3.1. Code: self.margin = 10 netloss_live.py.
  • Level and image counts per mini-batch. The paper's LIVE setup uses five distortion levels §4.2. Code uses 6 levels, 4 distortions and 2 images per (distortion, level) cell, giving 48 images per batch netloss_live.py rank_layer_live.py train_vgg.prototxt. The paper's multi-task IQA batch is 72 images §6.1, and that setting is not in this repository.
  • Pair construction. The paper forms all comparable pairs in the batch §3.3. The code loops over each distortion block of 12 images and pairs images from different levels of the same block netloss_live.py. Whether the two images in a pair share a source image depends on the data files; this was not checked against the data lists.
  • Normalisation. The paper's ranking loss has no normalisation §3.1. Code divides the summed hinge by the batch size netloss_live.py.
  • Gradient. Code's backward pass adds -1 to the lower-ranked image and +1 to the other, for each active pair, divided by batch size netloss_live.py. This is the sign convention in §3.3 and consistent with the loss in the same file.
  • Crop. Code takes random 224 × 224 crops rank_layer_live.py with np.random.randint(x - sp). The upper bound is exclusive, so the last valid start position is never sampled. This is a small deviation from "random sub-image" in §4.3.
  • Head. The IQA network has one fully connected output train_vgg.prototxt, as in §4.3. Dropout ratio 0.5 is present in the model train_vgg.prototxt and is not mentioned in the paper.
  • Test protocol. The paper averages 30 random sub-images per image §6.1. The evaluation scripts in src/eval/ compute LCC and SROCC over the set; no sub-image averaging with 30 crops was found in the scripts we read.

Counting code. The counting code is in a separate repository, https://github.com/xialeiliu/CrowdCountingCVPR18, owned by the first author. It is described as the code of the CVPR 2018 paper on the same topic §5. We did not clone it: the card's single code field points to the IQA repository, and the counting repository has no license stated on its page. Counting hyperparameters are therefore not checked against code, and no discrepancies are claimed for the counting part.

Pitfalls

  • Margin is missing. The hinge margin \varepsilon is never given §3.1. The code uses 10 netloss_live.py. This is a scale-dependent choice, because the regression output has its own scale.
  • Loss normalisation is not stated. Summing versus averaging changes the effective learning rate of the ranking term relative to \lambda §3.2. The code averages over batch size netloss_live.py.
  • Multi-task batch differs by task. IQA: 72 images §6.1. Counting: batch 25 for each loss, 50 in total §6.2. The table above follows the paper; the two numbers are not interchangeable.
  • \lambda differs by task and is not tuned. IQA uses 1 §6.1; counting uses 100 §6.2. The authors say held-out cross-validation is expected to help §6.1 §6.2.
  • Anchor region is ambiguous. "The anchor region is 1/r the size of the original image" alg. 2 could mean side length or area. With r = 8 the two readings give very different spans. The paper does not decide.
  • Crop order. alg. 2 does not say which end of the output list is the smallest count. Check the sign against \varphi before implementing.
  • Ranking-only training can latch onto the wrong thing. The authors report that Ranking+FT is worse than plain ImageNet fine-tuning for counting (MAE 325.7 vs 328.5 tab. 5, roughly equal) and explain that the proxy task may learn arbitrary features such as hats, trees or edges §6.2.
  • Hard-pair mining cost. The paper states that hard mining increases total computation by up to 80% for the cited work §3.3. The paper's own method adds no forward passes but its batch structure must be kept intact.
  • Efficient Siamese speed-up is an upper bound. The n - 1 speed-up is for forward passes §3.3. The paper expects a smaller gain in convergence because pairs within a batch are highly correlated §3.3.
  • Unlabeled data is not released. Keyword and query-by-example sets are collected from web image search §5.2. The paper does not say they are published, and the images may carry third-party rights. Do not assume the datasets are available; build your own sets from data you may use.
  • Active learning claims differ. The abstract says "up to 50%". The IQA section says 75% §6.1. The counting section says 50% §6.2. Use the per-task numbers.
  • Active learning warm-starts each cycle. alg. 1 trains each cycle from the previous cycle's parameters, \theta_t \leftarrow \mathbf{train}(D, \theta_{t-1}). The paper does not say how many epochs or iterations each cycle runs.
  • Density map sigma is fixed. A Gaussian of standard deviation 15 px per head §5.1 is a choice that changes with head size and image resolution. The paper does not discuss it.
  • Density target at multiple scales. Multi-scale training uses 56 to 448 px patches §5.3. The counting label is a sum over the patch, so the targets must be recomputed per crop.
  • Two-image comparison requires one source. Only crops of the same image are comparable §3.3. Mixing crops from different images in one batch without this check gives false orderings.
  • Dataset provenance. IQA ranked data uses the Waterloo image set §4.2. TID2013 has distortion types the authors could not reproduce (#12, #13, #20, #21, #24) and others they reproduced approximately (#03, #04, #09 settings unspecified) app. A. Use the appendix mapping before claiming a TID2013 distortion is reproduced.

Lineage

  • Builds on: the authors' ICCV 2017 RankIQA paper (ranking pre-training then fine-tuning for IQA) and CVPR 2018 crowd-counting ranking work, both cited in the paper §1.
  • Related: Siamese hard-pair mining §3.3; N-pair loss and a smooth upper-bound loss, which the paper says were developed concurrently §3.3.
  • Compared against: MCNN, Switching-CNN, CP-CNN, ACSCP, CSRNet, ic-CNN, Cross-scene, CompositionLoss (counting); BRISQUE, CORNIA, HOSA, SOM, DNN, CNN (IQA) tab. 2 tab. 4 tab. 5 tab. 6.
  • Superseded by: not stated on the card's sources.

Source and license

Adapted from “Exploiting Unlabeled Data in CNNs by Self-supervised Learning to Rank” by Xialei Liu, Joost van de Weijer, Andrew D. Bagdanov (arXiv:1902.06285), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/xialeiliu/RankIQA at commit 22ca65cd0156 (MIT). Any quoted code is shown under that license.