FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding

In plain words

Teaching a detector to find an object it has seen only a few times is hard. The paper's observation is that the detector usually finds the new object correctly, but then gives it the label of a similar-looking familiar class. The fix adds one extra learning signal during fine-tuning. Boxes that cover the same object well should get similar internal descriptions, and boxes from different classes should get dissimilar ones. The signal works like contrastive learning applied to the detector's box proposals.

The extra signal sits in a small added head, and training runs as an ordinary detector fine-tuning. The authors also make a few changes to how proposals are kept and sampled, which they describe as a stronger baseline before the contrastive part is added.

TL;DR

• Few-shot detection here means fine-tuning a Faster R-CNN (ResNet-101 + FPN) from a base-class model on a balanced set of novel shots §3 §4.
• Three changes: keep the ResNet frozen but train FPN, RPN and the RoI head; double the proposals kept after NMS and halve the RoI sample; and add a contrastive head with loss weight λ = 0.5 §3.1 §3.3.
• The largest gap the text reports is PASCAL VOC split 3 at 10 shots: 58.5 nAP50 against 49.7 for MPSR. The best unstarred baseline in that column is TFA w/ fc at 50.2, so the gap to it is 8.3 tab. 2.
• The released code differs from the paper in several places (loss normalisation, head depth, IoU threshold, λ schedule). See "Paper vs code".

Problem

In few-shot detection, a detector has many labelled base-class examples and only K labelled examples (K = 1, 2, 3, 5, 10 on PASCAL VOC) for each novel class §4.1. Earlier meta-learning detectors were beaten by fine-tuning methods, and the most cited baseline, TFA, fine-tunes only the last two layers, with the rest frozen §1 §3.1.

The paper argues the main error is classification, not localisation. Novel objects are found by the proposal network and boxed accurately, but are assigned to confusable base classes §1 fig. 2. Prototype cosine similarity between similar categories reaches 0.39 in the base model, and up to 0.59 in the few-shot setting; background averages −0.21 §1 fig. 2. Manually fixing misclassified but well-localised boxes would raise novel nAP by more than 20 points for the TFA baseline, by the authors' own estimate §1.

Margin-based classifiers (the paper cites CosFace and ArcFace) did not help in this data-scarce regime, by the authors' experiments §1 §3.2 §4.3.

Method

Training has two stages §3:

1. Base training. Train a standard Faster R-CNN on base-class data only. The paper gives no base-training hyperparameters. The code starts from ImageNet-pretrained ResNet-101 weights and trains 18,000 iterations on the 15 base VOC classes R101_FPN_base_training_split1.yml. A surgery script then resets the final classifier and box regressor for the full 20 classes, keeping base-class weights and randomly initialising novel ones (or initialising them from a predictor fine-tuned on novel data) TRAIN_INST.md. The paper does not describe this step.

2. Fine-tuning on a balanced set. Use K shots of every novel class plus randomly sampled base instances §3. The code draws the shots per seed (seeds 1 to 19 by default), sampling K instances per category prepare_voc_few_shot.py. The paper gives no seed.

   During fine-tuning the ResNet backbone is frozen. The FPN, the RPN and the RoI box head are trained §3 fig. 3. In the code this is done by FREEZE_AT: 5, which freezes every ResNet stage resnet.py while BACKBONE.FREEZE stays False 10shot_CL_IoU.yml.

   Strong-baseline changes §3.1 tab. 1:

   • keep twice as many RPN proposals after NMS (2000 for training, against 1000 in the base config) 10shot_CL_IoU.yml Base-RCNN-FPN.yaml;
   • sample half as many RoIs for the loss (256 per image, against the default 512) 10shot_CL_IoU.yml defaults.py. The authors say the discarded half contains only background RoIs in fine-tuning §3.1.

Contrastive proposal encoding. For each sampled RoI, the RoI feature x (post-ReLU, dimension 1024 for FPN Faster R-CNN §3.2) feeds a contrastive head. It outputs an encoding z of dimension D_C = 128 by default §3.2. The classifier is a scaled cosine classifier, with logit α·cos(x, w_j) and α = 20 §3.2 eq. 1. The contrastive loss is computed on z, and the total loss adds it to the usual detector losses §3.3 eq. 5.

The CPE loss averages a supervised contrastive term over RoIs whose IoU with their matched ground truth reaches a threshold φ §3.3 eq. 2 eq. 3 eq. 4. Within a mini-batch, RoIs with the same label are positives and all other RoIs in the batch are negatives. Distances use the cosine similarity of normalised encodings divided by a temperature τ.

# CPE loss for one mini-batch of sampled RoIs  [eq. 2] [eq. 3] [eq. 4]
# Input:  features (M, D_C), L2-normalised;  labels (M,);  ious (M,)
# Output: scalar loss

def cpe_loss(features, labels, ious, tau, phi):
    # Cosine similarity of every pair, divided by temperature  [eq. 3]
    similarity = features @ features.T / tau         # (M, M)
    # Mask that is 1 for every pair with j != i  [eq. 3]
    not_self = 1 - identity(M)                       # (M, M)
    # Mask that is 1 for pairs with the same label, excluding self  [eq. 3]
    same_label = (labels[:, None] == labels[None, :]) * not_self
    # Softmax over k != i in the denominator  [eq. 3]
    exp_sim = exp(similarity) * not_self
    log_prob = similarity - log(exp_sim.sum(dim=1, keepdim=True))
    # Average over the other positives of each RoI, N_yi - 1  [eq. 3]
    num_positive = same_label.sum(dim=1).clamp(min=1)  # guards 0 / 0
    per_roi = -(log_prob * same_label).sum(dim=1) / num_positive
    # Keep RoIs with IoU >= phi, weight g = 1  [eq. 4]
    keep = ious >= phi
    # Divide by all N RoIs in the batch, as the paper writes it  [eq. 2]
    return (per_roi * keep).sum() / M

Notes on the pseudocode: the paper's denominator has N_yi − 1 positives, which is undefined for an RoI that has no other positive. The code instead keeps such RoIs and gives them zero loss. See Pitfalls.

Proposal consistency. Only RoIs with IoU at least φ are used as anchors §3.3 eq. 4. The paper's reweighting function g is not specified in the main text. The ablation shows that a hard cut-off with g = 1 works best at 5 and 10 shots tab. 7. The code default is none, which equals g = 1 defaults.py.

Total loss §3.3 eq. 5: L = L_rpn + L_cls + L_reg + λ·L_CPE, with λ = 0.5.

Hyperparameters

Results

Author-reported. Setup: PASCAL VOC, Faster R-CNN with ResNet-101 + FPN; base classes 15, novel classes 5; novel nAP50 on the VOC 2007 test set, averaged over three novel splits §4.1. Starred rows average over 10 random seeds. The paper compares against TFA, MPSR and several meta-learning methods tab. 2.

Table 1: Strong baseline, novel AP50 on PASCAL VOC split 1 tab. 1.

Table 2: Few-shot detection on PASCAL VOC, nAP50 tab. 2. "†" = meta-learning; "‡" = N-way K-shot meta-testing protocol (different from the others); "⋆" = average over 10 seeds. The source LaTeX contains the typo 42..2 for one FSIW⋆ value, read here as 42.2.

The unstarred FSCE row carries no seed note in the table, so whether it averages seeds is not stated.

Table 3: Few-shot detection on COCO tab. 3. Novel AP and AP75. "†" = meta-learning; "⋆" = average over 10 seeds.

Table 4: Base forgetting on PASCAL VOC split 1 tab. 4. Base AP50 before fine-tuning is 80.8.

Table 5: Components, novel nAP50 on PASCAL VOC split 1 tab. 5.

Table 6: Contrastive head dimension and temperature, 10 shots, split 1 tab. 6.

Table 7: Proposal consistency options, novel nAP on PASCAL VOC split 1 tab. 7.

Headline claims in the text. The authors say FSCE is the first to exceed 50 nAP50 on splits 2 and 3 §4.2. They also say the gain is up to +8.8 nAP50 on split 3 §4.2. That matches the 10-shot cell against MPSR (58.5 vs 49.7) in Table 2, but TFA w/ fc is higher than MPSR in that column (50.2), so the gap to the best unstarred row is 8.3. On COCO, the abstract's +2.7 matches the 30-shot nAP cell against TFA w/ cos (16.4 vs 13.7). Against MPSR (14.1) it is +2.3. The body's +1.7 nAP §4.2 matches no row checked; the 10-shot gap to TFA w/ cos is +1.9 tab. 3. On split 1 at 3 shots, FSCE ties MPSR at 51.4 tab. 2, so "best on every split" is strictly "best or tied".

Paper vs code

Searched: the abstract links the repository MegviiDetection/FSCE, which is the authors' organisation. It was cloned at commit 94ba3f4737d5e40af795db49b8c6914526c912b6, Apache-2.0 LICENSE. The repository is built on FsDet/Detectron2 and contains the training loop, the contrastive loss and the configs.

1. Loss normalisation. The paper divides the CPE sum by N, the number of RoIs in the batch §3.3 eq. 2. The code averages over the RoIs that pass the IoU filter contrastive_loss.py. Results with fewer passing RoIs get a larger effective loss.
2. Positive count. The paper uses N_yi − 1, the other same-label RoIs §3.3 eq. 3. The code divides by the count including self contrastive_loss.py.
3. Head depth. The paper describes a 1-layer MLP §3.2. The code uses two linear layers with a ReLU between them, then L2 normalisation contrastive_loss.py.
4. IoU threshold φ. The paper says φ = 0.7 is a good cut-off §3.3, and Table 7 favours φ = 0.7 at 5 and 10 shots tab. 7. The main configs use φ = 0.8 for 10 shots on split 1 10shot_CL_IoU.yml, 0.7 for 5 shots 5shot_CL_IoU.yml, 0.6 for 3 shots 3shot_CL_IoU.yml, and 0.5 for split 2 10shot_CL_IoU.yml. A φ = 0.7 ablation config exists 10shot_CL_phi07.yml. The 10-shot value 63.4 in Table 2 matches the φ = 0.7 row of Table 7, not the φ = 0.8 config.
5. Loss weight λ. The paper sets λ = 0.5 §3.3. The code uses 0.2 for split 1 at 3 shots 3shot_CL_IoU.yml, 0.1 for split 2 at 2 shots 2shot_CL_IoU.yml, and 0.5 for the other configs read.
6. λ schedule. Every contrastive config read enables a decay that multiplies λ by 0.5 at listed iterations 10shot_CL_IoU.yml roi_heads.py. The paper does not mention it.
7. Temperature default. The code default is 0.1 defaults.py. All contrastive configs read use τ = 0.2, which matches the paper tab. 6.
8. Which 3-shot setting. Table 2 and Table 5 give 51.4 for split 1 at 3 shots. Table 7 gives 51.4 for the φ = 0 weighting g(x) = x, and 50.8 for the φ = 0.7 hard clip. The repository's 3-shot config uses φ = 0.6 and λ = 0.2. The paper does not say which setting produced 51.4.
9. Backbone. The paper freezes the backbone §3 fig. 3. The code does this through FREEZE_AT: 5. Naming is misleading: BACKBONE.FREEZE stays False in the configs, and that flag would also freeze FPN rcnn.py.
10. Input and test-time settings. Multi-scale training (480 to 800), crop and test-time augmentation are on in the configs 10shot_CL_IoU.yml. The paper mentions none of them.
11. Missing configs. The paper reports FSCE at 1 shot for every split and at 2 shots for splits 1 and 3 tab. 2. The repository has no contrastive config for those cells: 1-shot on all splits, and 2-shot on splits 1 and 3 (only baseline configs exist). COCO has only baseline configs 10shot_baseline.yml 30shot_baseline.yml. COCO FSCE and the 1-shot and split 1/3 2-shot VOC cells cannot be reproduced from the released configs.
12. Table 4 vs Table 2. TFA w/ cos on split 1 reads 44.6 and 55.6 at 3 and 5 shots in Table 4, but 44.7 and 55.7 in Table 2. Both are reported as is.
13. Few-shot sample seeds. The code draws seeds 1 to 19 prepare_voc_few_shot.py. The paper gives no seed, and its results averaging over seeds are the starred rows.

No code for the paper's COCO results is in the repository, so the COCO rows of Table 3 are paper-only.

Pitfalls

• Empty batches. If no RoI in the batch passes φ, the code averages an empty tensor, which gives NaN contrastive_loss.py. Guard this case in a re-implementation.
• RoIs with no other positive. The paper's N_yi − 1 is zero for a singleton class, which is undefined. The code gives such RoIs zero loss but still counts them when averaging contrastive_loss.py. The pseudocode above skips them in the keep mask; choose one explicitly.
• Hard-coded GPU. The label mask calls .cuda() contrastive_loss.py. The loss fails on CPU or other devices unless you change it.
• Memory. The similarity is an M × M matrix over all sampled RoIs. With 16 images and 256 RoIs each, M = 4096 and the matrix has about 16.8 million entries, computed from the config values, not measured 10shot_CL_IoU.yml. The paper calls the head's cost negligible §3.2; the pairwise matrix is the larger cost.
• Background RoIs. Background RoIs are negatives in every denominator, but never anchors when φ is at least the 0.5 foreground threshold, because their IoU is below 0.5 defaults.py contrastive_loss.py.
• IoU of proposals, not boxes. φ is applied to the IoU of each sampled proposal with its matched ground truth, computed at sampling time, not to the final regressed box roi_heads.py roi_heads.py.
• Contrastive head at test time. The encoder also runs at inference, but its output is discarded roi_heads.py.
• Fixed cosine scale. α = 20 comes from the default, and the code also supports a learnable scale. The configs keep the default defaults.py.
• Iteration counts. The paper says steps scale with shots but gives no values. The code uses 15,000, 9,000 and 8,000 iterations for 10, 5 and 3 shots on split 1. Compare results only under the same schedule.
• Averages. Only the starred rows are stated to average over 10 seeds. The paper says variance across random runs is large §4.1 and reports averaged results only where a method provides them.
• COCO. The paper's COCO setting uses 5,000 images from COCO 2014 val §4.1, but the released repository has no FSCE COCO config to check that.

Lineage

• Builds on: TFA, the two-stage fine-tuning baseline, cited by the repository README as arXiv:2003.06957 README.md §3.1 §2. Also Faster R-CNN with FPN §3 §4, and supervised contrastive learning, cited in code as arXiv:2004.11362 contrastive_loss.py.
• Compared against: TFA (fc and cos), MPSR, FSIW, Meta R-CNN, MetaDet, FSRW, RepMet, LSTD, FRCN-ft and YOLOv2-ft tab. 2 tab. 3.
• Superseded by: not stated.