Long-tailed recognition

Overview

Many real datasets are long-tailed: a few classes have many training instances and many classes have only a handful. Plain training then favours the frequent classes. On LVIS, single-stage Eff-B7 FPN (640) reaches 9.7 AP_r on rare classes against 35.1 AP_f on frequent ones 2012.07177 tab. 8. Detection and segmentation results are reported per group as AP_r, AP_c and AP_f. Classifiers are reported per split as Many, Medium and Few accuracy 2203.14197 tab. 1.

Eight cards cover eight settings. On LVIS instance segmentation, the rare-class gains come from a second training stage and from Copy-Paste augmentation 2012.07177 §4.7 2012.07177 §3. On LVIS detection with a sigmoid classifier, the EQL v2 authors reweight the classification loss inside a single end-to-end stage 2012.08548 §3.2. On image classification, weight balancing retrains only the last layer after a normal training stage 2203.14197 §3.3. Distribution Alignment instead learns a per-class scale and offset on the frozen classifier scores, which lifts ResNet-50 ImageNet-LT accuracy from 41.6% to 51.3% 2103.16370 tab. 1. On off-road LiDAR and image segmentation, the authors attribute weak results mainly to class imbalance and unclear boundaries 2011.12954 §V-B, and for the point-cloud model they report a label-distribution sampling gain 2011.12954 §V-C. The scene graph survey notes that frequent predicates dominate plain recall, and recommends mean recall per predicate 2104.01111 §6.2. GPS-Net instead softens a predicate frequency prior and gates it per pair, and reweights a focal-style loss by each object's share of triplets 2003.12962 §3.2 2003.12962 §3.3. In web image-caption pre-training, a looser filter keeps more rare words, and the authors report the gain after fine-tuning 2102.08981 §1 2102.08981 tab. 3.

Measure the rare-class number next to the overall one. Scene graph recall can look high while mean recall is low 2104.01111 §6.2. Each paper uses its own benchmark and backbone. EQL v2 reports LVIS v1.0 with Mask R-CNN R-50 at 1x 2012.08548 tab. 1, and Copy-Paste reports LVIS v1.0 with Eff-B7 FPN at 640 2012.07177 tab. 8. Compare numbers only within one paper's table. In the EQL v2 card, rows marked as copied are taken from other papers 2012.08548 tab. 4. The RELLIS-3D authors' imbalance explanation is not tested by an ablation 2011.12954 §V-B.

Variants

Two-stage classifier retraining with class-balanced loss

Stage 1 trains the ordinary single-stage model. Stage 2 fine-tunes only the final classification layer of Mask R-CNN, using the classification loss alone and a 3x schedule 2012.07177 §4.7. Each class gets a weight from its instance count n_c 2012.07177 §4.7, footnote:

w_c = \frac{1-\beta}{1-\beta^{n_c}}, \quad \beta = 0.999

The weights are divided by their mean and clipped to [0.01, 5] 2012.07177 §4.7, footnote. On Eff-B7 FPN (640), the two-stage model reaches 25.0 AP_r, against 9.7 for the single-stage baseline [arxiv:2012.07177 tab. 8] 2012.07177 tab. 9. Card: Simple Copy-Paste.

Repeat Factor Sampling

The paper uses Repeat Factor Sampling (RFS), with threshold t = 0.001, as a second rare-class method 2012.07177 §4.7. In the single-stage setting it lifts AP_r from 9.7 to 15.4 2012.07177 tab. 8. In the two-stage setting it lowers AP_r from 25.0 to 21.8 2012.07177 tab. 9. The card does not describe the sampling mechanism, so read the paper's §4.7 before implementing it. Card: Simple Copy-Paste.

Copy-Paste augmentation

Copy-Paste builds each training example from two images, pastes a random subset of instance masks from one onto the other, and updates the occluded masks and boxes 2012.07177 §3. It needs instance masks, not only boxes 2012.07177 §3. The paper presents it as general augmentation for instance segmentation; the rare-class gain is a measured result, not the stated goal 2012.07177 §1. On LVIS it raises AP_r for every two-stage model in the paper's comparison, by +2.7 to +6.1 2012.07177 tab. 7. On Eff-B7 FPN (640) it raises AP_r from 9.7 to 12.8 single-stage 2012.07177 tab. 8 and from 25.0 to 27.3 two-stage 2012.07177 tab. 9. Card: Simple Copy-Paste.

Equalization loss v2 in end-to-end detection

EQL v2 changes the classification loss of a sigmoid detector and does not resample the data. Each class keeps a running ratio of accumulated positive to negative gradient magnitude. A sigmoid of that ratio sets the weight of negatives, and positives get a larger weight as negatives are suppressed 2012.08548 §3.2 2012.08548 eq. 5 2012.08548 eq. 6. The paper sets α = 4, γ = 12 and μ = 0.8 2012.08548 §4.2. Training is end-to-end with a random sampler and no second stage 2012.08548 tab. 1. On LVIS v1.0 with Mask R-CNN R-50 at 1x, the reported figures are 23.7 AP and 14.9 AP_r. The same table gives 18.6 and 2.1 for EQL, and 16.1 and 0.0 for softmax 2012.08548 tab. 1. The best decoupled row, BAGS with 12 plus 12 epochs, reaches 23.1 AP and 13.1 AP_r 2012.08548 tab. 1. The ablation adds the parts one at a time: an extra category-agnostic objectness output gives 18.1 AP, the negative term 19.7 and the positive term 23.7 [arxiv:2012.08548 tab. 2]. The paper reports a 2C-softmax classifier as comparable, at 23.7 AP and 14.9 AP_r 2012.08548 tab. 9. On Open Images with a ResNet-50 detector, the LVIS hyperparameters give 52.6 mAP@0.5 against 43.1 for Faster R-CNN 2012.08548 tab. 10. Card: Equalization Loss v2.

Weight balancing in image classification

Weight balancing changes no loss in the feature stage. Stage 1 trains the backbone with cross-entropy and tuned weight decay. Stage 2 freezes the features and retrains the classifier with a class-balanced loss, weight decay and MaxNorm 2203.14197 §3.3. The paper tunes the stage-2 weight decay and the MaxNorm threshold with Bayesian optimisation 2203.14197 §3.3. On CIFAR100-LT with imbalance factor 100, the ablation gives 38.38% for plain cross-entropy, 46.08% for tuned weight decay alone and 53.35% for the full recipe 2203.14197 tab. 1. On ImageNet-LT and iNaturalist2018, the same recipe reaches 53.9% and 70.2% all-class accuracy, against 38.0% and 46.1% for the naive baseline 2203.14197 tab. 3. Post-hoc L2 normalisation helped on CIFAR100-LT, from 46.08% to 49.60% with weight decay [arxiv:2203.14197 tab. 1]. On iNaturalist it dropped to 51.3% from 65.4% with weight decay 2203.14197 tab. 3. Card: Long-Tailed Recognition via Weight Balancing.

Distribution alignment of a frozen classifier

Stage 1 trains the backbone and the original classifier head on the imbalanced data. Stage 2 freezes both and trains only a few new parameters 2103.16370 §3.2 app. A.2. Each class gets a learned scale alpha and offset beta on its original score. A per-example confidence, a learned layer followed by a non-linearity (a sigmoid in the paper's example), blends the transformed score with the original one 2103.16370 §3.2.1 eq. 4 eq. 5. Stage 2 minimises a cross-entropy whose class weights are inverse frequency raised to a power rho. Rho = 0 gives instance balance and rho = 1 gives class balance 2103.16370 §3.2.2 eq. 11. On ImageNet-LT with ResNet-50, the paper reports 51.3% top-1 average with a dot-product head and 52.9% with a cosine head, against 41.6% for the joint baseline [arxiv:2103.16370 tab. 1]. Its ablation gives 50.1 with the re-weighting and a learned scale, 49.9 with the re-weighting and a learned offset, and 51.3 with both [arxiv:2103.16370 tab. 5]. The same formula is applied per pixel for ADE20K and per box for LVIS 2103.16370 §3.1 app. C.2. Card: Distribution Alignment.

Label-distribution sampling in 3D segmentation

On RELLIS-3D, the KPConv point-cloud model uses label-distribution sampling. The authors report that it adds 0.6 mIoU 2011.12954 §V-C. The paper's description is brief, and the card does not verify the implementation. Card: RELLIS-3D Dataset. The same card's per-class table lists 0.00 IoU for KPConv on mud, puddle and rubble 2011.12954 tab. 2.

Rare predicates in scene graph generation

The survey groups the approaches to long-tailed predicates into zero-, one- and few-shot learning, causal debiasing, and an energy-based loss over joint object and relation likelihood 2104.01111 §4. It also names language, statistical and knowledge-graph priors, which it suggests help rare relations most 2104.01111 §3. Mean recall per predicate, mR@100 under the PredCls setting, is 8.0 for IMP, 22.0 for ResCAGCN with PUM and 29.97 for VCTree-TDE with the energy-based loss 2104.01111 tab. 2. These values come from different papers and were not re-run by the survey, so they are not a controlled ranking. The survey says the energy-based loss gives the best mR@K among the methods it investigated 2104.01111 §6.2. Card: A Comprehensive Survey of Scene Graphs.

Gated frequency prior and node-priority loss in one-stage scene graphs

GPS-Net keeps a one-stage Faster R-CNN based generator and adds two long-tail parts. The first is a predicate prior: the frequency of each subject-object class pair, softened by a log-softmax 2003.12962 eq. 9. A sigmoid gate computed from each pair's union feature rescales the prior before it is added to the classifier logits 2003.12962 §3.3 2003.12962 eq. 10. The second is a node-priority loss. Each object's priority is its share of ground-truth triplets. That share sets a focusing parameter, capped at 2, in a focal-style term 2003.12962 §3.2 2003.12962 eq. 6-8. The paper's best exponent is mu = 4 2003.12962 §4.4. On Visual Genome, the card's mR@100 table gives 22.8 for GPS-Net under PredCls, against 19.4 for VCTree-HL and 15.3 for MOTIFS. All three rows use the Neural Motifs detector and metric 2003.12962 tab. 2. The ablation adds recall gains for each part on its own: the adaptive reasoning module alone reaches 68.5 PredCls R@100 against 67.3 for the baseline, and the node-priority loss alone 67.7 2003.12962 tab. 5. Recall is not mean recall, so these two results are not directly comparable. Card: GPS-Net: Graph Property Sensing Network for Scene Graph Generation.

Rare-concept coverage in web pre-training data

This variant changes the data, not the training loss. CC12M keeps about four times as many image-caption pairs as CC3M by relaxing the CC3M filters. The authors argue that the strict filters drop long-tail concepts [arxiv:2102.08981 §1] 2102.08981 §2.2. Rare words occur more often: "mangosteen" goes from 0 to 212 occurrences and "quiche" from 50 to 1109, counted in CC3M and then CC12M 2102.08981 §2.3. The captions are less precise, at 76.6% against 90.3% for CC3M on a 100-example check by two annotators 2102.08981 §2.3. A captioner pre-trained on CC12M and fine-tuned on COCO reaches 87.4 overall CIDEr on nocaps val, against 54.7 with no pre-training 2102.08981 tab. 3. Its out-of-domain CIDEr is 91.3, against 65.3 for CC3M with fine-tuning 2102.08981 tab. 3. Without fine-tuning, CC12M is weaker than CC3M overall on nocaps, 27.1 against 29.7, but it leads on out-of-domain objects, 41.6 against 37.3 2102.08981 tab. 3. Card: Conceptual 12M.

When to use which

The gain from Copy-Paste on AP_r is smaller with the two-stage recipe (+2.3) than with the single-stage one (+3.1) on Eff-B7 FPN (640) [arxiv:2012.07177 tab. 8] 2012.07177 tab. 9.

Implementation sketch

Five recipes, one block each, since they share no common core. Shapes and units are noted at the end of each line.

The two-stage LVIS recipe from the paper, with Copy-Paste as an optional augmentation in stage 1. RFS is not part of this recipe.

# Stage 1: train the single-stage model, Copy-Paste optional,
# 180k steps at batch 256 [arxiv:2012.07177 §4.7]
model = train_maskrcnn(lvis_train, augment=copy_paste)

# Stage 2: freeze everything except the final classifier layer
# [arxiv:2012.07177 §4.7]
freeze_all_except(model.classifier)

# Class-balanced weight per class from its instance count n_c,
# beta = 0.999; divide by mean, clip to [0.01, 5]
# [arxiv:2012.07177 §4.7, footnote]
beta = 0.999
w = (1 - beta) / (1 - beta ** n)  # n: [num_classes]
w = clip(w / w.mean(), 0.01, 5)

# Classification loss only, weighted by w, 3x schedule
# [arxiv:2012.07177 §4.7]
train(model, loss=weighted_cls_loss(w), schedule="3x")

The EQL v2 loss for one batch of RoIs. The per-class accumulators run over all iterations and are never reset 2012.08548 eq. 9.

# EQL v2 loss for one batch with a sigmoid head [arxiv:2012.08548 §3.2]
# State per class j: pos_sum[j], neg_sum[j], accumulated over iterations
def eql_v2_loss(logits, labels):
    # One-hot over C classes plus background; background is not reweighted
    # [arxiv:2012.08548 §4.2]
    y = one_hot(labels, C + 1)                     # (n, C + 1)
    # Ratio of accumulated positive to negative magnitude
    # [arxiv:2012.08548 eq. 9]
    g = pos_sum / (neg_sum + 1e-10)                # (C,)
    # Negative weight r = sigmoid(gamma * (g - mu)) [arxiv:2012.08548 eq. 6]
    r = concat(sigmoid(gamma * (g - mu)), ones(1)) # (C + 1,)
    # Positive weight q = 1 + alpha * (1 - r) [arxiv:2012.08548 eq. 5]
    q = 1 + alpha * (1 - r)                        # (C + 1,)
    w = q * y + r * (1 - y)                        # (n, C + 1)
    loss = sum(bce_with_logits(logits, y) * w) / n
    # Add reweighted gradient magnitudes to the accumulators
    # [arxiv:2012.08548 eq. 7] [arxiv:2012.08548 eq. 8]
    grad = abs(sigmoid(logits) - y) * w
    pos_sum += sum(grad * y, axis=0)[:C]
    neg_sum += sum(grad * (1 - y), axis=0)[:C]
    return loss

Stage 2 of weight balancing, as the paper describes it. The paper applies MaxNorm after each batch update; the release differs (see Contradictions).

# Stage 2: retrain the classifier on frozen features  [arxiv:2203.14197 §3.3]
# Input: stage-1 backbone, class counts n_c, weight decay lam, radius delta
def retrain_classifier(backbone, fc, train_set, n_c, lam, delta):
    # Class-balanced weights; beta is not stated in the paper
    # [arxiv:2203.14197 §3.3]
    w = class_balanced_weights(n_c, beta)          # (K,)
    # Weight decay on the classifier [arxiv:2203.14197 §3.1]
    opt = sgd(fc.parameters(), weight_decay=lam)
    for images, labels in train_set:
        with no_grad():
            feats = backbone(images)               # frozen features
        loss = class_balanced_loss(fc(feats), labels, w)
        opt.zero_grad(); loss.backward(); opt.step()
        # MaxNorm: rescale any class row whose norm exceeds delta
        # [arxiv:2203.14197 eq. 4] [arxiv:2203.14197 eq. 5]
        fc.weight.data = project_rows(fc.weight.data, delta)
    return fc

Distribution alignment, stage 2, as the paper's equations describe it. The initial values of alpha and beta are from the appendix.

# Stage 2: calibrate frozen scores  [arxiv:2103.16370 §3.2.1] [eq. 4-5]
# Input:  features x (B, d), scores z (B, K) from the frozen head
# Learned: alpha (K,) = 1, beta (K,) = 0 [app. A.2], confidence layer g
def calibrated_scores(x, z, alpha, beta, g):
    # Per-class scale and offset on the original score  [eq. 4]
    s = alpha * z + beta                              # (B, K)
    # Per-example confidence in (0, 1) [§3.2.1]
    sigma = sigmoid(g(x))                             # (B, 1)
    # Blend the transformed and the original score  [eq. 5]
    return sigma * s + (1 - sigma) * z                # (B, K)

# Class weights: inverse frequency to the power rho, summing to one
# [arxiv:2103.16370 eq. 11]
w = (1.0 / n) ** rho                                  # n: (K,) class counts
w = w / w.sum()                                       # (K,)

# Loss: weighted negative log-likelihood, batch mean  [eq. 9]
def alignment_loss(z_hat, y):
    logp = log_softmax(z_hat)                         # (B, K)
    return mean(-w[y] * logp[arange(len(y)), y])      # scalar

Node-priority focal term of GPS-Net, as the paper's equations describe it. The paper does not say whether the focusing weight gets a gradient; the released detector detaches it refine_obj_feats.py.

# Node-priority focal term  [arxiv:2003.12962 §3.2] [eq. 6-8]
# Input: logits (N, O), gt_classes (N,), t (N,) triplets per node
def nps_loss(logits, gt_classes, t, total, mu=4):
    # Priority: share of ground-truth triplets with this node  [eq. 6]
    theta = t / total                                     # (N,)
    # Focusing parameter, capped at 2  [eq. 7]
    gamma = minimum(2, -((1 - theta) ** mu) * log(theta)) # (N,)
    p = softmax(logits)[arange(N), gt_classes]            # (N,)
    # Focal-style term  [eq. 8]; the reduction is not stated
    return mean(-((1 - p) ** gamma) * log(p))             # scalar

Contradictions

• Repeat Factor Sampling. It raises single-stage AP_r by +5.7 (9.7 to 15.4) 2012.07177 tab. 8 and lowers two-stage AP_r by 3.2 (25.0 to 21.8) 2012.07177 tab. 9. The paper states that RFS helps single-stage and hurts two-stage training 2012.07177 §4.7. The card gives no reason for the reversal.
• Headline gains differ by setting, not by result. The "+6.1 mask AP_r" headline is a two-stage model 2012.07177 tab. 7. Single-stage RFS with Copy-Paste gives +8.7 AP_r over the baseline 2012.07177 tab. 8. Compare numbers only within the same training recipe.
• EQL v2 headline gain. The paper's abstract says "about 4 points" over EQL overall. Its Table 1 and §4.4 give +5.1 AP 2012.08548 tab. 1 2012.08548 §4.4. The card notes the abstract's rare-class figure is measured against softmax, not EQL.
• Weight balancing, paper against release. The paper reports 46.08% for weight decay alone and 53.35% for the full recipe 2203.14197 tab. 1. The released logs record 47.18% for weight decay alone [code: exp/demo_1/with_WD_model_note_bestModel.log]. For the full recipe they record 51.98% best-epoch accuracy MaxNorm_WD_note_bestModel.log and 51.83% at the final epoch MaxNorm_WD_train.log. The release selects the best epoch on the test loader trainval.py, while the paper tunes on a validation set 2203.14197 §4.1.
• Weight balancing, paper against paper. Table 2 gives 38.32% for cross-entropy on CIFAR100-LT with imbalance factor 100 [arxiv:2203.14197 tab. 2], and Table 1 gives 38.38% for the same setting [arxiv:2203.14197 tab. 1]. Its "+" labels mean different things in different places: "+ L2norm" and "+ τ-norm" in Table 2 match the post-hoc numbers in Table 1, which use no class-balanced loss 2203.14197 tab. 2 2203.14197 tab. 1.
• Weight balancing, MaxNorm details. The paper applies MaxNorm after each batch 2203.14197 §3.1. The release applies it once per epoch trainval.py and sets the radius from the initial row norms regularizers.py. The paper does not state δ.
• DisAlign, paper against its release. The paper states rho 0.3 for segmentation 2103.16370 app. B.2; the ADE20K config uses 0.2 disalign_fcn_r50-d8_512x512_160k_ade20k.py. Stage 2 is 8k iterations in the paper app. B.2 and 4000 in the config disalign_fcn_r50-d8_512x512_160k_ade20k.py. For LVIS, the paper gives 9k iterations at LR 0.02 app. C.2; the config runs 4500 with the LR step at 4000 config.py.
• DisAlign, loss scale and weights. The paper's weights sum to one and its loss is a batch mean 2103.16370 eq. 9 eq. 11. The LVIS code rescales the foreground weights to sum to the number of foreground classes fast_rcnn.py and takes a weighted mean fast_rcnn.py. The segmentation weights are rescaled the same way grw_cross_entropy_loss.py. The step size changes with the reduction, so copy the code's choice or retune the learning rate.
• DisAlign, ImageNet-LT baselines. The paper gives 52.6 for the ResNeXt-50 DisAlign row and 44.4 for its baseline 2103.16370 tab. 1. The repository README lists 52.438 and 46.918 for the same rows README.md. The repository does not say how its baseline was trained.
• DisAlign, ResNet-152 ablation. The text gives 54.9 for ResNet-152 2103.16370 §4.1. The appendix table gives 54.8 for the cosine variant and 53.7 for the dot-product variant 2103.16370 app. A.
• GPS-Net, paper against code. The paper's best exponent is mu = 4 2003.12962 §4.4; both code paths use 5 nps_loss.py model_builder_rel_gpsnet.py. The paper priority is the triplet share 2003.12962 eq. 6. The lib/ loss counts distinct related neighbours and uses 2θ [code: lib/nps_loss.py#L9-L23], and the detector counts proposal overlaps model_builder_rel_gpsnet.py. The gated prior of eq. 10 is not in the detector [code: gpsnet-detectron/lib/modeling_rel/model_builder_rel_gpsnet.py#L563-L570]. The ablation numbers in the card therefore do not test the paper's definitions.
• Scene graph mean recall, GPS-Net against the survey. The survey lists 29.97 mR@100 under PredCls for VCTree-TDE 2104.01111 tab. 2. GPS-Net's table lists 19.4 for VCTree-HL and 22.8 for GPS-Net under the Neural Motifs protocol 2003.12962 tab. 2. These come from different papers and protocols, so they are not a ranking.
• Cross-card comparisons. EQL v2 reports LVIS v1.0 with Mask R-CNN R-50 at 1x 2012.08548 tab. 1 and LVIS v0.5 with several detectors 2012.08548 tab. 4. The Copy-Paste card reports LVIS v1.0 with Eff-B7 FPN at 640 2012.07177 tab. 8. Their AP_r values are not a ranking.

Papers

• Simple Copy-Paste: Copy-Paste augmentation, the LVIS two-stage class-balanced recipe, Repeat Factor Sampling and the rare-class results.
• Equalization Loss v2: a per-class gradient-ratio reweighting of a sigmoid detector's classification loss, trained end-to-end on LVIS and Open Images.
• Long-Tailed Recognition via Weight Balancing: a two-stage classifier recipe with weight decay, MaxNorm and class-balanced loss, on CIFAR100-LT, ImageNet-LT and iNaturalist2018.
• RELLIS-3D Dataset: an off-road segmentation benchmark whose authors attribute weak rare-class results to imbalance, with a reported label-distribution sampling gain for the point-cloud model.
• A Comprehensive Survey of Scene Graphs: a survey of rare-predicate methods for scene graphs, and of mean recall as the metric.
• Conceptual 12M: relaxed-filter web image-caption data with more rare words, and its nocaps results after fine-tuning.
• Distribution Alignment: a two-stage recalibration of a frozen classifier with per-class scale and offset and a per-example gate, with results on classification, segmentation and LVIS.
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation: a one-stage scene graph generator with a gated predicate frequency prior and a node-priority focal loss, with mean recall per predicate on Visual Genome.