Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection

In plain words

Rare object types are hard to learn in a dataset where a few types dominate. A detector that is trained on such data learns to say "not this" to almost every candidate box, because most candidates are not the rare type. The rare type's "yes" signal is drowned out by the flood of "no" signals.

The authors measure, for each object type separately, how much the "yes" signal and the "no" signal add up to during training. For common types the two are roughly balanced. For rare types the "no" side is far larger. Their method, Equalization Loss v2, watches this running balance for each type and adjusts the weights: "yes" signals are amplified and "no" signals are reduced until the balance evens out. Every type gets the same rule, so no list of rare and common types has to be fixed in advance.

The method needs no separate second training stage. The authors report that it beats the earlier end-to-end method and also beats two-stage approaches on the LVIS benchmark.

TL;DR

EQL v2 is a drop-in replacement for the classification loss of a sigmoid detector. It keeps, per class, a running ratio of accumulated positive to negative gradient magnitude and maps that ratio through a sigmoid to get a weight for negatives. Positives get a weight that grows as negatives are suppressed. The authors report that it trains end-to-end and beats decoupled methods on LVIS v1.0 and on Open Images, with the reported numbers coming from their own runs. The official MMDetection-based code reproduces the method closely, with a few implementation details the paper leaves out.

Problem

In long-tailed detection the rare categories have few annotated instances, so the classifier is biased toward frequent categories §1. The authors argue the cause is not only data scarcity: with one sigmoid output per category, every instance is a positive for one category and a negative for the other C−1 categories, so a rare category receives many more negative gradients than positive ones §3.1 eq. 1 eq. 2.

Earlier end-to-end fixes reweight or resample without tracking this balance: repeat-factor sampling costs extra training time, and the equalization loss (EQL) suppresses some negatives but the gradient ratio of tail categories rises only slightly §1 fig. 1. Decoupled methods (cRT, LWS, BAGS) add a second fine-tuning stage §1 tab. 1.

The method assumes a sigmoid classifier per category and a Mask R-CNN style detector. The authors state the reweighting is not applied to the category-agnostic objectness task §4.2.

Method

EQL v2 has three parts: a per-category view of the detection problem, a gradient-ratio statistic, and a reweighting rule driven by that statistic.

1. Per-category sub-tasks. Each of the C category logits is treated as an independent binary task. For a classifier j, one instance is a positive for j only if its label is j, and a negative otherwise §3.1. The expected positive-to-negative count ratio is 1 / (N / n_j − 1), where n_j is the instance count of category j and N the total eq. 2. The authors argue this count ratio is a poor indicator of balance, because each sample's gradient differs in size §3.2.
2. Gradient magnitudes. For category j and sample i with predicted probability p, the positive gradient is y (p − 1) and the negative gradient is (1 − y) p, each averaged over the batch §3.2 eq. 3 eq. 4. Their absolute values are the magnitudes used below.
3. Accumulated ratio. g_j is the ratio of accumulated, already reweighted, positive magnitudes to accumulated, already reweighted, negative magnitudes for category j, summed over all iterations up to the current one §3.2 eq. 9.
4. Weights. At iteration t the positive weight is q = 1 + α (1 − f(g)) and the negative weight is r = f(g), with f a sigmoid centred at μ with steepness γ §3.2 eq. 5 eq. 6. The paper sets α = 4, γ = 12 and μ = 0.8 §4.2.
5. Apply and accumulate. The weights multiply the positive and negative gradients of the current batch §3.2 eq. 7 eq. 8. The accumulators then include the reweighted magnitudes, and g is updated for the next iteration eq. 9.
6. Objectness task. An extra, category-agnostic output is trained so that every other category's positive sample is also its positive sample. It is not reweighted §4.2. At inference the score of each category is multiplied by the objectness probability p_obj: p'_j = p_j · p_obj §4.2 eq. 10.
7. Training. Models are trained end-to-end with a random sampler, without repeat-factor sampling tab. 1.

Pseudocode for the loss of one batch of RoIs. The paper gives no algorithm box. The loss form and the accumulator update are reconstructed from the equations above and checked against the official code, which is noted in ## Paper vs code.

# EQL v2 loss for one batch of RoIs  [§3.2] [eq. 5] [eq. 6]
# Input:  logits (n, C + 1); labels (n,), background label is C
# State:  pos_grad, neg_grad (C,) accumulated over all iterations
# Output: scalar loss

def eqlv2_loss(logits, labels):
    # One-hot targets; the last column is the background channel  [§4.2]
    targets = one_hot(labels, num_classes + 1)      # (n, C + 1)
    # Negative weight r = f(g) with f a sigmoid of the ratio  [eq. 6]
    neg_weight = sigmoid(gamma * (pos_neg - mu))    # (C,)
    # The background channel is not reweighted  [§4.2]
    neg_weight = concat(neg_weight, ones(1))        # (C + 1,)
    # Positive weight q = 1 + alpha * (1 - r)  [eq. 5]
    pos_weight = 1 + alpha * (1 - neg_weight)       # (C + 1,)
    # Weight per entry: q for positives, r for negatives  [eq. 5]
    weights = pos_weight * targets + neg_weight * (1 - targets)
    # Weighted sigmoid cross entropy, divided by n  [eq. 5]
    loss = sum(bce_with_logits(logits, targets) * weights) / n
    # Gradient magnitude |p - y| per entry, reweighted  [eq. 3] [eq. 4] [eq. 7]
    grad = abs(sigmoid(logits) - targets) * weights
    # Accumulate reweighted positive and negative magnitudes  [eq. 9]
    pos_grad += sum(grad * targets, axis=0)[:C]
    neg_grad += sum(grad * (1 - targets), axis=0)[:C]
    # Ratio for the next iteration; 1e-10 guards division  [eq. 9]
    pos_neg = pos_grad / (neg_grad + 1e-10)
    return loss

Key equations, as numbered in the paper:

q_j^{(t)} = 1 + \alpha\,(1 - f(g_j^{(t)})), \qquad r_j^{(t)} = f(g_j^{(t)}) eq. 5

f(x) = \frac{1}{1 + e^{-\gamma(x - \mu)}} eq. 6

g_j^{(t+1)} = \frac{\sum_{t^{*}=0}^{t} \lvert \nabla^{pos'}_{z_j}(\mathcal{L}^{(t^{*})}) \rvert}{\sum_{t^{*}=0}^{t} \lvert \nabla^{neg'}_{z_j}(\mathcal{L}^{(t^{*})}) \rvert} eq. 9

p'_{j} = p_{j} \cdot p_{obj} eq. 10

Hyperparameters

Setting: LVIS v1.0 (1203 categories, about 100k training images with 1.3M instances, 20k validation images) §4.1. Framework: MMDetection §4.2 eqlv2.py.

Results

Author-reported. Setup: LVIS v1.0 val, ResNet-50-FPN Mask R-CNN, 1x schedule, unless stated §4.4. AP metrics are as the paper defines them: AP, AP_r (rare, 1–10 images), AP_c (common), AP_f (frequent), AP_b (box AP) §4.1.

Table 1 tab. 1: end-to-end and decoupled methods, LVIS v1.0 val

Table 2 tab. 2: component ablation, LVIS v1.0 val, random sampler, 1x

Table 3 tab. 3: larger backbones, 3x schedule, LVIS v1.0 val, random sampler

Table 4 tab. 4: LVIS v0.5 val. Rows marked † are copied from the cited papers

Table 5 tab. 5: LVIS v1.0 val, 2x schedule per the caption. Rows marked * are the authors' re-implementation; † rows are copied

Table 6 tab. 6: decoupled re-training on top of EQL v2 pretraining, LVIS v1.0 val

Table 7 tab. 7: varying γ and μ, LVIS v1.0 val, α = 4

Table 8 tab. 8: varying α, γ = 12, μ = 0.8

Table 9 tab. 9: C-sigmoid vs 2C-softmax, LVIS v1.0 val

Table 10 tab. 10: Open Images Challenge 2019 val (mAP@0.5), five groups of 100 categories by instance count, AP1 rarest

Appendix A app. A: other mapping functions (AP, AP_r, AP_c, AP_f), neg only and neg + pos

Appendix B app. B: further γ and μ values (AP, AP_r, AP_c, AP_f)

Headline numbers and comparisons the authors make:

• Against softmax on LVIS v1.0 with the same 1x schedule: +7.6 AP (23.7 vs 16.1) §4.4.
• Against EQL on the same setting: +5.1 AP (23.7 vs 18.6) §4.4 tab. 1.
• Against the best decoupled method (BAGS, 12+12 epochs): +0.6 AP (23.7 vs 23.1) tab. 1.
• LVIS v0.5, Mask R-CNN R-50: +4.3 AP over EQL (27.1 vs 22.8) §4.5 tab. 4.
• Open Images: +9.5 AP over Faster R-CNN and +7.3 AP over EQL, R-50 §4.9 tab. 10.
• The paper reports a rare-category gain of 17.1 AP_r with a 3x schedule (2.0 to 19.1, EQL v2 on Mask R-50) tab. 3.

Paper vs code

Official code: https://github.com/tztztztztz/eqlv2, commit 907fad905b0cac6372f7ec4035f48ef3d0801098, an MMDetection-based tree under Apache-2.0. The method is EQLv2 in mmdet/models/losses/eqlv2.py and is selected in configs/end2end/eqlv2_r50_8x2_1x.py eqlv2_r50_8x2_1x.py. I checked the loss, the weights, the accumulators, the mapping function and the objectness handling against the paper; they agree on the method. The differences are:

• Classifier bias. The paper says the bias of the last classification layer is initialised with the value 0.001 §4.2. The code calls bias_init_with_prob(0.001), which is a prior probability: the bias is -log((1 - 0.001) / 0.001), about −6.9 convfc_bbox_head.py. The weights are initialised with normal noise of std 0.01 in the same place. The paper does not state the weight init.
• Batch and GPUs. The paper says 16 GPUs with 1 image each §4.2. The config sets samples_per_gpu=2 mask_rcnn_r50_sample1e-3_8x2_1x.py. The name 8x2 suggests 8 GPUs with 2 images each. Both give a total batch of 16. The paper's wording does not match the config; the config is the only record of the setup in the repository.
• Sampler. The paper uses a random sampler tab. 1. The end-to-end config sets oversample_thr=0.0 mask_rcnn_r50_8x2_1x.py. According to the dataset wrapper's documentation, a threshold of 0 repeats no category dataset_wrappers.py, so this is the random sampler.
• Objectness channel. The paper describes an extra task that treats all other categories' positives as its positives §4.2. The code gives the objectness output as a background column: the target is 1 for background RoIs, and the label of background RoIs is num_classes eqlv2.py. At inference the scores are multiplied by 1 - sigmoid(background) eqlv2.py. This matches p_j · p_obj if p_obj = 1 - p_background. The code's switch test_with_obj defaults to True.
• Schedule details. The paper says "1x" and gives no step or warm-up values. The code uses 12 epochs, learning-rate steps after epochs 8 and 11, and a 500-iteration linear warm-up from 0.001× [code: configs/base/schedules/schedule_1x.py#L2-L11].
• Gradient statistics across GPUs. The code sums the per-class accumulators over all GPUs with all_reduce eqlv2.py, so the ratio is global across the batch. The paper does not say whether the statistics are local or global.
• Initial ratio. The code sets the initial ratio to 100 eqlv2.py. With that value the weights start at r ≈ 1 and q = 1. The paper says only "a high value" in the code comment; it does not state the value.
• Accumulators are checkpointed. The accumulators are registered buffers, so they are saved in and restored from checkpoints eqlv2.py.
• Loss normalisation. The code divides the weighted sum by the number of RoIs in the batch n_i eqlv2.py eqlv2.py. The paper does not write the loss in full, so this normalisation is not stated there.
• Objectness not reweighted. The code sets the background weight to 1 eqlv2.py, which matches §4.2.
• Defaults. The code's defaults are gamma=12, mu=0.8, alpha=4.0 eqlv2.py, as in the paper.
• Evaluation. The LVIS config sets a score threshold of 0.0001 and up to 300 detections per image mask_rcnn_r50_fpn_sample1e-3_mstrain_1x_lvis_v1.py, as in §4.2.

Pitfalls

• Scale of the positive weight. q = 1 + α(1 − r) runs from 1 (when r = 1) up to 1 + α (when r = 0) eq. 5. With α = 4 the largest positive weight is 5, and with α = 8 (the Open Images or the best-AP setting in tab. 8) it is 9.
• Sigmoid steepness. With γ = 12 and μ = 0.8, r is 0.5 at ratio 0.8 and about 0.03 at ratio 0.5 (computed from eq. 6; not stated in the paper). Small changes in the ratio can move the negative weight a lot, which is why tab. 7 and app. B vary both parameters.
• Initial phase. Until the first iterations have built up a ratio, the code's initial value of 100 keeps the weights at 1. An implementation that starts at 0 or 1 would give different early weights; the paper does not state the start value.
• Never reset. The accumulators run over all of training and are never decayed eq. 9. Earlier iterations keep their influence, and a resumed run inherits the saved accumulators. Changing the number of classes between runs breaks the saved buffers.
• Background label. The objectness output relies on background RoIs carrying label C and on the bias and weight rules above. A change to the label convention silently changes the objectness target.
• Loss scale. The weights change the magnitude of the loss, so the loss values are not comparable with a plain sigmoid loss. The paper compares AP, not loss.
• Numbers the abstract and text state differently. The abstract says EQL v2 is "about 4 points" above EQL overall; Table 1 and §4.4 give 5.1. The abstract's "14 to 18 points" on rare categories is measured against the softmax baseline (14.9 in Table 1, 17.1 in Table 3), not against EQL.
• EQL baseline. The EQL numbers in Table 5 are the authors' re-implementation with a λ they tuned for LVIS v1.0 (1.1e-3 §4.5). The original λ (1.76e-3) was for LVIS v0.5. Comparisons with the EQL row need this in mind.
• Schedule labels. Table 5 says 2x; the code configs for the 8x2 end-to-end setup are 1x. The paper's text calls its main setup "1x" §4.4.
• Open Images tuning. The tuned rows tab. 10 use hyperparameters searched on Open Images; the other rows use LVIS settings unchanged §4.9.
• Decoupled comparisons. Table 1's decoupled rows use two stages (12+12 epochs), so their compute is not the same as the single-stage EQL v2 row (12 epochs) tab. 1.

Lineage

• Builds on: Equalization Loss (EQL, Tan et al., cited as [tan2020equalization] in the paper; not compiled here), whose negative-suppression idea this paper extends to both positives and negatives.
• Compared against: repeat-factor sampling (RFS), class-balanced decoupled methods (cRT, LWS), BAGS, SimCal, Forest R-CNN, LST and De-confound-TDE tab. 1 tab. 4 tab. 5.
• Related modification in the same paper: 2C-softmax as an alternative classifier, reported as comparable tab. 9.
• Superseded by: not known from the content of this wiki.