Class-balanced loss

Overview

When some classes have many training examples and others have few, plain cross-entropy gives the frequent classes more influence over training. The class-balanced loss counters this by multiplying each example's loss by a weight that depends only on its class. The weight is set from how often that class, or pixel label, occurs in training. The variants differ in the formula. The first card uses the effective number of samples, which saturates as the count grows 2203.14197 §3.3. The second uses inverse frequency raised to a power rho 2103.16370 eq. 11. The effective-number formula below is specific to the first variant.

The formula the first card gives for the weight is

w_c = \frac{1-\beta}{1-\beta^{n_c}},

where n_c is the number of training samples in class c and \beta is a hyperparameter 2203.14197 §3.3. The weights are then rescaled so they sum to the number of classes 2203.14197 §3.3 [code: utils/class_balanced_loss.py#L73-L75]. The paper's text gives neither a value for \beta nor the loss form 2203.14197 §3.3. The released code uses \beta = 0.9999 demo2_second-stage-training.py.

Three wiki sources use the technique. In "Long-Tailed Recognition via Weight Balancing", the class-balanced loss is one component of a two-stage recipe, and it is used only to retrain the final classifier on frozen features 2203.14197 §3.3. In "Distribution Alignment", the weight is inverse frequency raised to a power, and it trains a per-class calibration on frozen features 2103.16370 §3.2.2. A third card, EGNet, does not use either weight in its paper. Its paper's edge loss is unweighted, but its released code weights edge and background pixels by class frequency 1908.08297 eq. 4 solver.py.

Variants

Classifier retraining on frozen features

Stage 1 trains the backbone with plain cross-entropy and tuned weight decay. Stage 2 freezes the features and retrains only the classifier, with the class-balanced loss, weight decay and a MaxNorm projection [arxiv:2203.14197 §3.3]. The paper's ablation on CIFAR100-LT with imbalance factor 100 reports 53.35% class-averaged accuracy for the full stage-2 recipe, against 46.08% for weight decay alone 2203.14197 tab. 1. Card: Long-Tailed Recognition via Weight Balancing.

Within that setting the loss matters most when combined with stage-2 weight decay. On tuned features, the class-balanced loss alone gives 47.09%, against 46.08% for cross-entropy. With stage-2 weight decay, it gives 52.42%, against 47.22% for cross-entropy with the same decay 2203.14197 tab. 1. The paper does not separate these effects further.

Training from scratch with the loss

The same loss can also replace cross-entropy during end-to-end training. On CIFAR-100-LT with imbalance factor 100, the paper reports 39.60% for cross-entropy plus class-balanced loss, against 38.32% for cross-entropy 2203.14197 tab. 2. On ImageNet-LT it reports 33.2%, against 44.4% for cross-entropy, and on iNaturalist2018 54.0%, against 61.7% [arxiv:2203.14197 tab. 3]. The card does not explain the gap between these datasets. Card: Long-Tailed Recognition via Weight Balancing.

Generalised re-weighting with a learnable calibration

Distribution Alignment weights each class by (1/r_c)^{\rho}, normalised over classes, where r_c is the class's training-set frequency and \rho is a scale hyper-parameter 2103.16370 eq. 11. The paper calls \rho = 0 instance-balanced and \rho = 1 class-balanced 2103.16370 §3.2.2. The weight of each example's true class enters a cross-entropy over calibrated scores 2103.16370 eq. 9 eq. 10. Stage 2 freezes the backbone and the original classifier. It learns a per-class scale and offset, and an input-dependent gate that blends them with the original scores [arxiv:2103.16370 §3.2.1] eq. 5.

The paper's values for \rho are 1.2 for a dot-product classifier and 1.5 for a cosine one on classification, 0.3 for segmentation and 0.8 for LVIS 2103.16370 app. A.2 app. B.2 app. C.2. On ImageNet-LT with ResNet-50 and a cosine head, the full method reports 52.9% top-1, against 48.4% for the same cosine baseline 2103.16370 tab. 1. That number is for the whole calibration, not for the weight alone. The paper's component ablation reports 41.6% with none of the three components, and 50.1%, 49.9% and 51.3% when the re-weighting is combined with a learnable magnitude, a learnable margin, or both 2103.16370 tab. 5. The table has no row with the re-weighting alone, so its own effect is not isolated 2103.16370 §4.1. Card: Distribution Alignment: A Unified Framework for Long-tail Visual Recognition.

Class-weighted edge loss in a salient-object detector

EGNet trains its edge branch with a binary cross-entropy whose equation is unweighted 1908.08297 eq. 4. Its released code instead weights edge pixels by the fraction of negative pixels, and background pixels by 1.1 times the fraction of positive pixels, in the EGNet repository [code: solver.py#L160] solver.py. This is a per-pixel frequency weight, not the effective-number weight. The paper does not say whether its reported results used the weighted or the plain loss 1908.08297 §4.3. Card: EGNet: Edge Guidance Network for Salient Object Detection.

When to use which

Implementation sketch

The core is the per-class weight and the weighted loss. Stage-2 projection and weight decay come from the same recipe and are not shown here. Values are the released defaults for CIFAR100-LT, not values stated in the text.

# Per-class weight from effective number of samples  [§3.3]
# Input: class_counts n_c (K,), beta
# Output: weights (K,), summing to K  [code: cb#L73-L75]

def class_balanced_weights(class_counts, beta):
    # Effective number of samples per class, saturating in n_c  [§3.3]
    effective = 1 - beta ** class_counts            # (K,)
    weights = (1 - beta) / effective                # (K,)
    # Rescale so the weights sum to the number of classes
    return weights / weights.sum() * len(class_counts)


# Weighted loss on softmax probabilities  [code: cb#L92-L94]
# Each entry carries the weight of its sample's true class
def class_balanced_loss(logits, labels, class_weights):
    probs = softmax(logits)                         # (B, K)
    targets = one_hot(labels, num_classes)          # (B, K)
    sample_weights = class_weights[labels][:, None] # (B, 1)
    bce = -(targets * log(probs) + (1 - targets) * log(1 - probs))
    # Mean over all B*K entries, as the released code does
    return mean(bce * sample_weights)

The power-law variant changes only the weight. It uses (1/n_c)^{\rho}, normalised to sum to one, and a batch-mean loss over the weighted log-likelihood 2103.16370 eq. 9 eq. 11. The two normalisations differ, as the Contradictions section explains.

Contradictions

• Labels in the paper's tables. The paper's "+" notation is inconsistent. The row "WD + CB" in the main comparison table reports 52.42%, the same number as "WD tuned + CB & WD" in the ablation 2203.14197 tab. 2 [tab. 1]. Read each row's exact definition in the card before comparing.
• Reproduced headline number. The paper reports 53.35% for the full recipe. The release logs show 51.98% best-epoch accuracy at stage 2 MaxNorm_WD_note_bestModel.log. The training loop picks the best epoch on the test loader trainval.py, although the paper says it tunes on a validation set 2203.14197 §4.1. Budget for lower accuracy on a held-out split.
• Loss form. The first card's method section gives the weight formula with a section citation, while its comparison section says the paper does not give the loss form. The softmax form above comes from the code only.
• Same name, different weight. Both papers use "class-balanced" for their \rho = 1 or effective-number weights, but the formulas differ. The first paper uses (1-\beta)/(1-\beta^{n_c}) 2203.14197 §3.3. The second uses (1/r_c)^{\rho} and calls \rho = 1 class-balanced [arxiv:2103.16370 §3.2.2]. Results from one do not transfer to the other without a check.
• Weight normalisation. The first paper rescales its weights to sum to the number of classes 2203.14197 §3.3. The second paper's eq. 11 normalises them to one 2103.16370 eq. 11. Its segmentation and LVIS code rescale to the number of classes instead [code: semantic_seg/disalign/models/losses/grw_cross_entropy_loss.py#L34-L35] [code: instance_seg/lvis0.5/mask_rcnn/res50/mask_rcnn.res50.fpn.lvis.multiscale.cos_norm.disalign.1x/roi_heads/fast_rcnn.py#L51-L58]. Neither side is checked against reported numbers.
• Loss reduction. The first paper's code averages over all batch-by-class entries class_balanced_loss.py. The second paper's eq. 9 is a batch mean, but its LVIS code uses a weighted mean that divides by the sum of the batch weights fast_rcnn.py. Each changes the effective step size, so copy one reduction and keep it.
• Segmentation \rho. The second paper states \rho = 0.3 for all segmentation experiments 2103.16370 app. B.2. Its config sets 0.2 disalign_fcn_r50-d8_512x512_160k_ade20k.py.
• Edge loss, paper vs code. The EGNet paper's edge loss is plain cross-entropy 1908.08297 eq. 4. Its release weights pixels by class frequency, and a comment in that code describes the opposite weighting from the one it applies solver.py. Neither side is checked against the reported numbers.

Papers

• Long-Tailed Recognition via Weight Balancing: uses the class-balanced loss as the classifier loss in a two-stage recipe, and compares it against cross-entropy and other long-tailed methods.
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition: uses an inverse-frequency power weight in a stage-2 calibration of frozen classifier scores, for classification, segmentation and detection.
• EGNet: Edge Guidance Network for Salient Object Detection: trains an edge branch with an unweighted cross-entropy in the paper; its released code weights edge and background pixels by class frequency.