PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions

In plain words

Training a classifier needs a loss function, a score that says how wrong the model is. The common choice, cross-entropy, is one fixed formula. Focal loss is a well-known variant that was designed for crowded object detection. The paper notices that both are the same kind of object: a sum of simple building blocks, each a power of how far the model is from being sure of the right answer. Cross-entropy uses all of these blocks with fixed weights, and focal loss only moves them along the power axis.

The authors ask whether the weights should be fixed at all. Their proposal, PolyLoss, lets you change the weight of each block. In practice they find that changing only the weight of the first block is enough to beat both cross-entropy and focal loss on several tasks. This version, called Poly-1, adds one extra number and one line of code. The right value depends on the task and dataset: a positive value helped one ImageNet setup, and a negative value helped object detection on COCO.

TL;DR

• PolyLoss writes a loss as a weighted sum of powers of (1 - Pt), where Pt is the predicted probability of the true class §3.
• Cross-entropy and focal loss are two particular weightings; focal loss shifts all weights along the power axis §3 §4.
• Poly-1 changes only the first weight: cross-entropy plus epsilon * (1 - Pt), or the same change added to focal loss §4.3 §5.3.
• Author-reported gains with one tuned value: ImageNet-21K pretraining of EfficientNetV2-L, COCO Mask R-CNN and Waymo 3D detection §5.
• All baseline training hyperparameters were kept as they were tuned for the baseline loss; the authors note this favours the baseline §4.3 app..

Problem

Cross-entropy and focal loss are the standard classification losses, but neither is shown to be best across tasks. Focal loss helps detection with heavy class imbalance but the paper finds it hurts ImageNet-21K §1. Designing a loss over the full space of functions is hard, and recent learned-loss methods use search spaces that grow with the polynomial order §2. The paper asks whether small, cheap changes to an existing loss give consistent gains §1 §4.

Method

Polynomial view of the losses

Write the loss as a sum over j >= 1 of a coefficient times (1 - Pt)^j §3.

• Cross-entropy is minus log Pt, which equals the sum of (1/j) (1 - Pt)^j eq. 1.
• Focal loss -(1 - Pt)^gamma log Pt equals the sum of (1/j) (1 - Pt)^(j + gamma) eq. 2. Its effect is a horizontal shift of all coefficients by gamma §3 fig. 1.
• The gradient of cross-entropy with respect to Pt is the sum of (1 - Pt)^(j - 1), so its leading term is a constant 1 eq. 3 §3.
• The gradient of focal loss has leading coefficient (1 + gamma), which is no longer a constant, because the first term is shifted eq. 4 §3.

The general form is a sum of alpha_j (1 - Pt)^j with alpha_j >= 0 §3. Convergence needs the coefficients to satisfy a root-test bound, and the paper restricts to non-negative alpha_j and integer j §3, footnotes.

Dropping higher-order terms (explored, not adopted)

Truncating cross-entropy to its first N terms is shown to need several hundred terms before accuracy matches cross-entropy on ImageNet-1K with ResNet-50 §4.1. The residual after N terms is bounded by a function of N and a lower bound delta on Pt §4.1, theorem 1, app. proof. Learning-rate changes do not close the gap §4.1.

Poly-N (explored, not adopted)

Poly-N perturbs the first N coefficients: the j-th coefficient 1/j becomes 1/j + epsilon_j, and the rest are unchanged §4.2 eq. 5 eq. 6. A grid over N=1 gives nearly the same gain as N=2 for much less search §4.2.

Poly-1 (the proposed loss)

Poly-1 changes only the first coefficient.

L_Poly-1 = -log(Pt) + epsilon_1 * (1 - Pt)            [§4.3] [eq. 7]

The constraint epsilon_1 >= -1 keeps the first coefficient non-negative. The authors' code comments state this constraint §6 listing. At epsilon_1 = -1 the first term of cross-entropy is removed.

For focal loss (Section 5.3), the same idea is applied to the focal form:

L_FL-Poly-1 = -(1 - Pt)^2 log(Pt) + epsilon_1 * (1 - Pt)^3

and for RSN the first term is dropped and the new leading term is tuned:

L_FL-Poly-1* = -(1 - Pt)^2 log(Pt) - (1 - Pt)^3 + epsilon_2 * (1 - Pt)^4

§5.3 (the focal-versus-Poly comparison table in §5.3).

Pseudocode for cross-entropy Poly-1, following the paper's listing:

# Poly-1 cross-entropy  [§4.3] [eq. 7] [§6 listing 1]
# Input:  logits (B, C); labels (B, C) one-hot; epsilon >= -1
# Output: per-example loss (B,)

def poly1_cross_entropy(logits, labels, epsilon):
    # Probability of the true class  [§3]
    pt = sum(labels * softmax(logits), axis=-1)     # (B,)
    # Standard cross-entropy, minus log pt  [§3] [eq. 1]
    ce = softmax_cross_entropy(labels, logits)      # (B,)
    # Add the perturbed first term  [§4.3] [eq. 7]
    poly1 = ce + epsilon * (1 - pt)                 # (B,)
    return poly1

Pseudocode for focal Poly-1 (sigmoid form), following the paper's listing:

# Poly-1 focal loss  [§5.3] [§6 listing 3] [eq. 2]
# Input:  logits (B, K); labels (B, K) in {0, 1}; epsilon >= -1
# Output: loss (B, K); reduction over examples is not stated

def poly1_focal_loss(logits, labels, epsilon, gamma=2.0):
    # Probability of the labelled outcome per class  [§6 listing 3]
    p = sigmoid(logits)                             # (B, K)
    pt = labels * p + (1 - labels) * (1 - p)        # (B, K)
    # Baseline focal loss  [§5.3] [eq. 2]
    focal = -((1 - pt) ** gamma) * log(pt)          # (B, K)
    # Add the perturbed first term  [§5.3]
    poly1 = focal + epsilon * (1 - pt) ** (gamma + 1)  # (B, K)
    return poly1

Listing variants in the paper: a label-smoothing version of the cross-entropy Poly-1 (smoothed labels inside the 1 - Pt term), and a focal version with a class-balance weight alpha (default 0.25) §6 listings 2 and 4.

Hyperparameters

Gradient share: for most of ResNet-50 training, the first polynomial term makes up over half of the cross-entropy gradient §4.3 (plot only, no table).

Results

Author-reported. All numbers are validation-set results from the paper, with the baseline and setup stated for each row.

Waymo Open Dataset, validation set. Columns are AP/APH for BEV and 3D, Level 1 and Level 2 §5.3:

Other ResNet-50 comparisons on ImageNet-1K, from the appendix app.:

Source: app. (tables on tuning and training techniques).

Paper vs code

No official code.

What was checked:

• The paper's footnotes point to three upstream baseline codebases, not to a PolyLoss repository: google/automl (EfficientNetV2, ImageNet) §5.1 footnote, tensorflow/tpu models/official (ResNet-50 and Mask R-CNN) §4 footnote §5.2 footnote, and tensorflow/lingvo lingvo/tasks/car (PointPillars) §5.3 footnote. These are the training codebases the authors reused unchanged. They were not cloned, so no code-level comparison was made, and the Poly-1 change itself is only in the paper's listings.
• A GitHub repository search for "polyloss" returned only third-party reimplementations (for example yiyixuxu/polyloss-pytorch and abhuse/polyloss-pytorch). None is owned by an author or by the authors' organisations, so none is used.
• The paper's listings in the conclusion section are the only code for the method §6 listings 1 to 4. They are the reference implementation a reader can check against.

Pitfalls

• The listed functions depend on focal_loss(pt, gamma) and focal_loss(pt, gamma, alpha), which the paper does not define. Implement the focal term from eq. 2 and §5.3. The sigmoid form uses an elementwise pt over all classes §6 listing 3.
• Epsilon_1 >= -1 keeps the first coefficient non-negative. The paper restricts to non-negative coefficients §3, footnote, so values below -1 are outside the studied range §6 listing.
• The best epsilon has opposite signs on different tasks: the ImageNet-21K classification value is positive (2) and the COCO Mask R-CNN value is negative (-1) §5.2. Do not carry one value across tasks; grid-search per dataset as the paper did §5.1 §5.2.
• Training hyperparameters were not re-tuned for Poly-1 in the main tables. The authors state that this disadvantages Poly-1 §4.3 app.. The one tuned example is weight decay for ResNet-50, which improved Poly-1 to 77.1 while cross-entropy stayed at 76.3 app..
• For RSN, epsilon_2 is task-specific (-0.4 for vehicle, +0.2 for pedestrian) §5.3 tab. 5. The paper does not give a rule for choosing it.
• The RetinaNet ablation reports Focal loss and PolyLoss rows without stating the epsilon for PolyLoss. The authors suspect a local maximum at the default (gamma, alpha) app.. Treat the RetinaNet gains as a secondary, unstated-setting result.
• In the label-smoothing listing, one_minus_pt is computed from the smoothed labels, while the CE term uses the framework's built-in label smoothing. The listing is the only specification of this variant §6 listing 2.
• Commented-out source text (not in the published paper) claims that positive epsilon_1 caused earlier overfitting on ImageNet-21K and that the authors raised RandAugment magnitude to 35 and epochs to 450 to compensate §5.1, commented LaTeX. Do not assume this schedule; the published text only says the EfficientNetV2 schedule was used unmodified §5.1.
• Ideas that are only in the appendix are not the main method: collective exponential-decay coefficients (72.3) did not beat Poly-1 app..
• The paper's own explanation for the ImageNet-21K gain (higher mean Pt) and for the COCO gain (lower overconfident Pt) are observations from plots, not a proof §5.1 §5.2.

Lineage

• Builds on: the focal loss of Lin et al. (2017) §1, §5.3; cross-entropy's Taylor-like view of Feng et al. (2020) (Taylor cross-entropy, a truncated form) §2; the TaylorGLO learned polynomial loss of Gonzalez et al. (2020) §2 app..
• Compared against: cross-entropy, focal loss, Drop-N truncation (Feng et al., 2020), label smoothing, dropout on FC, and an exponential-decay coefficient variant §4 app..
• Superseded by: not known from content already in this wiki.