Cross-entropy loss

Overview

For a classifier with one-hot labels, cross-entropy is the negative log of the probability the model assigns to the true class, written Pt [arxiv:2204.12511 §3] 2204.12511 eq. 1. It is the default loss that other classification losses are compared against. The PolyLoss paper rewrites it as a sum of powers of (1 - Pt) with fixed coefficients, and shows that the same form covers focal loss. Every variant on this page is a non-negative weighted sum of powers of (1 - Pt) with different coefficients 2204.12511 §3.

This page currently rests on one paper card. Other cards in the wiki mention cross-entropy in passing, but none lists this concept, so none is used here.

Variants

Plain cross-entropy

Minus log Pt. Its gradient with respect to Pt begins with a constant term, so every power of (1 - Pt) contributes with a fixed weight [arxiv:2204.12511 eq. 3]. This is the baseline in every comparison in the paper. Card: PolyLoss.

Focal loss

Multiplies cross-entropy by (1 - Pt) raised to gamma. The paper shows this shifts every coefficient of the polynomial by gamma rather than changing their values 2204.12511 eq. 2 2204.12511 §3. Gamma is 2 in the paper's PointPillars and RSN experiments 2204.12511 §5.3. See Focal loss.

Poly-1 (add one tuned term)

Adds epsilon_1 * (1 - Pt) to cross-entropy, with epsilon_1 >= -1 [arxiv:2204.12511 §4.3] 2204.12511 eq. 7. The same change applied to focal loss is described in 2204.12511 §5.3. Epsilon_1 is set by grid search on held-out data for each task 2204.12511 §5.1 2204.12511 §5.2.

Poly-N and truncated cross-entropy (explored, not adopted)

Truncating cross-entropy to its first N terms needs several hundred terms to match full cross-entropy on ImageNet-1K with ResNet-50 2204.12511 §4.1. Poly-N changes the first N coefficients instead of dropping terms. In the paper's ResNet-50 runs, N=1 gives nearly the same gain as N=2 for less search 2204.12511 §4.2.

When to use which

The paper reports that training hyperparameters were kept as tuned for the baseline, which favours the baseline 2204.12511 §4.3 [arxiv:2204.12511 app.]. The reported Poly-1 gains are therefore a lower bound on what a retuned recipe might give, not a guarantee.

Implementation sketch

Plain cross-entropy and the Poly-1 change, with one-hot labels. Shapes are (batch, classes).

# Plain cross-entropy  [arxiv:2204.12511 §3] [eq. 1]
# Input:  logits (B, C); labels (B, C) one-hot
# Output: per-example loss (B,)

def cross_entropy(logits, labels):
    # Probability the model gives the true class  [§3]
    pt = sum(labels * softmax(logits), axis=-1)     # (B,)
    # Negative log of that probability  [eq. 1]
    return -log(pt)                                 # (B,)


# Poly-1: cross-entropy plus one perturbed term  [§4.3] [eq. 7]
# epsilon >= -1 keeps the first coefficient non-negative  [§4.3]
def poly1_cross_entropy(logits, labels, epsilon):
    pt = sum(labels * softmax(logits), axis=-1)     # (B,)
    # Baseline term plus epsilon times (1 - pt)  [eq. 7]
    return -log(pt) + epsilon * (1 - pt)            # (B,)

Choose epsilon on held-out data for each dataset and task. The paper's grid for the ImageNet classifier was 0 to 7 2204.12511 §5.1, and for the Mask R-CNN classification loss it was -1.0 to 1.0 2204.12511 §5.2.

Contradictions

None known across cards; only one card lists this concept.

Within the paper, the best epsilon_1 is positive for ImageNet-21K pretraining (2) and negative for COCO Mask R-CNN (-1) 2204.12511 §5.1 2204.12511 §5.2. The authors give observations from plots for why each direction helped, not a proof 2204.12511 §5.1 2204.12511 §5.2.

Papers

• PolyLoss: writes classification losses as weighted sums of powers of (1 - Pt), and adds one tunable term to cross-entropy or focal loss.