---
kind: concept
slug: cross-entropy-loss
title: Cross-entropy loss
summary: >-
  The standard loss for training classifiers: the negative log of the probability the model gives the true class. Its tail of higher-order terms can be re-weighted, e.g. Poly-1 adds one tuned term epsilon * (1 - Pt), which reported small gains on ImageNet, COCO and Waymo when epsilon is tuned per task.
tags: [loss-function, classification]
papers: ["2204.12511"]
related: [focal-loss, label-smoothing]
status: draft
updated_at: 2026-11-08
---

## Overview

For a classifier with one-hot labels, cross-entropy is the negative log of the
probability the model assigns to the true class, written Pt [arxiv:2204.12511
§3] [arxiv:2204.12511 eq. 1]. It is the default loss that other classification
losses are compared against. The PolyLoss paper rewrites it as a sum of powers
of (1 - Pt) with fixed coefficients, and shows that the same form covers focal
loss. Every variant on this page is a non-negative weighted sum of powers of
(1 - Pt) with different coefficients [arxiv:2204.12511 §3].

This page currently rests on one paper card. Other cards in the wiki mention
cross-entropy in passing, but none lists this concept, so none is used here.

## Variants

### Plain cross-entropy

Minus log Pt. Its gradient with respect to Pt begins with a constant term, so
every power of (1 - Pt) contributes with a fixed weight [arxiv:2204.12511
eq. 3]. This is the baseline in every comparison in the paper. Card:
[PolyLoss](/papers/2204.12511).

### Focal loss

Multiplies cross-entropy by (1 - Pt) raised to gamma. The paper shows this
shifts every coefficient of the polynomial by gamma rather than changing their
values [arxiv:2204.12511 eq. 2] [arxiv:2204.12511 §3]. Gamma is 2 in the
paper's PointPillars and RSN experiments [arxiv:2204.12511 §5.3]. See
[Focal loss](/concepts/focal-loss).

### Poly-1 (add one tuned term)

Adds epsilon_1 * (1 - Pt) to cross-entropy, with epsilon_1 >= -1 [arxiv:2204.12511
§4.3] [arxiv:2204.12511 eq. 7]. The same change applied to focal loss is
described in [arxiv:2204.12511 §5.3]. Epsilon_1 is set by grid search on
held-out data for each task [arxiv:2204.12511 §5.1] [arxiv:2204.12511 §5.2].

### Poly-N and truncated cross-entropy (explored, not adopted)

Truncating cross-entropy to its first N terms needs several hundred terms to
match full cross-entropy on ImageNet-1K with ResNet-50 [arxiv:2204.12511 §4.1].
Poly-N changes the first N coefficients instead of dropping terms. In the
paper's ResNet-50 runs, N=1 gives nearly the same gain as N=2 for less search
[arxiv:2204.12511 §4.2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| No budget to search a new hyperparameter | Plain cross-entropy | Poly-1 needs a tuned epsilon_1 per task [arxiv:2204.12511 §5.1] [arxiv:2204.12511 §5.2] |
| Classification pretraining where a small grid search is affordable | Poly-1 | Reported ImageNet-21K top-1 46.4 vs 45.8 with epsilon_1 = 2 [arxiv:2204.12511 §5.1] |
| ResNet-50 on ImageNet-1K with default training recipe | Poly-1 with N=1 | Reported 76.7 vs 76.3 for cross-entropy; the paper's text does not give a rule for choosing epsilon_1 [arxiv:2204.12511 §4] [arxiv:2204.12511 §4.3] |
| Object detection with a cross-entropy classification head | Poly-1 on that head | Only the classification loss was replaced; reported COCO box AR 49.7 vs 47.2 for Mask R-CNN [arxiv:2204.12511 §5.2] |
| Lidar detection already trained with focal loss | Poly-1 on focal loss, epsilon per class | Small AP gains on Waymo; RSN epsilon_2 is -0.4 for vehicle and +0.2 for pedestrian [arxiv:2204.12511 §5.3] |
| Need a fixed, parameter-free loss across tasks | Plain cross-entropy | The paper finds the best epsilon_1 changes sign between tasks, so one value does not transfer [arxiv:2204.12511 §5.1] [arxiv:2204.12511 §5.2] |

The paper reports that training hyperparameters were kept as tuned for the
baseline, which favours the baseline [arxiv:2204.12511 §4.3] [arxiv:2204.12511
app.]. The reported Poly-1 gains are therefore a lower bound on what a retuned
recipe might give, not a guarantee.

## Implementation sketch

Plain cross-entropy and the Poly-1 change, with one-hot labels. Shapes are
(batch, classes).

```pseudocode
# Plain cross-entropy  [arxiv:2204.12511 §3] [eq. 1]
# Input:  logits (B, C); labels (B, C) one-hot
# Output: per-example loss (B,)

def cross_entropy(logits, labels):
    # Probability the model gives the true class  [§3]
    pt = sum(labels * softmax(logits), axis=-1)     # (B,)
    # Negative log of that probability  [eq. 1]
    return -log(pt)                                 # (B,)


# Poly-1: cross-entropy plus one perturbed term  [§4.3] [eq. 7]
# epsilon >= -1 keeps the first coefficient non-negative  [§4.3]
def poly1_cross_entropy(logits, labels, epsilon):
    pt = sum(labels * softmax(logits), axis=-1)     # (B,)
    # Baseline term plus epsilon times (1 - pt)  [eq. 7]
    return -log(pt) + epsilon * (1 - pt)            # (B,)
```

Choose epsilon on held-out data for each dataset and task. The paper's grid
for the ImageNet classifier was 0 to 7 [arxiv:2204.12511 §5.1], and for the
Mask R-CNN classification loss it was -1.0 to 1.0 [arxiv:2204.12511 §5.2].

## Contradictions

None known across cards; only one card lists this concept.

Within the paper, the best epsilon_1 is positive for ImageNet-21K pretraining
(2) and negative for COCO Mask R-CNN (-1) [arxiv:2204.12511 §5.1]
[arxiv:2204.12511 §5.2]. The authors give observations from plots for why each
direction helped, not a proof [arxiv:2204.12511 §5.1] [arxiv:2204.12511 §5.2].

## Papers

- [PolyLoss](/papers/2204.12511): writes classification losses as weighted sums of powers of (1 - Pt), and adds one tunable term to cross-entropy or focal loss.