Focal loss

Overview

Plain cross-entropy gives every training example the same kind of push, so a large number of easy examples can dominate the gradient when positives are rare. Focal loss multiplies the per-example loss by a modulating factor (1 - p_t)^{\gamma}, with \gamma \ge 0, where p_t is the model's probability for the true class. The factor is close to zero when p_t is near 1, so well-classified examples are down-weighted 2101.07511 eq. 2. An \alpha_t weight is often added on top to balance classes 2101.07511 §3.2.2.

The wiki cards use the loss in several shapes: binary classification with imbalanced classes 2101.07511 §3.2.1, per-pixel heatmaps where positives are single pixels 2304.05170 §4.3 or object centres 2011.04841 §3.2, per-pixel depth distributions over bins 2103.01100 §3.5, per-step action classes in a video 2202.07925 §3.3, softmax classification of boundary candidates in a temporal localiser 2103.13137 §3.4, confidence values for 2D-3D matches 2205.12257 §3.2, classification over many set-prediction outputs 2202.13065 §3.4, a generalised form for continuous targets 2201.01501 §3.3, a per-node focusing parameter in a scene-graph loss 2003.12962 eq. 8, a per-anchor confidence in a one-stage 2D detector 1905.12365 §4.2, a survey's pixel-wise binary form with separate foreground and background weights 1910.07655 §4.3, a classification term in a DETR-style decoder 2108.06152 §3.5, a category-agnostic term in a vision-language tokenizer [arxiv:2305.11175 app. B], and a survey's generic definition with no default values 2307.02694 §5.1.6. In every case the card gives the loss as its authors define it, and the values for \alpha and \gamma differ from card to card.

Variants

Alpha-balanced focal loss (binary classification)

The form in the COVID-19 federated paper. The loss is -\alpha_t (1 - p_t)^{\gamma} \log p_t 2101.07511 eq. 2. Setting \gamma = 0 leaves an \alpha_t-weighted cross-entropy. The card does not say how \alpha_t is set per class or per sample, and gives no value for \alpha or \gamma 2101.07511 §3.2.2. Card: Federated COVID-19 diagnosis from X-ray and ultrasound.

Foreground and background weights (survey form)

The segmentation review writes a binary version with two weights. The foreground term is multiplied by \alpha and the background term by 1 - \alpha, both with the (1 - p)^{\gamma} modulator, and the review states that \gamma = 0 gives binary cross-entropy 1910.07655 §4.3. Here \alpha is not the class weight \alpha_t of the variant above, so a value copied from one card does not carry over to the other. The review's plots use \alpha = 0.25 and \gamma = 2, which it chose for its own loss-behaviour figures; these are not recommended settings 1910.07655 §4.4. The review runs no training experiments of its own. Card: Deep Semantic Segmentation of Natural and Medical Images: A Review.

Penalty-reduced focal loss on heatmaps

The tracker uses a CornerNet-style loss over a centre heatmap. Pixels where the target is exactly 1 get (1 - \hat h)^{\gamma} \log \hat h. All other pixels get an extra (1 - h)^{\beta} factor, so pixels near a centre are down-weighted 2304.05170 eq. 3. The card sets \gamma = 2 and \beta = 4 and says these follow CornerNet; the paper has no alpha term 2304.05170 §4.3. Card: SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes.

The radar-camera detector trains its primary centre heatmap with the same shape, written with exponents \alpha and \beta. The paper states neither value. Its code sets them to 2 and 4 2011.04841 §3.2 [code: src/lib/model/losses.py#L51-L55]. These are the same numbers the tracker card reports, but the radar paper's \alpha is an exponent, not a class weight. Card: CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection.

Focal loss over depth bins

The monocular 3D detector trains its depth-distribution network with a focal loss on each feature-map pixel. The target is a one-hot depth bin, and the loss is averaged over all pixels 2103.01100 §3.5 2103.01100 eq. 3. The card gives \gamma = 2.0 and an alpha of 3.25 for pixels inside a 2D object box and 0.25 elsewhere 2103.01100 §3.5. The code uses a single alpha of 0.25 and multiplies foreground and background losses by 13 and 1 CaDDN.yaml. The two give the same effective per-pixel factors, but by different mechanisms (see Contradictions). Card: Categorical Depth Distribution Network for Monocular 3D Object Detection.

Focal classification over time steps

The temporal action localiser sums a focal loss over action classes at every time step and every pyramid level. The sum is divided by the number of positive steps 2202.07925 eq. 7. The paper states neither \alpha nor \gamma 2202.07925 §3.3. The code uses \alpha = 0.25 and \gamma = 2.0 losses.py. Card: ActionFormer: Localizing Moments of Actions with Transformers.

Softmax focal classification with a class-split alpha (boundary-based localiser)

The anchor-free action localiser applies a softmax focal loss to both its coarse and its refined classification heads. Both terms enter the total loss with a localisation term and a quality term 2103.13137 §3.4 2103.13137 eq. 11 2103.13137 eq. 12. The paper states neither \alpha nor \gamma. The code uses \gamma = 2, and an \alpha of 0.25 on the background class and 0.75 on the action classes multisegment_loss.py multisegment_loss.py. Note the placement: the review's \alpha sits on the foreground, so this 0.25 weights the opposite class (see Contradictions). Card: Learning Salient Boundary Feature for Anchor-free Temporal Action Localization.

Focal confidence loss for 2D-3D matches

The pose-estimation paper applies a focal loss to match confidences. Each match is scored against a ground-truth label, and the loss is -(1 - C')^{\gamma} \log C', with C' the match confidence when the pair is a ground-truth match and 1 - C otherwise 2205.12257 §3.2. The paper does not state \gamma. The code uses \gamma = 2, \alpha = 0.5 and class weights of 0.5 for positives and negatives 2205.12257 §3.2 [code: configs/experiment/train_GATsSPG.yaml#L52-L55]. Card: OnePose: One-Shot Object Pose Estimation without CAD Models.

Focal classification in set prediction

The crowd-localisation transformer trains its classification head with a focal loss over all predictions, and adds an L1 point term with weight 2.5 2202.13065 §3.4. The card states neither \alpha nor \gamma. Its ablations vary transformer size, matching cost and the number of neighbours, not the loss 2202.13065 tab. 7 2202.13065 tab. 8 2202.13065 tab. 9. Card: An End-to-End Transformer Model for Crowd Localization.

Focal confidence in an anchor-based 2D head

The monocular 3D detector uses a RetinaNet-style 2D head. Each anchor's confidence is trained with a focal loss, where the target is 1 if the anchor's reference box has an IoU above 0.5 with a ground-truth box, and 0 otherwise 1905.12365 §4.2 1905.12365 §7.2. The card gives the loss only as a weighted form with \alpha and \gamma, and states neither value 1905.12365 §4.2. The box regression on positive anchors uses a signed IoU loss, not focal loss 1905.12365 eq. 1. Card: Disentangling Monocular 3D Object Detection.

Unified focal loss for continuous targets

The multi-view stereo paper extends focal loss to targets between 0 and 1. The generalised form scales by the target, and the unified form divides the error by the positive target and bounds the factor with a sigmoid-like function with base b 2201.01501 §3.3 2201.01501 eq. 6 2201.01501 eq. 9. Positives carry no \alpha, and negatives use a \alpha_- that the paper sets per stage. \gamma is set per stage to 2, 1 and 0, from coarse to fine 2201.01501 §4. Card: Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation.

Focal classification in a DETR-style decoder

The Conditional DETR card trains its per-query classifier with focal loss, inherited from Deformable DETR, with 300 queries 2108.06152 §3.5. The loss is a sigmoid classification term. The code's focal values are \alpha = 0.25 and \gamma = 2, and the classification term has weight 2 main.py conditional_detr.py. The paper does not state the weights 2108.06152 §3.5. Card: Conditional DETR for Fast Training Convergence.

Node-priority focusing parameter (scene graphs)

The scene-graph generator replaces the fixed focusing parameter with one per node. A node's priority is the share of ground-truth triplets that contain it 2003.12962 eq. 6. That share sets \gamma through a mapping capped at 2, with an exponent \mu that controls how strongly priority matters 2003.12962 eq. 7. The per-node loss is then a focal-style term with this \gamma 2003.12962 eq. 8. The paper's best \mu is 4 2003.12962 §4.4 2003.12962 tab. 6. In its ablation, the \mu = 4 setting sits 0.3 points above a plain focal-loss row at SGCLS R@20 and R@100 (36.1 and 40.1 against 35.8 and 39.8) 2003.12962 tab. 6. Card: GPS-Net: Graph Property Sensing Network for Scene Graph Generation.

Category-agnostic focal term in a vision-language tokenizer

The VisionLLM card uses a focal loss on the scores of object queries in its tokenizer, with no class labels, plus an L1 term on centre points 2305.11175 app. B. The card leaves the focal parameters unstated and does not say how queries are matched to objects 2305.11175 app. B. Card: VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks.

Generic multi-class form in a loss survey

The loss survey gives -\alpha_t (1 - p_t)^{\gamma} \log p_t and describes \alpha_t as class-dependent, often inversely proportional to class frequency, with no formula for it. It says \gamma is commonly between 1 and 5, with no source and no value per task 2307.02694 §5.1.6. The survey gives no default for either. Use it for the definition, not for values. Card: Loss Functions and Metrics in Deep Learning.

Named but not specified

The YOLOv8 review lists a focal term for classification in its training loss. It gives no formula, weight or \gamma 2408.15857 §3.4.1. Do not implement this as a specific variant from that card. Card: What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector.

When to use which

Implementation sketch

Two shapes cover most cards. The first is the alpha-balanced form for one prediction, following the paper's equation 2. The second is the penalty-reduced form on a heatmap. Batch averaging and the choice of \alpha_t are left to the caller, since the COVID-19 card does not specify them. The survey's foreground and background weighting is a third form; use its card only if you need that exact convention.

# Alpha-balanced focal loss for one sample  [arxiv:2101.07511 eq. 2]
# p_t: probability the model gives the true class, in (0, 1)
# alpha_t: class weight, rule not stated in the paper  [§3.2.2]
# gamma: focusing parameter, gamma >= 0, value not stated  [§3.2.2]

def focal_loss(p_t, alpha_t, gamma):
    # Modulating factor: near 0 for easy examples, near 1 for hard ones  [eq. 2]
    modulator = (1 - p_t) ** gamma
    # Cross-entropy term, scaled by the modulator and the class weight  [eq. 2]
    return -alpha_t * modulator * log(p_t)

# Mean over a batch, as a training loss  [§3.2.2]
def batch_loss(p_t_list, alpha_t_list, gamma):
    losses = [focal_loss(p, a, gamma) for p, a in zip(p_t_list, alpha_t_list)]
    return sum(losses) / len(losses)

# Penalty-reduced focal loss on a heatmap  [arxiv:2304.05170 eq. 3]
# h: target heatmap, exactly 1 at each centre; p: predicted heatmap in (0, 1)
# gamma = 2 and beta = 4 in the card  [§4.3]

def heatmap_focal_loss(p, h, gamma=2, beta=4):
    # Positives: pixels whose target is exactly 1  [eq. 3]
    pos = h == 1
    pos_term = (1 - p[pos]) ** gamma * log(p[pos])
    # Negatives: extra (1 - h)^beta factor shrinks pixels near a centre  [eq. 3]
    neg = ~pos
    neg_term = (1 - h[neg]) ** beta * p[neg] ** gamma * log(1 - p[neg])
    # Negated sum, as in eq. 3; normalisation is not shown there  [eq. 3]
    return -(pos_term.sum() + neg_term.sum())

Contradictions

None between papers' reported numbers, because no two cards report values for the same setting. Paper-versus-code gaps are noted in the variants above. The pose-estimation card's code adds an \alpha and class weights the paper does not state 2205.12257 §3.2. The crowd card's code uses a focal-style matching cost where the paper subtracts the predicted confidence 2202.13065 §3.3 matcher.py. The depth-estimation card states a per-pixel alpha of 3.25 and 0.25, while its code uses one alpha of 0.25 with 13 and 1 multipliers. The card notes the effective factors match, but the paper does not say which form produced its results 2103.01100 §3.5 CaDDN.yaml ddn_loss.py.

The scene-graph card has two gaps between paper and code. The paper's best exponent is \mu = 4, but both released code paths use 5 2003.12962 §4.4 nps_loss.py. The paper does not say whether the focusing weight is differentiated. Both code paths detach it, so only the -\log p term receives gradient 2003.12962 eq. 8 nps_loss.py. A standard focal-loss implementation does not detach it, so the two can train differently.

The survey gives γ as commonly between 1 and 5, without a source 2307.02694 §5.1.6. The cards that state γ use 2 2304.05170 §4.3 2103.01100 §3.5 2108.06152 §3.5, which lies in that range. The unified form sets γ per stage to 2, 1 and 0 2201.01501 §4, and its stage value of 0 lies outside the survey's range. The survey does not say where its range comes from.

The weighting conventions also differ. The COVID-19 card's \alpha_t is a class weight on the whole term 2101.07511 eq. 2. The survey's \alpha weights the foreground term and 1 - \alpha weights the background term 1910.07655 §4.3. The anchor-free localiser's code puts 0.25 on the background and 0.75 on the action classes multisegment_loss.py, so its smaller value sits on the opposite class to the survey's convention. The same number therefore means different things in the cards. The radar-camera card's \alpha is a heatmap exponent 2011.04841 §3.2, a third meaning of the same symbol.

Papers

• Disentangling Monocular 3D Object Detection: focal confidence on 2D anchors, with alpha and gamma not stated.
• Deep Semantic Segmentation of Natural and Medical Images: A Review: survey of the binary focal form with foreground and background weights; no training values, plots use alpha 0.25 and gamma 2 as the review's own choice.
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation: node-priority focusing parameter in a scene-graph loss, with mu 4 as the paper's best and the code using 5.
• CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection: penalty-reduced focal loss on a centre heatmap, with exponents not stated in the paper and 2 and 4 in the code.
• Federated COVID-19 diagnosis from X-ray and ultrasound: alpha-balanced focal loss for imbalanced binary COVID-19 classification, with no reported alpha or gamma.
• Categorical Depth Distribution Network for Monocular 3D Object Detection: focal loss per pixel over depth bins, with gamma 2 and foreground/background alpha from the paper.
• Learning Salient Boundary Feature for Anchor-free Temporal Action Localization: softmax focal classification on coarse and refined heads, with alpha and gamma taken from the code.
• ActionFormer: Localizing Moments of Actions with Transformers: focal classification over time steps, with alpha and gamma taken from the code.
• Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation: unified focal loss for continuous targets, with per-stage gamma and alpha in an ablation.
• An End-to-End Transformer Model for Crowd Localization: focal classification over all set-prediction outputs, with no loss ablation.
• OnePose: One-Shot Object Pose Estimation without CAD Models: focal loss on 2D-3D match confidences, with gamma from the code.
• SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes: penalty-reduced focal loss on a centre heatmap, with gamma 2 and beta 4.
• What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector: names a focal classification term with no formula.
• Conditional DETR for Fast Training Convergence: focal classification in a DETR-style decoder, with alpha 0.25 and gamma 2 from the code; the ablation changes the loss and query count together.
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks: category-agnostic focal loss on tokenizer query scores, with its parameters not stated.
• Loss Functions and Metrics in Deep Learning: survey of the focal loss definition, with gamma commonly 1 to 5 and no default values.