Label smoothing

Overview

Label smoothing replaces the one-hot target of a classification loss with a softened target. The labelled class keeps less than all of the probability mass, and the rest is spread over the other classes. The survey in the wiki says it is meant to counter overfitting and overconfidence [arxiv:2201.03299 §5.1]. Three cards use it in three ways: as a direct change to the loss 2110.06207 app. D, as part of the soft targets that Mixup-style augmentation produces in the released Conformer code (the paper itself gives no value) 2105.03889 §4.2, and as a schedule that stops smoothing part-way through training 2201.03299 §5.2. In every variant the target remains a probability vector.

Variants

Plain smoothed cross-entropy

The loss compares the prediction with a target in which the true class gets 1 − s and each other class gets s / (C − 1). The open-set paper uses s = 0.9 on TinyImageNet and s = 0 on the other benchmarks 2110.06207 app. D Open-Set Recognition: a Good Closed-Set Classifier is All You Need?. In its TinyImageNet ablation, adding this step to a recipe that already had 600 epochs, cosine restarts and RandAugment moved closed-set accuracy from 82.5 to 84.2, AUROC from 78.4 to 83.0 and OSCR from 70.3 to 74.3 [arxiv:2110.06207 app. C]. The ablation is cumulative, one step at a time, and the logit score was used throughout, so this is not a clean effect of label smoothing alone.

Smoothing carried by soft targets

The Conformer paper says it follows the DeiT regularization recipe but does not list the values 2105.03889 §4.2. The released training script sets label smoothing to 0.1, and applies it through the soft targets produced by Mixup and CutMix. The card reads the code as using a plain label-smoothing cross-entropy when Mixup is turned off Conformer: Local Features Coupling Global Representations for Visual Recognition, Paper vs code. The paper itself does not state these values.

Smoothing for part of training (TSLA)

TSLA smooths the targets only for the first part of training, then switches to hard 0 and 1 targets. The survey gives one setting: ResNet-18 on CIFAR-100, smoothing until epoch 160 of 200 2201.03299 §5.2 Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks. The survey calls TSLA the best label method, but that is its own opinion from a compiled table, not a test 2201.03299 §7.1. The survey gives no formulas for TSLA, so the schedule above is the only detail the wiki holds.

When to use which

Implementation sketch

Common core: build the soft target, then take cross-entropy against it. The variants differ only in where the smoothing value comes from and whether a mixed target replaces the one-hot target 2110.06207 app. D.

# Smoothed target for one label  [arxiv:2110.06207 app. D]
# Input:  label y (int), smoothing s in [0, 1), class count C
# Output: target t (C,), sums to 1

def smoothed_target(y, s, C):
    # Spread s over the C - 1 wrong classes  [arxiv:2110.06207 app. D]
    t = full(C, s / (C - 1))
    # The labelled class keeps the rest, 1 - s  [arxiv:2110.06207 app. D]
    t[y] = 1 - s
    return t

# Cross-entropy against the smoothed target  [arxiv:2110.06207 app. D]
# Input:  logits (B, C), labels (B,), smoothing s
# Output: scalar loss

def smoothed_loss(logits, labels, s):
    # Build a target per example  [arxiv:2110.06207 app. D]
    t = stack([smoothed_target(y, s, C) for y in labels])
    # Negative sum of target times log-probability  [arxiv:2110.06207 app. D]
    return -sum(t * log_softmax(logits)) / B

With soft targets from Mixup or CutMix, replace t with the mixed target and keep the same loss 2105.03889 §4.2.

Contradictions

None known between the cards on the effect itself. The values do not compare directly: 0.9 on TinyImageNet 2110.06207 app. D and 0.1 in the Conformer script and in the survey's example 2105.03889 §4.2 2201.03299 §5.2 are for different datasets and recipes. Only the open-set card states which convention its value uses, so the other two values may mean something else.

Papers

• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?: smoothing at s = 0.9 on TinyImageNet, with the convention stated, and a cumulative ablation.
• Conformer: Local Features Coupling Global Representations for Visual Recognition: smoothing of 0.1 applied through Mixup and CutMix soft targets in the released code.
• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks: taxonomy of label methods, including smoothing and TSLA; compiles reported results without rerunning them.