---
kind: concept
slug: label-smoothing
title: Label smoothing
summary: >-
  Softening one-hot training targets so the classifier is less sure of its training labels. Covers a plain smoothed cross-entropy, smoothing carried by mixup soft targets, and TSLA's smooth-then-hard schedule. Trade-offs: the number means different things by convention, and the cited gains come from larger training recipes, not from label smoothing alone.
tags: [regularization, training, classification, loss]
papers: ["2105.03889", "2110.06207", "2201.03299"]
related: [data-augmentation, dropout, calibration, out-of-distribution-detection, open-set-recognition, convolutional-neural-network]
status: draft
updated_at: 2026-11-01
---

## Overview

Label smoothing replaces the one-hot target of a classification loss with a
softened target. The labelled class keeps less than all of the probability
mass, and the rest is spread over the other classes. The survey in the wiki
says it is meant to counter overfitting and overconfidence [arxiv:2201.03299
§5.1]. Three cards use it in three ways: as a direct change to the loss
[arxiv:2110.06207 app. D], as part of the soft targets that Mixup-style
augmentation produces in the released Conformer code (the paper itself gives
no value) [arxiv:2105.03889 §4.2], and as a schedule that stops smoothing
part-way through training [arxiv:2201.03299 §5.2]. In every variant
the target remains a probability vector.

## Variants

### Plain smoothed cross-entropy

The loss compares the prediction with a target in which the true class gets
1 − s and each other class gets s / (C − 1). The open-set paper uses s = 0.9
on TinyImageNet and s = 0 on the other benchmarks [arxiv:2110.06207 app. D]
[Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](/papers/2110.06207).
In its TinyImageNet ablation, adding this step to a recipe that already had
600 epochs, cosine restarts and RandAugment moved closed-set accuracy from
82.5 to 84.2, AUROC from 78.4 to 83.0 and OSCR from 70.3 to 74.3 [arxiv:2110.06207
app. C]. The ablation is cumulative, one step at a time, and the logit score
was used throughout, so this is not a clean effect of label smoothing alone.

### Smoothing carried by soft targets

The Conformer paper says it follows the DeiT regularization recipe but does
not list the values [arxiv:2105.03889 §4.2]. The released training script sets
label smoothing to 0.1, and applies it through the soft targets produced by
Mixup and CutMix. The card reads the code as using a plain label-smoothing
cross-entropy when Mixup is turned off [Conformer: Local Features Coupling
Global Representations for Visual Recognition](/papers/2105.03889), Paper vs
code. The paper itself does not state these values.

### Smoothing for part of training (TSLA)

TSLA smooths the targets only for the first part of training, then switches to
hard 0 and 1 targets. The survey gives one setting: ResNet-18 on CIFAR-100,
smoothing until epoch 160 of 200 [arxiv:2201.03299 §5.2]
[Avoiding Overfitting: A Survey on Regularization Methods for Convolutional
Neural Networks](/papers/2201.03299). The survey calls TSLA the best label
method, but that is its own opinion from a compiled table, not a test
[arxiv:2201.03299 §7.1]. The survey gives no formulas for TSLA, so the
schedule above is the only detail the wiki holds.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Single-label classifier, no soft-target augmentation, want a one-line change to the loss | Plain smoothed cross-entropy with a moderate s, after fixing the convention | The only card that isolates it is the TinyImageNet ablation, which is cumulative [arxiv:2110.06207 app. C] |
| Already using Mixup or CutMix | Smoothing carried by the soft targets, as in the released Conformer code | The card reads it as the code's choice, not a paper result [arxiv:2105.03889 §4.2] |
| Long schedule where you want less confidence late in training | A TSLA-style switch-off schedule | Described in the survey with one example setting; no card compares it with the plain form [arxiv:2201.03299 §5.2] |
| Using confidence scores for unknown-class detection | Check the effect on your own score before adopting a value | The open-set card reports label smoothing only with max-logit scoring [arxiv:2110.06207 app. C]; its effect on softmax-based scoring is not shown |

## Implementation sketch

Common core: build the soft target, then take cross-entropy against it. The
variants differ only in where the smoothing value comes from and whether a
mixed target replaces the one-hot target [arxiv:2110.06207 app. D].

```pseudocode
# Smoothed target for one label  [arxiv:2110.06207 app. D]
# Input:  label y (int), smoothing s in [0, 1), class count C
# Output: target t (C,), sums to 1

def smoothed_target(y, s, C):
    # Spread s over the C - 1 wrong classes  [arxiv:2110.06207 app. D]
    t = full(C, s / (C - 1))
    # The labelled class keeps the rest, 1 - s  [arxiv:2110.06207 app. D]
    t[y] = 1 - s
    return t
```

```pseudocode
# Cross-entropy against the smoothed target  [arxiv:2110.06207 app. D]
# Input:  logits (B, C), labels (B,), smoothing s
# Output: scalar loss

def smoothed_loss(logits, labels, s):
    # Build a target per example  [arxiv:2110.06207 app. D]
    t = stack([smoothed_target(y, s, C) for y in labels])
    # Negative sum of target times log-probability  [arxiv:2110.06207 app. D]
    return -sum(t * log_softmax(logits)) / B
```

With soft targets from Mixup or CutMix, replace `t` with the mixed target and
keep the same loss [arxiv:2105.03889 §4.2].

## Contradictions

None known between the cards on the effect itself. The values do not compare
directly: 0.9 on TinyImageNet [arxiv:2110.06207 app. D] and 0.1 in the
Conformer script and in the survey's example [arxiv:2105.03889 §4.2]
[arxiv:2201.03299 §5.2] are for different datasets and recipes. Only the
open-set card states which convention its value uses, so the other two values
may mean something else.

## Papers

- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](/papers/2110.06207): smoothing at s = 0.9 on TinyImageNet, with the convention stated, and a cumulative ablation.
- [Conformer: Local Features Coupling Global Representations for Visual Recognition](/papers/2105.03889): smoothing of 0.1 applied through Mixup and CutMix soft targets in the released code.
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](/papers/2201.03299): taxonomy of label methods, including smoothing and TSLA; compiles reported results without rerunning them.