Temperature scaling

Overview

A classifier that outputs logits can be over-confident. Temperature scaling fixes this after training by dividing every logit by one number t > 0 before the softmax. The top class stays the same, because dividing by a positive number keeps the order of the logits, but the confidence changes. The calibration fixes for neural networks (temperature, matrix and vector scaling) act on logits and assume a softmax output layer 1910.12656 §1.

The Dirichlet calibration paper places temperature scaling inside a wider family. Temperature scaling is the case W = I/t, b = 0, applied to the probability vector softmax(z) 1910.12656 §3 app. B, Prop. 1. The paper's other members of the family are described in the variants below.

Temperature also appears inside a training loss. LogitNorm uses a temperature during training rather than fitting one afterwards, so it is a different method that shares the word. It is described in its own variant below.

Variants

Temperature scaling (one scalar on logits)

One parameter t, fitted on held-out data. The paper calls this a special case of its family, not a separate method 1910.12656 §3. Results are author-reported. On 21 UCI datasets with 11 classifiers, the average log-loss rank is 4.61, against 2.25 for Dirichlet L2 and 2.92 for beta calibration; uncalibrated models rank 5.54 1910.12656 tab. 2. On the p-cw-ECE measure, temperature scaling ranks 4.37, against 2.34 for Dirichlet L2 1910.12656 tab. 1. On 14 deep-network cases, it has the average cw-ECE rank 3.71 among the three general-purpose methods, against 3.79 for Dirichlet L2 and 2.79 for Dirichlet ODIR 1910.12656 tab. 3.

Vector and matrix scaling (per-class parameters on logits)

The card names vector and matrix scaling as logit-based fixes [arxiv:1910.12656 §1] and says they are not general-purpose 1910.12656 §4.2. In the deep-network study, vector scaling has the best average cw-ECE rank of all six methods, 2.29 1910.12656 tab. 3. Matrix scaling with ODIR regularisation (MS-ODIR) has the best average log-loss rank, 1.64 1910.12656 tab. 4. The card does not define these methods. Their definitions are in Guo et al. (ICML 2017), which has no card here, so this page does not describe them further.

Dirichlet calibration (linear map on log-probabilities)

Card: Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. The method takes any probability vector, not only logits, and applies a k × k matrix and a bias to its logarithm, followed by a softmax [arxiv:1910.12656 §3]. It has k² + k parameters, so 10,100 for 100 classes 1910.12656 §3. Regularisation comes in two forms: L2 (Dirichlet L2) and off-diagonal and intercept regularisation (ODIR). On 21 UCI datasets, Dirichlet L2 has the best average log-loss rank, 2.25 1910.12656 tab. 2. On deep networks, Dirichlet ODIR has the best average cw-ECE rank of the three general-purpose methods, 2.79, and an average log-loss rank of 2.93, against 3.50 for temperature scaling 1910.12656 tab. 3 1910.12656 tab. 4. The paper says this difference in the deep-network results is "without statistical significance" 1910.12656 §4.2.

Training-time temperature (LogitNorm)

Not a post-hoc fit. LogitNorm replaces cross-entropy on raw logits with cross-entropy on logits divided by their L2 norm and by a temperature tau. Each training logit vector then has length 1/tau 2205.09310 §3.2 eq. 4. Tau is a training hyperparameter, tuned on Gaussian noise used as an OOD validation set 2205.09310 §4.1. The main evaluation is OOD detection. The softmax score's average FPR95 on CIFAR-10 with WRN-40-2 falls from 49.52% to 15.65% over six OOD sets 2205.09310 tab. 1. Calibration is reported in one table. Without temperature scaling, the LogitNorm model's ECE on CIFAR-10 is 66.95%, against 3.20% for cross-entropy. With temperature scaling, it is 0.41%, against 0.66% 2205.09310 tab. 4. These are author-reported, on one architecture, and the text does not discuss the uncalibrated row 2205.09310 §4.2. Card: Mitigating Neural Network Overconfidence with Logit Normalization.

When to use which

Fit any of these on held-out data, not training data. The Dirichlet loss is defined over validation instances 1910.12656 §3. Fitting on training data would not measure the miscalibration the method corrects.

Implementation sketch

The card gives the log-loss fit for the whole family 1910.12656 §3. Applying it with W = I/t and b = 0 is this page's reading, not a separate result in the card. Dividing log-probabilities by t gives the same softmax as dividing logits by t, because the softmax removes the constant log-normaliser.

# Temperature scaling as the one-parameter case of the linear form [§3]
# Input:  logits z (n, k) from a softmax model; labels y (n,) on held-out data
# Output: scalar t > 0; calibrated probabilities are softmax(z / t)

def fit_temperature(z, y):
    # Minimise log-loss over t > 0, on validation data only  [§3, "Fitting"]
    # Parametrise by log t so that t stays positive
    ...
    return t

def calibrate(z, t):
    # Same as W = I / t, b = 0 in softmax(W ln q + b) with q = softmax(z)  [§3]
    return softmax(z / t)

Contradictions

The paper's text and its average ranks point in different directions on Dirichlet ODIR against temperature scaling. The text says Dirichlet ODIR beats temperature scaling on cw-ECE "at a small log-loss cost" [arxiv:1910.12656 §4.2]. The average log-loss ranks put Dirichlet ODIR ahead, 2.93 against 3.50 1910.12656 tab. 4. Both are the paper's results. This page does not resolve them.

The two papers' calibration numbers are not comparable. The Dirichlet study reports average ranks over 21 UCI datasets and 14 deep-network cases 1910.12656 tab. 3. The LogitNorm table gives one ECE per setting, for CIFAR-10 and CIFAR-100 on WRN-40-2 2205.09310 tab. 4. This page does not combine them.

Papers

• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration: places temperature scaling as a special case of a wider family, with deep-network and non-neural comparisons.
• Mitigating Neural Network Overconfidence with Logit Normalization: uses a temperature inside the training loss for OOD detection; reports ECE with and without post-hoc temperature scaling.
• Guo et al., "On Calibration of Modern Neural Networks", ICML 2017: the source of temperature, vector and matrix scaling as named in the card. No card here, so nothing from it is used.