Expected calibration error

Overview

Expected calibration error (ECE) turns a calibration check into one number. Group the predictions into bins by their stated probability. In each bin, compare the mean predicted probability with the observed frequency of the event. Average the absolute gaps, weighted by how many predictions fall in each bin 2112.10327 §4.2.

The number is only as precise as the choice behind it. Three things must be fixed before a value means anything: which prediction is scored (one binary score, one class at a time, or the top class), how many bins are used, and how bin edges are placed. The survey treats ECE as a summary to read next to a proper scoring rule, because a model that always outputs the overall class frequencies gets an ECE of zero without being useful 2112.10327 §4.2–§4.4. The worked values in the survey come from a 30-instance toy example, so they show how the formulas behave and are not benchmark results 2112.10327 §4.2.

Variants

Binary ECE

Use one score per instance and one 0/1 label. This is the form used for a single decision, such as a spam filter's score 2112.10327 §4.2. The definition sums bin weight times the absolute gap between the observed positive rate and the mean score in that bin. Card: Classifier Calibration survey.

Classwise ECE

For a K-class model, score each class against the rest. Compute a binary ECE for each class and take the mean, so every class has the same weight 2112.10327 §4.3. This is the notion that checks the whole probability vector, since each class's column is held to account. Card: Classifier Calibration survey.

Confidence ECE

Score only the top class. The confidence is the largest probability, and the label is whether the top class was correct. Bin on confidence and compare accuracy with mean confidence 2112.10327 §4.4. It is the common form in deep-network reports, but it says nothing about the other classes: a model can score well on it and still be badly calibrated for the second-ranked class 2112.10327 §2.3. Card: Classifier Calibration survey.

Maximum calibration error (MCE)

Replace the weighted average with the largest bin gap. The binary and classwise forms exist, and the classwise form takes the worst gap over all classes and bins 2112.10327 §4.2, §4.3. MCE is sensitive to bins with few instances, which can dominate it. Card: Classifier Calibration survey.

When to use which

Calibration is a property of the score across many instances, not of one prediction. ECE is therefore a diagnosis tool. It does not show which fix to use. For post-hoc fixes such as temperature scaling, see temperature-scaling.

Implementation sketch

Confidence ECE, the most common form. Binary and classwise ECE differ only in which score and label enter the binning. Shapes are noted in the comments.

# Confidence ECE with M equal-width bins  [arxiv:2112.10327 §4.4]
# probs: (N, K) predicted vectors; labels: (N,) true class indices

def confidence_ece(probs, labels, num_bins):
    # Confidence is the top probability; correct if the argmax matches  [§4.4]
    conf = max(probs, axis=1)                     # (N,)
    correct = argmax(probs, axis=1) == labels     # (N,) booleans
    total = 0.0
    for m in range(num_bins):
        # Assign each instance to one of M equal-width bins  [§4.2]
        in_bin = bin_index(conf, num_bins) == m
        if count(in_bin) == 0:
            # Empty bins add nothing to the sum  [§4.2]
            continue
        # Compare observed accuracy with mean confidence in the bin  [§4.4]
        gap = mean(correct[in_bin]) - mean(conf[in_bin])
        total += count(in_bin) / len(conf) * abs(gap)
    return total

Classwise ECE calls the binary form once per class column and averages the results with equal weight 2112.10327 §4.3. Write it in the same style for the score and label of each class.

Contradictions

The survey's worked tables use 5 bins 2112.10327 §4.2, §4.4, and its calibration-map test uses 10 equal-width bins 2112.10327 §5.3. Its companion library defaults to 15 bins for ECE and MCE, while the paper's formulas are the same 2112.10327 §4.2. The bin count alone can change the value, so scores computed with different bin counts are not comparable. Card: Classifier Calibration survey, section "Paper vs code".

For how other wiki sources report ECE on their own benchmarks, see calibration.

Papers

• Classifier Calibration survey: defines binary, classwise and confidence ECE and MCE, shows how they relate to proper scoring rules, and gives a resampling test for calibration. Its values come from a small toy example.