---
kind: concept
slug: expected-calibration-error
title: Expected calibration error
summary: >-
  A binned gap between predicted probabilities and observed frequencies, averaged over bins weighted by size. Variants differ in what counts as the prediction: one binary score, each class one-vs-rest, or the top class only. Maximum calibration error takes the worst bin instead. Results depend on the bin count, so compare only runs with the same bins.
tags: [calibration, uncertainty-estimation, reliability-diagram, multiclass-classification, evaluation]
papers: ["2112.10327"]
related: [calibration, temperature-scaling]
status: draft
updated_at: 2026-11-08
---

## Overview

Expected calibration error (ECE) turns a calibration check into one number. Group
the predictions into bins by their stated probability. In each bin, compare the
mean predicted probability with the observed frequency of the event. Average the
absolute gaps, weighted by how many predictions fall in each bin [arxiv:2112.10327 §4.2].

The number is only as precise as the choice behind it. Three things must be
fixed before a value means anything: which prediction is scored (one binary
score, one class at a time, or the top class), how many bins are used, and how
bin edges are placed. The survey treats ECE as a summary to read next to a
proper scoring rule, because a model that always outputs the overall class
frequencies gets an ECE of zero without being useful [arxiv:2112.10327 §4.2–§4.4].
The worked values in the survey come from a 30-instance toy example, so they
show how the formulas behave and are not benchmark results [arxiv:2112.10327 §4.2].

## Variants

### Binary ECE

Use one score per instance and one 0/1 label. This is the form used for a
single decision, such as a spam filter's score [arxiv:2112.10327 §4.2]. The
definition sums bin weight times the absolute gap between the observed positive
rate and the mean score in that bin. Card: [Classifier Calibration survey](/papers/2112.10327).

### Classwise ECE

For a K-class model, score each class against the rest. Compute a binary ECE for
each class and take the mean, so every class has the same weight
[arxiv:2112.10327 §4.3]. This is the notion that checks the whole probability
vector, since each class's column is held to account. Card: [Classifier Calibration survey](/papers/2112.10327).

### Confidence ECE

Score only the top class. The confidence is the largest probability, and the
label is whether the top class was correct. Bin on confidence and compare
accuracy with mean confidence [arxiv:2112.10327 §4.4]. It is the common form in
deep-network reports, but it says nothing about the other classes: a model can
score well on it and still be badly calibrated for the second-ranked class
[arxiv:2112.10327 §2.3]. Card: [Classifier Calibration survey](/papers/2112.10327).

### Maximum calibration error (MCE)

Replace the weighted average with the largest bin gap. The binary and classwise
forms exist, and the classwise form takes the worst gap over all classes and bins
[arxiv:2112.10327 §4.2, §4.3]. MCE is sensitive to bins with few instances,
which can dominate it. Card: [Classifier Calibration survey](/papers/2112.10327).

## When to use which

| Situation | Use | Why |
|---|---|---|
| One probability per item, such as a yes/no score or a threshold decision | Binary ECE | Matches the single score the decision uses [arxiv:2112.10327 §4.2] |
| Several classes whose probabilities are all used downstream | Classwise ECE | Checks each class, not only the top one [arxiv:2112.10327 §4.3] |
| Only the reported top answer and its confidence are shown to users | Confidence ECE | Matches what is reported, but reports nothing about the other classes [arxiv:2112.10327 §2.3, §4.4] |
| A worst-case bound matters more than the average, and the test set is large | MCE, reported with ECE | Shows the worst bin; needs enough instances per bin to be stable [arxiv:2112.10327 §4.2] |
| Need to know whether a measured ECE differs from calibration by chance | A resampling test on the same measure | The survey's test keeps the scores fixed, redraws labels from the predicted vectors and compares the observed value with the resampled ones [arxiv:2112.10327 §6] |
| Comparing two methods | Same variant, same bin count, same bin edges | Values change with bins, so a mismatch invalidates the comparison [arxiv:2112.10327 §4.2] |

Calibration is a property of the score across many instances, not of one
prediction. ECE is therefore a diagnosis tool. It does not show which fix to use.
For post-hoc fixes such as temperature scaling, see [temperature-scaling](/concepts/temperature-scaling).

## Implementation sketch

Confidence ECE, the most common form. Binary and classwise ECE differ only in
which score and label enter the binning. Shapes are noted in the comments.

```pseudocode
# Confidence ECE with M equal-width bins  [arxiv:2112.10327 §4.4]
# probs: (N, K) predicted vectors; labels: (N,) true class indices

def confidence_ece(probs, labels, num_bins):
    # Confidence is the top probability; correct if the argmax matches  [§4.4]
    conf = max(probs, axis=1)                     # (N,)
    correct = argmax(probs, axis=1) == labels     # (N,) booleans
    total = 0.0
    for m in range(num_bins):
        # Assign each instance to one of M equal-width bins  [§4.2]
        in_bin = bin_index(conf, num_bins) == m
        if count(in_bin) == 0:
            # Empty bins add nothing to the sum  [§4.2]
            continue
        # Compare observed accuracy with mean confidence in the bin  [§4.4]
        gap = mean(correct[in_bin]) - mean(conf[in_bin])
        total += count(in_bin) / len(conf) * abs(gap)
    return total
```

Classwise ECE calls the binary form once per class column and averages the
results with equal weight [arxiv:2112.10327 §4.3]. Write it in the same style
for the score and label of each class.

## Contradictions

The survey's worked tables use 5 bins [arxiv:2112.10327 §4.2, §4.4], and its
calibration-map test uses 10 equal-width bins [arxiv:2112.10327 §5.3]. Its
companion library defaults to 15 bins for ECE and MCE, while the paper's
formulas are the same [arxiv:2112.10327 §4.2]. The bin count alone can change
the value, so scores computed with different bin counts are not comparable.
Card: [Classifier Calibration survey](/papers/2112.10327), section "Paper vs code".

For how other wiki sources report ECE on their own benchmarks, see
[calibration](/concepts/calibration).

## Papers

- [Classifier Calibration survey](/papers/2112.10327): defines binary, classwise and confidence ECE and MCE, shows how they relate to proper scoring rules, and gives a resampling test for calibration. Its values come from a small toy example.