Paper · 2022-09-17 · stat.ML, cs.LG, math.ST
A review of predictive uncertainty estimation with machine learning
Review of probabilistic prediction: how to score predicted quantiles, expectiles, distributions and intervals (quantile, expectile, CRPS, interval and log scores), and which model families issue them (Bayesian, quantile and GAMLSS models, forests, boosting, neural networks, ensembles). Its one simulation finds GAMLSS scores better than quantile regression on quantile scores (author-reported).
Uncertainty estimationProper scoring rulesQuantile lossTime series forecastingprobabilistic-forecastingreviewscoring-rulesuncertainty
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Most machine learning models answer a question with a single number, such as tomorrow's temperature or a house price. That number says nothing about how sure the model is. This review asks how a model can instead report a range of likely outcomes, or the full spread of possibilities, and how to tell whether such a report is any good.
It covers two sides. The first is the scoring side: rules that reward a forecast for being sharp (narrow) and honest (its stated chances match what happens). The second is the modelling side: the families of methods that produce these forecasts, from classic statistics to random forests, boosting and neural networks. The authors argue the two sides belong together, because the scoring rule you train with should match what you want to predict. They also list open problems that current methods do not solve well.
TL;DR
- A review, not a new method: it organises scoring rules and model families for probabilistic prediction §1.
- Each target needs a matching scoring function: quantiles and expectiles have their own consistent losses; a full distribution needs a proper scoring rule such as the CRPS or the log score §3.
- Model families are grouped as Bayesian and statistical, quantile and expectile regression, GAMLSS, forests, boosting, neural networks and combinations §4–§8.
- The only numerical content of its own is a small simulation: quantile regression versus GAMLSS, at two sample sizes §7, tab. 2.
Problem
Point predictions are standard in supervised learning, but they discard how uncertain the outcome is. Predicting a probability distribution, or several quantiles of it, supports better decisions §1. The difficulty is that many methods, scores and terminology exist, scattered across statistics, forecasting and machine learning, and existing reviews each cover only some of the topic (Table 1 of the paper compares them) §2.2, tab. 1. The paper also notes that scoring of predictive distributions and algorithms for them have rarely been treated together §2.2.
Assumptions behind the basic predictive distribution: the Bayesian predictive distribution eq. 3 is derived under independence of the observations, and it uses a prior on the parameters; in most cases it is simulated with MCMC §2.1. Time-series forecasting breaks the independence assumption, so the formulation is changed §2.1, eq. 31 §4.3.
Method
This is a review, so the "method" is the framework the paper proposes. Read it as a checklist for designing a probabilistic predictor §10.1, fig. 4, fig. 5.
- Decide what to predict. A functional (a quantile, an expectile, a prediction interval) or the full predictive distribution §10.1, fig. 4.
- Choose a scoring function matched to that target.
- Quantile level α: the tilted absolute loss eq. 9 eq. 10. It is consistent for the α-quantile only in the generalised piecewise linear form eq. 12 §3.1.
- Expectile level τ: the asymmetric squared loss eq. 14 §3.2. At τ = 1/2 it reduces to half the squared error eq. 15.
- Full distribution: a proper scoring rule eq. 17 eq. 18 §3.3. The CRPS, computed from samples as in the ensemble form eq. 23, and the log score eq. 26 are the most used. The log score is the only local rule §3.4.
- Prediction interval: the central interval score eq. 25 §3.3.4.
- Several quantiles at once: the quantile-based rule with weights eq. 24 §3.3.3.
- Pick a model family fig. 5 §5 §6:
- Linear, Bayesian or parametric: linear models with Bayesian predictive distributions §4.1, eq. 27–30; quantile and expectile regression linear in parameters §4.2; time-series variants §4.3; copulas §4.4.
- Distributional models: GAMLSS, which models every parameter of a chosen distribution as a function of predictors §5.2.
- Trees: quantile regression forests, which estimate quantiles from the weighted sample of tree responses §5.3; boosting with quantile or distributional losses, e.g. gradient boosting, XGBoost, LightGBM, NGBoost §5.4.
- Neural networks: Bayesian networks, dropout-based and ensemble-based uncertainty §6.1 §6.2; quantile and expectile networks §6.3; distributional networks §6.4.
- Other: Gaussian processes §5.1, conformal prediction §5.5.3, Bayesian additive regression trees §5.5.1.
- Combine, if needed. Linear or logarithmic opinion pools eq. 34 [eq. 36]; Bayesian model averaging eq. 37–39 §8.2; stacking, which estimates weights by out-of-sample performance §8.3. The equally weighted linear pool is hard to beat in practice, as cited by the paper §8.1.
The pseudocode below follows the equations as printed. Where the print is wrong, the comment says so; see Pitfalls.
# Quantile (pinball) loss at level alpha [eq. 9] [eq. 10]
# Input: alpha in (0, 1); prediction z; observation y
# Output: scalar penalty, >= 0
def quantile_loss(alpha, z, y):
# residual between observation and prediction [eq. 10]
residual = y - z
# weight alpha for under-prediction, alpha - 1 otherwise [eq. 9]
# rho_alpha(u) = u * (alpha - 1[u <= 0])
weight = alpha - (1.0 if residual <= 0 else 0.0)
return residual * weight# CRPS estimated from K ensemble members [eq. 23] [§3.3.2]
# Input: members (K,) in any order; observation y
# Output: scalar score, lower is better
def crps_ensemble(members, y):
# eq. 23 is defined on the ordered members z_(1) <= ... <= z_(K)
ordered = sort(members)
K = len(ordered)
total = 0.0
for i in range(1, K + 1):
# i-th order statistic, 1-based [eq. 23]
member = ordered[i - 1]
# indicator 1[y - z_(i) < 0]; printed as z_i in [eq. 23]
indicator = 1.0 if y - member < 0 else 0.0
total += (member - y) * (K * indicator - i + 0.5)
return 2.0 * total / K**2# Central prediction interval score [eq. 25] [§3.3.4]
# Input: lower, upper (quantiles at alpha/2 and 1 - alpha/2); observation y
# Output: scalar score, lower is better
def interval_score(lower, upper, alpha, y):
# width of the interval: the sharpness part of the score
width = upper - lower
# penalty when y falls below the lower bound [eq. 25]
below = (2.0 / alpha) * (lower - y) * (1.0 if y - lower < 0 else 0.0)
# penalty when y falls above the upper bound [eq. 25]
above = (2.0 / alpha) * (y - upper) * (1.0 if upper - y < 0 else 0.0)
return width + below + above# Predictive distribution, Bayesian Gaussian linear model [§4.1]
# [eq. 27] [eq. 28] [eq. 29] [eq. 30]; uniform prior on (beta, log sigma)
# Input: X (n, p) training design; y (n,); X_new (m, p) future rows
# Output: multivariate Student-t with n - p degrees of freedom
def linear_predictive(X, y, X_new):
n, p = shape(X)
# least-squares coefficient estimate [eq. 28]
coef = solve(transpose(X) @ X, transpose(X) @ y)
# residual variance with n - p degrees of freedom [eq. 30] [§4.1]
residual = y - X @ coef
s2 = dot(residual, residual) / (n - p)
# printed as V_u = X'X in [eq. 29]; the inverse is used here, see Pitfalls
V = inverse(transpose(X) @ X)
m = len(X_new)
location = X_new @ coef
scale = s2 * (identity(m) + X_new @ V @ transpose(X_new))
return student_t(df=n - p, loc=location, scale=scale)Hyperparameters
The review gives no training recipes for the methods it surveys. The values below are the only numerical settings the paper states.
| Name | Value | Where | Notes |
|---|---|---|---|
| Quantile levels in the example | 0.05, 0.50, 0.95 | fig. 2 tab. 2 | text says "multiple quantile levels" §7 |
| Sample sizes in the example | 100 and 1 000 | §7 tab. 2 | |
| Train / test split | 50 / 50 (n = 100); 500 / 500 (n = 1 000) | §7 | |
| Repetitions of the simulation | 10 000 | §7 | number of random seeds not stated |
| Simulated model | y = x1 + x2 + ε | eq. 33 | intercept not stated |
| Predictors | x1, x2 standard Gaussian | §7 eq. 33 | |
| Noise | ε ~ N(0, 1²) | eq. 33 | |
| Response family for GAMLSS | Gaussian | §7 | fitting software and estimation settings not stated |
| Prior for Bayesian linear model | uniform on (β, log σ) | §4.1 | |
| Predictive degrees of freedom | n − p | §4.1 | eq. 30 writes n − k; see Pitfalls |
| CRPS ensemble levels | (k − 0.5) / K for member k | §3.3.2 | Bröcker 2012, as cited |
| Central interval levels | α/2 and 1 − α/2 | §3.3.4 | |
| Linear pool weights | sum to one when quantile predictions are unbiased; constant omitted | §8.1 | Taylor and Bunn, as cited |
Results
Author-reported. The paper's only numerical result is one simulation, not a benchmark on real data. Setup: a linear model with Gaussian noise eq. 33; quantile regression and a Gaussian GAMLSS fitted on the training half; results averaged over 10 000 repetitions §7.
| Sample size | Metric | Quantile regression | GAMLSS | Source |
|---|---|---|---|---|
| 100 | coverage, α = 0.05 | 0.07921 | 0.07344 | tab. 2 |
| 100 | coverage, α = 0.50 | 0.49943 | 0.49890 | tab. 2 |
| 100 | coverage, α = 0.95 | 0.92112 | 0.92677 | tab. 2 |
| 100 | quantile score, α = 0.05 | 0.12044 | 0.11460 | tab. 2 |
| 100 | quantile score, α = 0.50 | 0.41847 | 0.41317 | tab. 2 |
| 100 | quantile score, α = 0.95 | 0.11988 | 0.11426 | tab. 2 |
| 1 000 | coverage, α = 0.05 | 0.05281 | 0.05211 | tab. 2 |
| 1 000 | coverage, α = 0.50 | 0.50034 | 0.50026 | tab. 2 |
| 1 000 | coverage, α = 0.95 | 0.94728 | 0.94793 | tab. 2 |
| 1 000 | quantile score, α = 0.05 | 0.10456 | 0.10391 | tab. 2 |
| 1 000 | quantile score, α = 0.50 | 0.40095 | 0.40027 | tab. 2 |
| 1 000 | quantile score, α = 0.95 | 0.10455 | 0.10389 | tab. 2 |
The authors say GAMLSS scores better at every level and sample size, and that its advantage shrinks as the sample grows. They attribute this to the example being built on a Gaussian response, which suits the parametric model by construction §7. The paper states that real data may reverse the ranking §7.
Paper vs code
No official code. The paper has no code link in its abstract, introduction, §§1–8, §10, final remarks or references, and the arXiv metadata has none. Searches done: a GitHub repository search for the author names returned no results; a search on the topic phrase returned one unrelated third-party project, which is not used. No discrepancies to check, because no code exists.
Pitfalls
- Printed equation (29) lacks an inverse. It reads V_u = X'X. The usual covariance for this flat-prior linear model has (X'X)^-1, and the predictive scale matrix in §4.1 depends on it. This reading is not stated by the paper: check against the textbook it cites (Gelman et al. 2013, ch. 14) before implementing. The pseudocode above uses the inverse.
- Degrees of freedom are printed two ways. The text gives n − p §4.1; eq. 30 divides by n − k, and k is not defined in the paper.
- Indicator in the CRPS ensemble formula is printed with z_i, not the ordered z_(i) eq. 23. Implement with the sorted members, which the formula assumes; the paper says unordered ensembles are not recommended for computational reasons §3.3.2.
- Coverage is not a proper scoring rule. The paper says so when it reports coverage §7 and that skill scores, which rescale scores for comparison across series, are not necessarily proper §3.5.
- Log score only applies to densities. It is local and needs a predictive density; it cannot score a quantile-only forecast §3.4.
- Interval scoring needs fixed levels. The paper notes the scoring rules for intervals mostly refer to fixed quantile levels, and that intervals with varying levels need new scoring methods §3.3.4 §10.3.
- Independent quantile fits can cross. A higher quantile can come out below a lower one; remedies are sorting, non-crossing constraints or joint estimation of several quantiles §4.2.1 §6.3.
- Parametric models win on their own assumptions. The GAMLSS advantage in §7 comes from a correctly specified response family. Choosing a wrong distribution is a stated risk for GAMLSS and Bayesian models §10.2.
- Scoring choice matters even with proper rules. Different consistent scores for the same functional can rank models differently in finite samples; the paper cites Patton (2020) for the three assumptions under which they agree §10.1.
- Extreme quantiles need large samples for point-prediction methods, and distributional approaches may be preferable with small data §10.2.
- Conformal prediction gives marginal, not conditional, coverage §5.5.3.
- Dropout as Bayesian approximation is the paper's citation of Gal and Ghahramani (2016), not a result it verifies §6.2.
- Simulation details are missing. The paper does not state the quantile regression implementation, the GAMLSS fitting method or software, the definition of the averaged quantile score, or the random seeds §7.
- The review is shallow on many methods. Several families (for example §5.5 and §9) are covered in one or two sentences with a citation. Implement from the original papers, not from this card.
- Card coverage. This card reads §§1–8, §10 and the first part of §9 (§9.1–9.2). §9.3–9.6 (spatial extremes, measurement error, scientific applications) and the reference list are not summarised.
Lineage
- Builds on: the foundations it cites, including proper scoring rules, quantile regression, expectiles, GAMLSS, quantile regression forests and boosting. No cards for these exist in this wiki.
- Compared against: earlier reviews on subtopics of probabilistic prediction, listed in the paper's Table 1 tab. 1. None compiled here.
- Superseded by: not known from content on this wiki.
Source and license
Adapted from “A review of predictive uncertainty estimation with machine learning” by Hristos Tyralis, Georgia Papacharalampous (arXiv:2209.08307), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.