Concept · 1 paper · updated 2026-11-12
Regression loss function
Scores a numeric prediction against its target during training. A survey compares 14 such losses (absolute, squared, relative, percentage, log-scale, signed, Huber, log-cosh and quantile) as objectives for one forecasting model. Choose by outliers, scale, bias and whether you need intervals. The evidence is one author-reported survey with single runs, and the best loss differs by dataset.
regressionloss-functionstime-seriesforecastingquantile-regressionrobust-losssurvey
Markdown4 min read
AI-compiled overview, updated 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
A regression model learns by being told how wrong each numeric prediction is. The loss function turns a prediction and its true value into that "how wrong" number, and the choice changes what the model learns. The survey's position is that no universal loss exists and that the choice depends on outliers, skew, model needs, compute and performance requirements 2211.02989 §V. The invariant all variants share is that each one maps a prediction and a target to a score that is averaged or summed over samples 2211.02989 §III.
Variants
The survey defines 14 losses and gives each an advantages and disadvantages list and use cases 2211.02989 §III.A–§III.N. They are grouped here by what they measure. Card: regression loss survey, 2211.02989.
Absolute and squared errors
- MAE weights all errors equally and is easy to read 2211.02989 §III.A.
- MSE suits data where large errors are significant anomalies 2211.02989 §III.B.
- RMSE is the square root of MSE. It is differentiable and cheap, with less penalty on large errors than MSE 2211.02989 §III.G. Caveat: the official code's RMSE model is compiled with MSE, so its RMSE rows are not RMSE-trained (Card: Paper vs code, item 1).
Relative and normalised errors
- RAE and RSE compare models across different units or scales 2211.02989 §III.D §III.E.
- NRMSE and RRMSE do the same for models trained on different scales or measurements 2211.02989 §III.J §III.K.
- Caveat: as implemented in the official code, the RSE loss has the same numerator and denominator, so it is constant and gives no gradient signal. The paper defines the denominator with the mean of the true values 2211.02989 §III.E (Card: Paper vs code, item 2).
Percentage and log scale
- MAPE expresses error as a percentage of the true value. It is undefined where the true value is zero 2211.02989 §III.F.
- MSLE and RMSLE penalise under-prediction more than over-prediction, for targets where over-prediction is acceptable 2211.02989 §III.H §III.I.
Signed bias
- MBE keeps the sign of the error. The paper warns that positive and negative errors cancel, so it is useful only when the net or cumulative bias matters 2211.02989 §III.C.
Robust combinations
- Huber, with a threshold δ, is quadratic below δ and linear above it 2211.02989 §III.L. The experiments use δ = 0.5 2211.02989 §IV.
- Log-cosh also combines quadratic and linear scoring and is twice differentiable everywhere 2211.02989 §III.M.
Quantile (pinball)
- The quantile loss targets a chosen quantile γ and is used to predict intervals rather than point estimates 2211.02989 §III.N. The paper's printed formula has a sign arrangement that the official code does not use; the code uses the standard pinball form (Card: Paper vs code, item 7).
When to use which
| Situation | Use | Why |
|---|---|---|
| Errors are equally costly | MAE 2211.02989 §III.A | Equal weight per error, easy to read |
| Large errors are anomalies to avoid | MSE, or RMSE 2211.02989 §III.B §III.G | Penalises large errors more heavily |
| Data has outliers | Huber or log-cosh 2211.02989 §III.L §III.M | Linear scoring above the threshold; §V names outliers as a selection factor |
| Need a prediction interval or quantile | Quantile loss 2211.02989 §III.N | Targets a chosen quantile; the paper uses P10, P50 and P90 §IV.B |
| Models on different scales must be compared | RAE, RSE, NRMSE or RRMSE 2211.02989 §III.D–§III.K | Relative or normalised scores |
| Percentage error matters and no target is zero | MAPE 2211.02989 §III.F | Percentage of the true value |
| Over-prediction is acceptable, under-prediction is not | MSLE or RMSLE 2211.02989 §III.H §III.I | Asymmetric penalty on log scale |
| Net bias is the concern | MBE 2211.02989 §III.C | Signed; cancels errors in the sum |
Author-reported results, on four forecasting datasets with a Temporal Fusion Transformer as the model 2211.02989 §IV:
- Electricity: quantile, MSE and RRMSE "performed well".
- Traffic: log-cosh, quantile and MAE "performed better".
- Favorita: RAE, log-cosh and quantile "outperformed other objective functions".
- Volatility: quantile, MAE and Huber "have performed better".
Read these as one-run comparisons. The paper reports no variance across runs 2211.02989 §IV.B, and the experiments use 1000 training and 100 validation samples, 15 epochs and hidden size 16 2211.02989 tab. 15. Those settings are much smaller than the official code's defaults, so the ranking at full scale is untested (Card: Paper vs code, items 12 and 13).
Implementation sketch
Pseudocode of the losses named above, using the paper's definitions. The Huber and pinball forms are written in their symmetric or standard shape, which differs from the printed formulas.
# Absolute error [arxiv:2211.02989 §III.A]
def mae(y, y_hat):
return mean(abs(y - y_hat))
# Squared error [arxiv:2211.02989 §III.B]
def mse(y, y_hat):
return mean((y - y_hat) ** 2)
# Huber, threshold delta [arxiv:2211.02989 §III.L]
# Symmetric form: the printed linear branch lacks |.|
def huber(y, y_hat, delta):
e = abs(y - y_hat)
quad = 0.5 * e ** 2
lin = delta * (e - 0.5 * delta)
return mean(where(e < delta, quad, lin))
# Quantile (pinball) for quantile q [arxiv:2211.02989 §III.N]
# Equals q*e for e >= 0 and (q-1)*e for e < 0, with e = y - y_hat
def pinball(y, y_hat, q):
e = y - y_hat
return mean(maximum(q * e, (q - 1) * e))Contradictions
- Paper text vs its own formula. The paper says the quantile loss "ranges from 0 to 1" 2211.02989 §III.N. The pinball loss is not bounded by 1 in general, so the claim does not hold for the formula as written.
- Paper rows vs official code. The official code does not match several of the paper's per-loss rows. RMSE is trained with MSE, the RSE loss is constant as implemented, and the non-quantile models fit all three quantile heads to one target (Card: Paper vs code, items 1, 2 and 6). The paper's tables present these rows as results for each named loss. This page does not resolve the difference.
- Dataset rankings. The best loss differs by dataset 2211.02989 §IV. Whether any loss is best in general is not established by this one paper.
Papers
- Regression loss survey, 2211.02989: compares 14 regression losses as training objectives for one forecasting model on four datasets.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.