wikillm

Concept · 1 paper · updated 2026-11-12

Regression loss function

Scores a numeric prediction against its target during training. A survey compares 14 such losses (absolute, squared, relative, percentage, log-scale, signed, Huber, log-cosh and quantile) as objectives for one forecasting model. Choose by outliers, scale, bias and whether you need intervals. The evidence is one author-reported survey with single runs, and the best loss differs by dataset.

regressionloss-functionstime-seriesforecastingquantile-regressionrobust-losssurvey

Markdown4 min read

AI-compiled overview, updated 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

A regression model learns by being told how wrong each numeric prediction is. The loss function turns a prediction and its true value into that "how wrong" number, and the choice changes what the model learns. The survey's position is that no universal loss exists and that the choice depends on outliers, skew, model needs, compute and performance requirements 2211.02989 §V. The invariant all variants share is that each one maps a prediction and a target to a score that is averaged or summed over samples 2211.02989 §III.

Variants

The survey defines 14 losses and gives each an advantages and disadvantages list and use cases 2211.02989 §III.A–§III.N. They are grouped here by what they measure. Card: regression loss survey, 2211.02989.

Absolute and squared errors

  • MAE weights all errors equally and is easy to read 2211.02989 §III.A.
  • MSE suits data where large errors are significant anomalies 2211.02989 §III.B.
  • RMSE is the square root of MSE. It is differentiable and cheap, with less penalty on large errors than MSE 2211.02989 §III.G. Caveat: the official code's RMSE model is compiled with MSE, so its RMSE rows are not RMSE-trained (Card: Paper vs code, item 1).

Relative and normalised errors

  • RAE and RSE compare models across different units or scales 2211.02989 §III.D §III.E.
  • NRMSE and RRMSE do the same for models trained on different scales or measurements 2211.02989 §III.J §III.K.
  • Caveat: as implemented in the official code, the RSE loss has the same numerator and denominator, so it is constant and gives no gradient signal. The paper defines the denominator with the mean of the true values 2211.02989 §III.E (Card: Paper vs code, item 2).

Percentage and log scale

  • MAPE expresses error as a percentage of the true value. It is undefined where the true value is zero 2211.02989 §III.F.
  • MSLE and RMSLE penalise under-prediction more than over-prediction, for targets where over-prediction is acceptable 2211.02989 §III.H §III.I.

Signed bias

  • MBE keeps the sign of the error. The paper warns that positive and negative errors cancel, so it is useful only when the net or cumulative bias matters 2211.02989 §III.C.

Robust combinations

Quantile (pinball)

  • The quantile loss targets a chosen quantile γ and is used to predict intervals rather than point estimates 2211.02989 §III.N. The paper's printed formula has a sign arrangement that the official code does not use; the code uses the standard pinball form (Card: Paper vs code, item 7).

When to use which

SituationUseWhy
Errors are equally costlyMAE 2211.02989 §III.AEqual weight per error, easy to read
Large errors are anomalies to avoidMSE, or RMSE 2211.02989 §III.B §III.GPenalises large errors more heavily
Data has outliersHuber or log-cosh 2211.02989 §III.L §III.MLinear scoring above the threshold; §V names outliers as a selection factor
Need a prediction interval or quantileQuantile loss 2211.02989 §III.NTargets a chosen quantile; the paper uses P10, P50 and P90 §IV.B
Models on different scales must be comparedRAE, RSE, NRMSE or RRMSE 2211.02989 §III.D–§III.KRelative or normalised scores
Percentage error matters and no target is zeroMAPE 2211.02989 §III.FPercentage of the true value
Over-prediction is acceptable, under-prediction is notMSLE or RMSLE 2211.02989 §III.H §III.IAsymmetric penalty on log scale
Net bias is the concernMBE 2211.02989 §III.CSigned; cancels errors in the sum

Author-reported results, on four forecasting datasets with a Temporal Fusion Transformer as the model 2211.02989 §IV:

  • Electricity: quantile, MSE and RRMSE "performed well".
  • Traffic: log-cosh, quantile and MAE "performed better".
  • Favorita: RAE, log-cosh and quantile "outperformed other objective functions".
  • Volatility: quantile, MAE and Huber "have performed better".

Read these as one-run comparisons. The paper reports no variance across runs 2211.02989 §IV.B, and the experiments use 1000 training and 100 validation samples, 15 epochs and hidden size 16 2211.02989 tab. 15. Those settings are much smaller than the official code's defaults, so the ranking at full scale is untested (Card: Paper vs code, items 12 and 13).

Implementation sketch

Pseudocode of the losses named above, using the paper's definitions. The Huber and pinball forms are written in their symmetric or standard shape, which differs from the printed formulas.

Pseudocode
# Absolute error  [arxiv:2211.02989 §III.A]
def mae(y, y_hat):
    return mean(abs(y - y_hat))

# Squared error  [arxiv:2211.02989 §III.B]
def mse(y, y_hat):
    return mean((y - y_hat) ** 2)

# Huber, threshold delta  [arxiv:2211.02989 §III.L]
# Symmetric form: the printed linear branch lacks |.|
def huber(y, y_hat, delta):
    e = abs(y - y_hat)
    quad = 0.5 * e ** 2
    lin = delta * (e - 0.5 * delta)
    return mean(where(e < delta, quad, lin))

# Quantile (pinball) for quantile q  [arxiv:2211.02989 §III.N]
# Equals q*e for e >= 0 and (q-1)*e for e < 0, with e = y - y_hat
def pinball(y, y_hat, q):
    e = y - y_hat
    return mean(maximum(q * e, (q - 1) * e))

Contradictions

  • Paper text vs its own formula. The paper says the quantile loss "ranges from 0 to 1" 2211.02989 §III.N. The pinball loss is not bounded by 1 in general, so the claim does not hold for the formula as written.
  • Paper rows vs official code. The official code does not match several of the paper's per-loss rows. RMSE is trained with MSE, the RSE loss is constant as implemented, and the non-quantile models fit all three quantile heads to one target (Card: Paper vs code, items 1, 2 and 6). The paper's tables present these rows as results for each named loss. This page does not resolve the difference.
  • Dataset rankings. The best loss differs by dataset 2211.02989 §IV. Whether any loss is best in general is not established by this one paper.

Papers

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.