---
kind: concept
slug: regression-loss-function
title: Regression loss function
summary: >-
  Scores a numeric prediction against its target during training. A survey compares 14 such losses (absolute, squared, relative, percentage, log-scale, signed, Huber, log-cosh and quantile) as objectives for one forecasting model. Choose by outliers, scale, bias and whether you need intervals. The evidence is one author-reported survey with single runs, and the best loss differs by dataset.
tags: [regression, loss-functions, time-series, forecasting, quantile-regression, robust-loss, survey]
papers: ["2211.02989"]
related: [time-series-forecasting, quantile-regression-forest, uncertainty-estimation, bounding-box-regression-loss, cross-entropy-loss, adaptive-loss-weighting]
status: draft
updated_at: 2026-11-12
---

## Overview

A regression model learns by being told how wrong each numeric prediction is.
The loss function turns a prediction and its true value into that "how wrong"
number, and the choice changes what the model learns. The survey's position is
that no universal loss exists and that the choice depends on outliers, skew,
model needs, compute and performance requirements [arxiv:2211.02989 §V]. The
invariant all variants share is that each one maps a prediction and a target
to a score that is averaged or summed over samples [arxiv:2211.02989 §III].

## Variants

The survey defines 14 losses and gives each an advantages and disadvantages
list and use cases [arxiv:2211.02989 §III.A]–[§III.N]. They are grouped here by
what they measure. Card: [regression loss survey, 2211.02989](/papers/2211.02989).

### Absolute and squared errors

- **MAE** weights all errors equally and is easy to read [arxiv:2211.02989 §III.A].
- **MSE** suits data where large errors are significant anomalies [arxiv:2211.02989 §III.B].
- **RMSE** is the square root of MSE. It is differentiable and cheap, with less penalty on large errors than MSE [arxiv:2211.02989 §III.G]. Caveat: the official code's RMSE model is compiled with MSE, so its RMSE rows are not RMSE-trained ([Card: Paper vs code](/papers/2211.02989), item 1).

### Relative and normalised errors

- **RAE** and **RSE** compare models across different units or scales [arxiv:2211.02989 §III.D] [§III.E].
- **NRMSE** and **RRMSE** do the same for models trained on different scales or measurements [arxiv:2211.02989 §III.J] [§III.K].
- Caveat: as implemented in the official code, the RSE loss has the same numerator and denominator, so it is constant and gives no gradient signal. The paper defines the denominator with the mean of the true values [arxiv:2211.02989 §III.E] ([Card: Paper vs code](/papers/2211.02989), item 2).

### Percentage and log scale

- **MAPE** expresses error as a percentage of the true value. It is undefined where the true value is zero [arxiv:2211.02989 §III.F].
- **MSLE** and **RMSLE** penalise under-prediction more than over-prediction, for targets where over-prediction is acceptable [arxiv:2211.02989 §III.H] [§III.I].

### Signed bias

- **MBE** keeps the sign of the error. The paper warns that positive and negative errors cancel, so it is useful only when the net or cumulative bias matters [arxiv:2211.02989 §III.C].

### Robust combinations

- **Huber**, with a threshold δ, is quadratic below δ and linear above it [arxiv:2211.02989 §III.L]. The experiments use δ = 0.5 [arxiv:2211.02989 §IV].
- **Log-cosh** also combines quadratic and linear scoring and is twice differentiable everywhere [arxiv:2211.02989 §III.M].

### Quantile (pinball)

- The quantile loss targets a chosen quantile γ and is used to predict intervals rather than point estimates [arxiv:2211.02989 §III.N]. The paper's printed formula has a sign arrangement that the official code does not use; the code uses the standard pinball form ([Card: Paper vs code](/papers/2211.02989), item 7).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Errors are equally costly | MAE [arxiv:2211.02989 §III.A] | Equal weight per error, easy to read |
| Large errors are anomalies to avoid | MSE, or RMSE [arxiv:2211.02989 §III.B] [§III.G] | Penalises large errors more heavily |
| Data has outliers | Huber or log-cosh [arxiv:2211.02989 §III.L] [§III.M] | Linear scoring above the threshold; §V names outliers as a selection factor |
| Need a prediction interval or quantile | Quantile loss [arxiv:2211.02989 §III.N] | Targets a chosen quantile; the paper uses P10, P50 and P90 [§IV.B] |
| Models on different scales must be compared | RAE, RSE, NRMSE or RRMSE [arxiv:2211.02989 §III.D]–[§III.K] | Relative or normalised scores |
| Percentage error matters and no target is zero | MAPE [arxiv:2211.02989 §III.F] | Percentage of the true value |
| Over-prediction is acceptable, under-prediction is not | MSLE or RMSLE [arxiv:2211.02989 §III.H] [§III.I] | Asymmetric penalty on log scale |
| Net bias is the concern | MBE [arxiv:2211.02989 §III.C] | Signed; cancels errors in the sum |

Author-reported results, on four forecasting datasets with a Temporal Fusion
Transformer as the model [arxiv:2211.02989 §IV]:

- Electricity: quantile, MSE and RRMSE "performed well".
- Traffic: log-cosh, quantile and MAE "performed better".
- Favorita: RAE, log-cosh and quantile "outperformed other objective functions".
- Volatility: quantile, MAE and Huber "have performed better".

Read these as one-run comparisons. The paper reports no variance across runs
[arxiv:2211.02989 §IV.B], and the experiments use 1000 training and 100
validation samples, 15 epochs and hidden size 16 [arxiv:2211.02989 tab. 15].
Those settings are much smaller than the official code's defaults, so the
ranking at full scale is untested ([Card: Paper vs code](/papers/2211.02989),
items 12 and 13).

## Implementation sketch

Pseudocode of the losses named above, using the paper's definitions. The
Huber and pinball forms are written in their symmetric or standard shape,
which differs from the printed formulas.

```pseudocode
# Absolute error  [arxiv:2211.02989 §III.A]
def mae(y, y_hat):
    return mean(abs(y - y_hat))

# Squared error  [arxiv:2211.02989 §III.B]
def mse(y, y_hat):
    return mean((y - y_hat) ** 2)

# Huber, threshold delta  [arxiv:2211.02989 §III.L]
# Symmetric form: the printed linear branch lacks |.|
def huber(y, y_hat, delta):
    e = abs(y - y_hat)
    quad = 0.5 * e ** 2
    lin = delta * (e - 0.5 * delta)
    return mean(where(e < delta, quad, lin))

# Quantile (pinball) for quantile q  [arxiv:2211.02989 §III.N]
# Equals q*e for e >= 0 and (q-1)*e for e < 0, with e = y - y_hat
def pinball(y, y_hat, q):
    e = y - y_hat
    return mean(maximum(q * e, (q - 1) * e))
```

## Contradictions

- **Paper text vs its own formula.** The paper says the quantile loss "ranges from 0 to 1" [arxiv:2211.02989 §III.N]. The pinball loss is not bounded by 1 in general, so the claim does not hold for the formula as written.
- **Paper rows vs official code.** The official code does not match several of the paper's per-loss rows. RMSE is trained with MSE, the RSE loss is constant as implemented, and the non-quantile models fit all three quantile heads to one target ([Card: Paper vs code](/papers/2211.02989), items 1, 2 and 6). The paper's tables present these rows as results for each named loss. This page does not resolve the difference.
- **Dataset rankings.** The best loss differs by dataset [arxiv:2211.02989 §IV]. Whether any loss is best in general is not established by this one paper.

## Papers

- [Regression loss survey, 2211.02989](/papers/2211.02989): compares 14 regression losses as training objectives for one forecasting model on four datasets.
