Are Transformers Effective for Time Series Forecasting?

In plain words

Transformers became the dominant architecture for language and images, and a wave of papers then used them to predict long stretches of future values in a time series, such as electricity load or road traffic. This paper asks whether that works. Its argument is that attention treats its inputs as an unordered set, so it loses the order of the points unless extra position signals are added, and in a time series the order is most of the information.

To test this, the authors build LTSF-Linear, a single linear layer that reads the recent values and writes out all the future values at once. Two variants add a split into a slow trend and a remainder (DLinear), or shift the input so that it is centred on its last value to handle drifting levels (NLinear). On nine public benchmarks, the best of these simple models beats the published Transformer forecasters in every dataset and horizon the paper tests, often by a large margin.

The takeaway is not that linear models are the answer. It is that the claims for Transformers need stricter checks, and that a simple linear baseline should be the first thing a new forecaster has to beat.

TL;DR

Long-term forecasting (LTSF) is the task of predicting many future steps from a long look-back window. The paper argues that self-attention's permutation invariance discards temporal order, and tests this with LTSF-Linear. Its three linear models, one layer each, are trained with MSE and reach lower MSE than the best published Transformer forecaster in each of the 36 dataset-horizon cells of the main table (author-reported, tab. 2). Each model is a few lines of code, but the comparison uses different look-back lengths for the two families, so a fair reuse needs the same look-back on both sides.

Problem

Long-term forecasting asks for many future steps at once. Earlier baselines were mostly iterated multi-step (IMS) models, which feed each prediction back as an input and so accumulate error over the horizon §1 §2. Transformer-based LTSF models (Informer, Autoformer, FEDformer, Pyraformer, LogTrans and others) were reported to beat those baselines. The paper hypothesises that the gains come largely from the direct multi-step (DMS) strategy these models use §4.

The paper's concern is attention itself. Self-attention is permutation-invariant, and positional encodings or sub-series tokens only partly restore order, which the paper says leaves "inevitable" temporal information loss §1 §3. The paper also measures the efficiency claim behind these models (cutting the O(L^2) cost of attention). Against a vanilla Transformer with the same decoder, most variants are slower and have more parameters §5.3 tab. 8. The evidence is limited to the nine benchmarks in §5.1 and app. B.1; the authors call for revisiting Transformers on other time series tasks, such as anomaly detection §1.

The authors present LTSF-Linear as a baseline for future work, not as the final model §6.

Method

The paper does not propose a new Transformer. It defines a baseline and two variants, all one-layer linear networks that map a look-back window to a forecast window, and treats them as a reference for LTSF-Transformers §4.

1. Linear. For each variate i, the forecast is \hat{X}_i = W X_i, with W \in \mathbb{R}^{T \times L} a single linear layer along the time axis. The same W is shared across all variates, so no cross-variate correlation is modelled §4. The paper's formula has no bias term; the code has one (see Paper vs code).
2. DLinear. The input is split into a trend, a moving average with kernel size 25, and a remainder (seasonal) part §4 app. B.2. Two independent linear layers map each part to the horizon, and the two outputs are summed §4. The paper says the decomposition is borrowed from Autoformer and FEDformer §3 §4.
3. NLinear. The last observed value of the input is subtracted from the whole window, the linear layer runs, and the value is added back to the output §4. The stated purpose is to reduce distribution shift between training and test periods §4 app. D.1.

Training uses MSE on the forecast window exp_main.py. The hyperparameters are in the next section. The paper's formulas are inline and unnumbered, so they are cited by section.

# LTSF-Linear  [§4] [code: models/Linear.py#L24]
# Input:  x (B, L, C)   Output: y_hat (B, T, C)

def ltsf_linear(x, weight, bias):
    # Move time last, so one layer maps L past steps to T steps  [§4]
    series = transpose(x, (0, 2, 1))              # (B, C, L)
    # One (T, L) weight shared by all C variates  [§4]
    # The paper's formula has no bias; the code adds one
    # [code: models/Linear.py#L24]
    y_hat = series @ weight.T + bias              # (B, C, T)
    return transpose(y_hat, (0, 2, 1))            # (B, T, C)

# DLinear  [§4] [code: models/DLinear.py#L17-L18, L48, L86]
# Input:  x (B, L, C)   Output: y_hat (B, T, C)

def dlinear(x, seasonal_layer, trend_layer):
    # Trend: moving average, kernel 25; first and last values repeated at
    # both ends so the length stays L  [§4] [app. B.2]
    trend = moving_average(x, kernel=25)          # (B, L, C)
    # Remainder: what the moving average leaves behind  [§3] [§4]
    seasonal = x - trend                          # (B, L, C)
    # One linear layer per component, then sum the two forecasts  [§4]
    seasonal_hat = seasonal_layer(transpose(seasonal, (0, 2, 1)))
    trend_hat = trend_layer(transpose(trend, (0, 2, 1)))
    return transpose(seasonal_hat + trend_hat, (0, 2, 1))   # (B, T, C)

# NLinear  [§4] [code: models/NLinear.py#L28-L37]
# Input:  x (B, L, C)   Output: y_hat (B, T, C)

def nlinear(x, linear):
    # Anchor on the last observed value of each variate  [§4]
    # The code detaches the anchor, so no gradient flows through it
    # [code: models/NLinear.py#L28]
    last = stop_gradient(x[:, -1:, :])            # (B, 1, C)
    # Forecast the offsets from the anchor, then add the anchor back  [§4]
    y_hat = linear(x - last) + last               # (B, T, C)
    return y_hat

Hyperparameters

The paper states the look-back length, the decomposition kernel and the dataset splits. Learning rate, batch size, epochs, patience and the optimiser are not in the paper text; it says "For more hyper-parameters of LTSF-Linear, please refer to our code" app. B.2. The values below come from the released code at code_commit.

Per-dataset run settings, from the scripts in scripts/EXP-LongForecasting/Linear/:

Global settings, with the paper's stated value where there is one:

Datasets, from tab. 1 (the paper lists ETT at "5min" granularity, which the appendix and code contradict; see Paper vs code):

Results

Author-reported. Setup: nine benchmarks, MSE and MAE, horizons T in {96, 192, 336, 720} (ILI: {24, 36, 48, 60}) §5.1 tab. 2. The Linear family uses L=336 and the Transformers L=96 (ILI: 104 in the code) app. B.2. The table note says its Transformer numbers, except Pyraformer, are copied from the FEDformer paper, not re-run. Pyraformer and the three linear models are the authors' runs tab. 2. Each linear setting is one run in the scripts (see Hyperparameters).

Main comparison, MSE / MAE, from tab. 2:

Remaining baselines, MSE / MAE, same source tab. 2:

Counts from these tables: the best linear model has lower MSE than the best of the Transformer baselines (FEDformer, Autoformer, Informer, Pyraformer, LogTrans) in all 36 cells. On Exchange-Rate, the Repeat baseline (last value) ties DLinear at T=96 on MSE (0.081) and beats it on MAE (0.196 vs 0.203) tab. 2.

Other results, author-reported:

• Look-back and temporal relations §5.3 fig. 4 app. C.2: as L grows, the linear models improve, while the Transformer baselines stay flat or get worse (figure only; the text gives no numbers).
• Close vs far input tab. 3: MSE with the same 720-step target, from the 96 steps just before the horizon ("Close") or 96 steps earlier ("Far").

• Informer simplified to one linear layer tab. 4 (look-back 96, per its caption): MSE for Informer, with attention replaced by linear (Att.-Linear), with other blocks dropped (Embed + Linear), and as a single Linear layer.

• Input shuffling tab. 5: the paper reports the average drop in MSE after shuffling the input order (Shuf.) or swapping its halves (Half-Ex.), averaged over horizons and five runs. Per-horizon values are in tab. 5.

• Embeddings tab. 6: removing positional or timestamp embeddings from Informer, FEDformer and Autoformer on Traffic (values in tab. 6).
• Shortened training data tab. 7, Traffic, MSE (full vs one year of training data):

• Cost tab. 8: Electricity, L=96, T=720, averaged over five runs.

The univariate ETT results app. C.1 show NLinear winning most settings; the table itself is in the appendix and is not copied here.

Paper vs code

Discrepancies found between the paper and the official code at code_commit:

• Bias term. The paper's formula is \hat{X}_i = W X_i with no bias §4, and the appendix counts parameters as T \cdot L per linear layer app. B.2. The code builds nn.Linear layers, which include a bias [code: models/Linear.py#L24] DLinear.py. Table 8's DLinear count of 139.7K matches the biased count: 2·96·720 + 2·720 = 139,680, while the appendix's 2·T·L gives 138,240 tab. 8 app. B.2. (This arithmetic is mine, from the code and the table.)
• Look-back for ILI. The paper gives L=336 for LTSF-Linear by default app. B.2. The ILI script sets seq_len=104 ili.sh.
• ETTm granularity. tab. 1 lists ETTm at 5-minute granularity. The appendix says the ETTm data is at 15-minute level app. B.1, the loader splits it into blocks of 30-day months ×4 data_loader.py and bins minutes by 15 data_loader.py. The 69,680 steps in tab. 1 is also 4 × 17,420, consistent with 15 minutes. The 5-minute value looks like a typo.
• Per-variate variant. The paper describes weights shared across variates §4. The code also has a per-channel --individual option for DLinear and the linear model run_longExp.py DLinear.py, run from scripts/EXP-LongForecasting/Linear-I.sh with different learning rates Linear-I.sh. The paper text does not mention it, and the per-dataset scripts in Linear/ do not set it.
• Error scale. The paper does not say what scale its errors are on. The code computes MSE and MAE on standardised predictions: test() never inverts the scaler exp_main.py before calling the metric exp_main.py. These are not the original units.
• Repeat baseline and other baselines. The table note says the Transformer numbers are from FEDformer tab. 2; the code for those baselines is not in this repository, so the paper's Transformer numbers were not reproduced here.
• Consistent items. Decomposition kernel 25 app. B.2 DLinear.py and the NLinear subtract-then-add structure §4 NLinear.py match.

Pitfalls

• Unequal look-back. The Linear family uses L=336 and the Transformers L=96 app. B.2. The main table therefore compares models at different input lengths. The look-back study §5.3 fig. 4 is a figure with no numbers in the text, so the paper's table does not show Transformers at L=336. For an architecture-only comparison, run both at the same L.
• "In all cases" needs qualifying. The abstract says LTSF-Linear beats the Transformers "in all cases" [abstract]. Table 2 supports this for the best linear model per cell, but not for every linear model: plain Linear has MSE 0.698 vs FEDformer's 0.463 at ETTh2, T=720, and DLinear 0.605 tab. 2.
• Single run, no variance. Each linear setting runs once with seed 2021 run_longExp.py etth1.sh. Table 2 gives no run count or spread. Table 5 and Table 8 say five runs. Small gaps between the linear models cannot be read as significant without spread: ETTh1 at T=96 gives 0.375, 0.374 and 0.375 MSE for Linear, NLinear and DLinear tab. 2.
• Hyperparameters are per dataset. The learning rate varies from 0.0001 to 0.05 across datasets and changes with horizon for ETTm2 [code: scripts/EXP-LongForecasting/Linear/ettm2.sh#L24-L52]. Batch size changes too. None of these is in the paper.
• Learning-rate schedule. type1 multiplies the learning rate by 0.5 per epoch, with the first halving before the third epoch tools.py exp_main.py. The schedule is called with epoch + 1 after each epoch, so the first two epochs use the base rate.
• Early stopping. Selection uses validation MSE with patience 3, up to 10 epochs exp_main.py. The test loss is printed each epoch but not used for selection exp_main.py. Keep it that way when you reproduce the numbers.
• Scaling. Fit the scaler on the training split only data_loader.py and report errors on the standardised scale, as the code does (see Paper vs code).
• Batch details. Training and validation both use the shuffled loader that drops the last incomplete batch data_factory.py. The validation MSE is a mean of batch means over that loader. Test does not shuffle or drop data_factory.py.
• NLinear gradient. The last-value anchor is detached, so the gradient ignores its effect on the offset NLinear.py. The paper does not say this.
• DLinear edge padding. The moving average repeats the first and last values (kernel 25) to keep the length at L DLinear.py. The paper does not state the padding.
• Table 4 is not Table 2. Table 4's Linear column uses L=96 tab. 4, so its values differ from Table 2's L=336 values (e.g. ETTh1, T=720: 0.515 vs 0.624).
• IMP. column. The "IMP." column in Table 2 is described only in its caption. It does not match a plain MSE ratio: for ETTh1, T=96 the best linear and best Transformer MSEs (0.374 vs 0.376) give about 0.5%, not 0.80%. It is not reproduced here.
• Costs are measured, not general. The timing and memory figures are for one GPU and one setting (Electricity, L=96, T=720) tab. 8. Parameter counts include the bias terms (see Paper vs code).
• Interpretation is a single view. The paper reads the learned weights on Traffic as daily and weekly periodicity app. D.3. That reading comes from the weight plots of one model; the paper does not test it on other models.

Lineage

• Builds on: decomposition with a moving average from Autoformer and FEDformer §3 §4; direct multi-step output from Informer §3.
• Compared against: FEDformer, Autoformer, Informer, Pyraformer, LogTrans, and the Repeat baseline §5.1.
• Superseded by: not recorded in this wiki.