Frequency-domain MLPs are More Effective Learners in Time Series Forecasting

In plain words

Forecasting means predicting the next stretch of a series, such as power use, traffic counts or exchange rates, from its recent past. Small multi-layer perceptrons (MLPs) are a cheap way to do this, but they look at the series one point at a time, and they can get lost in the noise between neighbouring points.

FreTS takes the input series and rewrites it as a spectrum, the same kind of view you get from a music equaliser. In that view, each component describes the whole series at once, and the important patterns (the repeating daily or weekly rhythms) sit in a few large components. The model runs small MLPs on those components: one pass mixes information across the series, and another mixes it across time. The result is turned back into a time series and passed to a simple head that outputs the forecast in a single step.

The authors report that this beats several recent forecasting models on most of the benchmarks they ran, while staying simple. The numbers are their own.

TL;DR

• Learns forecasts with two frequency-domain MLPs. One mixes channels and one mixes time steps, applied after a discrete Fourier transform §3.1 eq. 1 eq. 3 eq. 4.
• Each frequency-domain MLP is a complex-valued linear layer, written as real and imaginary parts §3.2 eq. 6 eq. 7.
• Author-reported: MAE 0.120 on Solar and 0.011 on Traffic at horizon 12 tab. 1. On long-term forecasting it is best on most rows of tab. 2.
• The released code is official, Apache-2.0, and covers the long-term experiments only. Several details differ from the paper, notably the frequency MLP (diagonal weights only) and the channel learner, which the code never runs. See Paper vs code.

Problem

Multivariate forecasting asks for the next τ values of N series given a lookback window of length L §3. The authors argue that point-wise MLPs capture the temporal mapping poorly, cannot model global dependencies, and suffer an information bottleneck on volatile local patterns §1. Transformer and graph models capture more structure but cost more §1 §4.3.

The assumptions behind the method are that the series is regularly sampled, the DFT is a good basis for its periodic and trend structure §3.1, and that most of the signal energy sits in a few frequency components §3.2. Baselines and data are described in §4 and app. B.1–app. B.2.

Method

Overview. The model has three stages §3.1 fig. 2:

1. Dimension extension. Each input value is multiplied by a learnable vector, so each series element becomes a d-dimensional vector. The authors compare this to word embeddings §3.1.
2. Frequency learning. A frequency channel learner runs over the N axis for every timestamp, and a frequency temporal learner runs over the L axis for every channel. Both use a discrete Fourier transform, a frequency-domain MLP, and an inverse transform §3.1 eq. 1 eq. 2 eq. 3 eq. 4.
3. Projection. A two-layer feed-forward network maps the flattened learned representation to τ outputs in one step, avoiding autoregressive error accumulation §3.1 eq. 5.

Domain conversion. The DFT of a signal H is written as the integral of H(v) times e^(−j2πfv) eq. 1. In the discrete setting, the code uses a real FFT, which gives L//2+1 bins for a length-L signal FreTS.py FreTS.py. The paper writes L/2 complex components §3.1. The inverse transform is given by eq. 2. Real and imaginary parts of each component are kept separately §3.1.

Frequency channel learner. For each timestamp l, the learner takes the N-by-d matrix H(:,l) and computes a DFT along the channel axis. It applies one frequency-domain MLP with weights W_chan and biases B_chan, then inverts the transform §3.1 eq. 3. Weights are shared across the L timestamps §3.1.

Frequency temporal learner. For each channel n, the learner computes a DFT along the time axis, applies a frequency-domain MLP with weights W_temp and biases B_temp, then inverts the transform §3.1 eq. 4. Weights are shared across the N channels §3.1.

Frequency-domain MLP. For a complex input H of shape m-by-d and a complex weight matrix W of shape d-by-d with bias B, one layer is σ(HW + B) with σ an activation §3.2 eq. 6. Expanding the complex product gives the real and imaginary outputs eq. 7. The activation σ is not named in the paper. The authors implement the real and imaginary parts as separate real-valued computations and stack them back into a complex tensor §3.2. Each learner uses one such layer by default app. B.3.

Projection. The output is σ(S_t φ1 + b1) φ2 + b2, where S_t is the temporal learner output, φ1 has shape (L·d)-by-d_h, φ2 has shape d_h-by-τ, and d_h is the hidden width §3.1 eq. 5. The activation σ is not named.

Theoretical claims. The paper states two results. (i) Parseval-type energy equality: the energy of a series in time equals its energy in frequency §3.2 eq. 8. The authors use it to argue that energy concentrates in few components §3.2 fig. 1. (ii) A frequency-domain MLP with complex weights equals a global circular convolution in the time domain §3.2 eq. 9. Both proofs are in app. D.1 and app. D.2.

Pseudocode for the whole model (the [code: …] tags show where the released implementation lives, see Paper vs code):

# FreTS forward pass  [§3.1] [fig. 2] [eq. 5]
# [code: models/FreTS.py#L88-L101]
# Input: series (B, L, N); output: forecast (B, tau, N)

def frets(series):
    # Dimension extension: lift each value to d dimensions  [§3.1]
    # [code: models/FreTS.py#L34-L40]
    lifted = series_to_hidden(series)            # (B, N, L, d)
    # Frequency channel learner, runs along the N axis  [§3.1] [eq. 3]
    hidden = frequency_channel_learner(lifted)   # (B, N, L, d)
    # Frequency temporal learner, runs along the L axis  [§3.1] [eq. 4]
    hidden = frequency_temporal_learner(hidden)  # (B, N, L, d)
    # The released code adds the lifted input back here; eq. 5 has no
    # such term  [code: models/FreTS.py#L93] [code: models/FreTS.py#L99]
    hidden = hidden + lifted                     # (B, N, L, d)
    # Two-layer feed-forward head maps the flattened window to tau steps
    # [eq. 5]
    forecast = feed_forward(flatten(hidden))     # (B, N, tau)
    return transpose(forecast)                   # (B, tau, N)

# Frequency channel learner  [§3.1] [eq. 3]
# [code: models/FreTS.py#L51-L60]
# Input and output: hidden (B, N, L, d)

def frequency_channel_learner(hidden):
    # Real DFT along the channel axis, one signal per timestamp  [eq. 1]
    spectrum = rfft(move_axis(hidden, to=channel))  # (B, L, N//2+1, d)
    # One frequency-domain MLP, weights shared over the L timestamps  [§3.2]
    spectrum = frequency_mlp(spectrum, channel_weights, channel_biases)
    # Inverse DFT back to N channels, then restore the axis order  [eq. 2]
    return move_axis(irfft(spectrum, n=N), to=time)  # (B, N, L, d)

# Frequency temporal learner  [§3.1] [eq. 4]
# [code: models/FreTS.py#L43-L48]
# Input and output: hidden (B, N, L, d)

def frequency_temporal_learner(hidden):
    # Real DFT along the time axis, one signal per channel  [eq. 1]
    spectrum = rfft(hidden, axis=time)           # (B, N, L//2+1, d)
    # One frequency-domain MLP, weights shared over the N channels  [§3.1]
    spectrum = frequency_mlp(spectrum, temporal_weights, temporal_biases)
    # Inverse DFT back to L steps  [eq. 2]
    return irfft(spectrum, n=L, axis=time)       # (B, N, L, d)

# One frequency-domain MLP layer  [§3.2] [eq. 6] [eq. 7]
# [code: models/FreTS.py#L65-L86]
# Input: complex spectrum (B, M, F, d); weights and biases are complex

def frequency_mlp(spectrum, weights, biases):
    # Split the complex input into real and imaginary parts  [eq. 7]
    x_real = real(spectrum)
    x_imag = imag(spectrum)
    # Real part of the complex product plus bias  [eq. 7]
    out_real = x_real @ real(weights) - x_imag @ imag(weights)
    out_real = out_real + real(biases)
    # Imaginary part of the complex product plus bias  [eq. 7]
    out_imag = x_real @ imag(weights) + x_imag @ real(weights)
    out_imag = out_imag + imag(biases)
    # Activation sigma is not named in the paper; the code uses ReLU
    # on each part, then a soft-shrink  [code: models/FreTS.py#L71-L84]
    out_real = activation(out_real)
    out_imag = activation(out_imag)
    # Stack the two parts back into one complex tensor  [§3.2]
    return complex(out_real, out_imag)           # (B, M, F, d)

Hyperparameters

Values in quotes are as written in the paper. "Not stated" means the paper does not give the value. Code defaults are listed where the code provides them, with the caveat that the released code covers the long-term experiments only.

Results

Author-reported. Setup: thirteen benchmarks, as described in §4 and app. B.1. Table 1 covers six short-term datasets with horizon 12 and thirteen baselines. Table 2 covers six long-term datasets. The paper does not state random seeds and reports no variance. The text says the paper uses seven short-term benchmarks (the seventh is METR-LA, in app. F.1).

Short-term, horizon 12 tab. 1. Columns are MAE and RMSE per dataset.

Source: tab. 1. The paper reports an average gain of 9.4% in MAE and 11.6% in RMSE over the baselines §4.1. This average is not recomputed here.

Long-term tab. 2. Cells are MAE / RMSE.

Source: tab. 2. Lookback 96 except Traffic (48 per §4.1). The paper says FreTS "outperforms all baselines on all datasets" §4.1. Table 2 does not support that wording for Traffic at horizons 48 and 96, or for Electricity at horizon 336 (see Pitfalls). The paper reports an average MAE and RMSE improvement of more than 20% over the best Transformer-based model §4.1.

Channel and temporal learner ablation tab. 3. Cells are MAE / RMSE. FreCL removes the temporal learner and FreTL removes the channel learner. Note that the released code cannot run the channel learner (see Paper vs code).

Source: tab. 3 §4.2.

Frequency MLP as a drop-in for DLinear and NLinear tab. 4. The paper replaces the linear layers in DLinear and NLinear with the frequency-domain MLP. Lookback 96. Cells are MAE / RMSE.

Source: tab. 4 §4.2.

Multi-step short-term on METR-LA tab. 8 app. F.1. Lookback 12, horizons 3, 6, 9 and 12. Cells are MAE / RMSE. Some baselines use pre-defined graphs (STGCN, DCRNN, TAMP-S2GCNets, not in this table).

Source: tab. 8. The text says FreTS beats all baselines at all steps app. F.1. In tab. 8, FreTS has the lowest value in every cell; the closest baseline is StemGNN (MAE 0.052 at h=3 against 0.050 for FreTS).

Multi-step short-term on COVID-19 tab. 9 app. F.1. Lookback 12. Cells are MAE / RMSE.

Source: tab. 9. In tab. 9, FreTS has the lowest value in every cell; the closest baseline at h=3 is GraphWaveNet (MAE 0.092 against 0.071 for FreTS).

Efficiency §4.3 fig. 4. The paper states complexity O(N log N + L log L). On Wiki (N from 1000 to 5000, horizon 12), it reports the number of parameters does not depend on N, and against AGCRN, about 30% fewer parameters and about 20% less training time §4.3. On Exchange (horizon 96 to 480), it reports at least 3× fewer parameters than FEDformer, Autoformer and Informer, and training 3× faster than Informer, 5× faster than Autoformer and more than 10× faster than FEDformer §4.3. These are author-reported.

Parameter sensitivity app. E.3 fig. 6. On ECG with horizon 12, the paper sweeps input length over {6, 12, ..., 60} and embedding size over {32, 64, 128, 256, 512}. It reports that accuracy first rises and then falls with input length, and with embedding size app. E.3. The per-value numbers are only in the figure.

Learned weights §4.4 fig. 5 app. E.2 app. B.4. Visualisations of the real and imaginary weight matrices on Traffic show clear diagonal patterns §4.4 fig. 5. The paper claims the real part matters more than the imaginary part app. E.2. The ablation table for real and imaginary inputs is tab. 7. This table is not transcribed here.

Paper vs code

The official code is at code and code_commit (repository: aikunyi/FreTS). Its LICENSE is Apache-2.0. Its header line reads "Copyright 2022 DLinear Authors", so the code appears to be derived from the DLinear/LTSF-Linear codebase. The repository only contains the long-term experiments run_longExp.py. The short-term experiments, including Solar, Wiki, METR-LA, ECG and COVID-19 in tab. 1, tab. 8 and tab. 9, have no script in the repository.

Differences between the paper and the code at code_commit:

• Frequency MLP weights are used diagonally. The paper writes the layer as a complex d-by-d matrix product §3.2 eq. 6 eq. 7. The code computes einsum('bijd,dd->bijd', x, r), which multiplies each feature by the diagonal entry r[d, d] and drops the off-diagonal entries [code: models/FreTS.py#L71-L81]. The off-diagonal entries of r1, i1, r2 and i2 get no gradient and have no effect.
• The channel learner is never executed. The code tests self.channel_independence == '1' FreTS.py. The flag is parsed as an integer run_longExp.py, so the comparison is always false. The temporal learner is the only one that runs. This also means the channel-learner results in tab. 3 cannot be reproduced from the code as shipped. The paper says the channel learner is part of the default model app. B.3 §3.1.
• Residual connection. The code adds the lifted input back before the head FreTS.py FreTS.py. Equation 5 has no such term eq. 5.
• Activation and soft-shrink. The paper leaves σ unnamed §3.2 eq. 6. The code applies ReLU to the real and imaginary parts, then a soft-shrink with threshold 0.01 FreTS.py. The soft-shrink is not in the paper.
• Head activation. The code's head is Linear, LeakyReLU, Linear [code: models/FreTS.py#L27-L31]. The paper leaves σ unnamed eq. 5.
• Rfft length. The code uses L//2+1 bins FreTS.py. The paper writes L/2 §3.1.
• Dataset split for ETT. The paper uses 7:2:1 §4. The ETT loader uses fixed borders of 12, 4 and 4 months (8640, 2880 and 2880 hours) [code: data_provider/data_loader.py#L47-L48].
• Training configuration. The paper leaves the optimizer, learning rate, epochs and patience unstated. The code uses Adam, learning rate 1e-4, batch size 8, 10 epochs, patience 3, and halves the rate each epoch [code: exp/exp_main.py#L43] run_longExp.py [code: utils/tools.py#L11-L12]. The code default batch size (8) lies in the paper's search set {4, 8, 16, 32} app. B.3, but the paper does not say which value it used for the long-term tables.
• Traffic lookback. §4.1 gives lookback 48 for Traffic. The caption of tab. 2 gives lookback 96 for all datasets except Traffic, whose horizons are 48 to 336. The code has no Traffic script, so this cannot be checked.
• No discrepancy found for embedding width 128, hidden width 256, MSE loss exp_main.py, min-max normalisation fitted on the training split data_loader.py, or lookback 96 and horizons for ETT and Weather run_longExp.py.

Pitfalls

• Diagonal versus full weight matrix. Implementations that follow eq. 6 and eq. 7 will have d² trainable weights per layer and will not match the released model. Implementations that follow the code will train only d diagonal entries per weight. Theorem 2 eq. 9 is stated for the full complex matrix, so it does not directly describe the released layer.
• The channel learner is off in the released code. Results for FreCL/FreTL tab. 3 and any short-term setup that depends on the channel learner cannot be checked with this repository.
• Activation and initialisation are unstated. The σ in the frequency MLP and the head, the initial scale of the frequency weights and the embedding, and the soft-shrink threshold are not in the paper. The code choices are in the Hyperparameters table and may change results.
• Residual term. The code's residual connection (lifted input added back before the head) is not in the paper's equations. Reproductions that follow the paper alone will differ from the code.
• Numerical sensitivity of the frequency representation. The frequency representation uses an orthonormal FFT (norm='ortho') in the code [code: models/FreTS.py#L45] FreTS.py, and the inverse uses the same normalisation FreTS.py. Changing the normalisation changes the scale of every frequency component, and the paper does not state it.
• Claims in the text are stronger than the tables. §4.1 says FreTS "outperforms all baselines on all datasets" for long-term forecasting. In tab. 2, PatchTST has the lower MAE and RMSE on Traffic at horizons 48 and 96 (MAE 0.016 vs 0.018 and 0.018 vs 0.020), and on Electricity at horizon 336 (MAE 0.043 vs 0.046). LTSF-Linear ties FreTS on ETTh1 at horizon 720.
• Baseline count. §4 refers to thirteen baselines for short-term forecasting, which matches tab. 1. app. B.2 refers to eighteen baselines.
• Short-term data loading is not in the repository. Solar, Wiki, METR-LA, ECG and COVID-19 short-term settings have no code. The ECG and COVID-19 CSVs are in dataset/, but the repository does not contain the short-term training loop.
• Averages are not recomputed. The paper's averaged gains (9.4% MAE, 11.6% RMSE for short-term; more than 20% for long-term against Transformers) are given in the text and can be checked against tab. 1 and tab. 2 only by computing them yourself.
• No variance reported. The paper gives no repeated runs or significance tests. The code fixes seed 2021 run_longExp.py. Gaps of 0.001 in MAE, as in several rows of tab. 2, have no error bars to judge them by.

Lineage

• Builds on: Discrete Fourier transform eq. 1 eq. 2. The paper adopts the LTSF-Linear experimental protocol for long-term forecasting app. B.3 and reuses the DLinear and NLinear linear-model setups as ablation targets §4.2 tab. 4. The lookback of 96 follows FEDformer and Autoformer app. B.3.
• Compared against: VAR, SFM, LSTNet, TCN, DeepGLO, Reformer, Informer, Autoformer, FEDformer, GraphWaveNet, StemGNN, MTGNN, AGCRN (short-term); PatchTST, LTSF-Linear, FEDformer, Autoformer, Informer, Reformer (long-term) §4 tab. 1 tab. 2. The paper also names TAMP-S2GCNets, DCRNN and STGCN for graph-based short-term comparisons §4 tab. 9 tab. 8.
• Related frequency-domain forecasters named by the authors: SFM, StemGNN, Autoformer, FEDformer, CoST and FiLM §2. These are not cards in this wiki yet, so they are not linked.
• Superseded by: not known from content in this wiki.