TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting

In plain words

Forecasting a time series is hard when the signal mixes several kinds of change at once: a slow drift over months, a repeating daily or weekly rhythm, and short bursts. A series looks different depending on how far you zoom out. Fine detail shows the short rhythms, while a coarse view shows the drift.

TimeMixer uses this idea. It makes a stack of copies of the past input, each one smaller than the last, so the model sees the same history at several zoom levels. For each zoom level it separates the signal into a repeating part and a slow-moving part. The repeating detail from fine levels is blended upward into coarser levels, and the slow trend from coarse levels is blended downward into finer ones. Last, a simple linear predictor at each zoom level makes its own guess at the future, and the guesses are added together.

The network uses only linear layers and GELU activations, with no attention. The authors report that this makes it efficient to run, and that it forecasts more accurately than the models they compare it against.

TL;DR

• TimeMixer forecasts from the past at several downsampled scales, using only MLP blocks §3.
• Each scale is split into seasonal and trend parts §3.2 eq. 3. Seasonal parts are mixed bottom-up and trend parts top-down eq. 4 eq. 5.
• The final forecast is the sum of one predictor per scale §3.3 eq. 6.
• Author-reported: best MSE and MAE on all eight long-term benchmarks at input length 96 tab. 2, and best error on the PEMS and M4 short-term tables tab. 3 tab. 4.
• Use it when you want a small MLP forecaster with explicit multiscale structure. Check the code-vs-paper differences before you reimplement it.

Problem

Real series combine many kinds of variation, such as increasing, decreasing and fluctuating patterns, and these are hard to separate §1. Earlier deep forecasters handled this with series decomposition (seasonal plus trend) or multiperiodicity analysis, and used CNN, RNN, Transformer or MLP backbones §1 §2. The authors argue that the signal at different sampling scales should be used directly: fine scales show detailed patterns and coarse scales show macro variation, and the future depends on both §1. Prior multiscale models (Pyraformer, SCINet) do not use past information from all scales together when they predict the future §2.

Method

The paper has three parts: a multiscale input, a past-mixing stack (PDM) and a future-prediction block (FMM) §3.

3.1 Multiscale mixing architecture §3.1. The past series x of length P and C variates is average-pooled M times. Scale m has length floor(P/2^m), and scale 0 is the input itself §3.1. Each scale is embedded into d_model channels §3.1. Stacked PDM blocks (L of them) then mix the past across scales eq. 1.

3.2 Past-Decomposable-Mixing (PDM) §3.2 eq. 3. For block l, each scale is split into a seasonal part and a trend part by a moving-average decomposition (from Autoformer) §3.2. Then:

1. Seasonal parts are mixed bottom-up: scale m gets a residual term from scale m-1 eq. 4. The mixer is two linear layers over time with a GELU between them, from length P/2^{m-1} to P/2^m §3.2. The width between the two layers is not stated, and the code uses P/2^m (see Pitfalls).
2. Trend parts are mixed top-down: scale m gets a residual term from scale m+1, computed the same way as step 1 but in the opposite direction eq. 5 §3.2.
3. The mixed seasonal and trend parts are added, passed through a feed-forward block (two linear layers with GELU, over channels), and added back to the block input as a residual eq. 3 §3.2.

The authors argue that seasonal detail is most useful when it flows from fine to coarse, and trend is most useful when it flows from coarse to fine §3.2. They also test the opposite directions and removing either mixing or decomposition, and report that these variants are worse tab. 5 §4.2.

3.3 Future-Multipredictor-Mixing (FMM) §3.3 eq. 2 eq. 6. After the last PDM block, each scale m has its own predictor. Each predictor is one linear layer over time from floor(P/2^m) to F, and then a projection to C variates §3.3. The final forecast is the sum over scales eq. 6. The authors state that the predictors use different information and so are complementary §3.3 fig. 4.

Overall pipeline, in the paper's notation:

• Multiscale input: X = {x_0, ..., x_M}, scale m of length floor(P/2^m) §3.1.
• Past mixing: X^l = PDM(X^{l-1}) for l = 1..L eq. 1.
• Future: x_hat = FMM(X^L) eq. 2.

Pseudocode for one forward pass. The paper does not describe normalization, so the per-scale normalization step is taken from the code and marked as such:

# TimeMixer forward pass, channel-independent setting (code default)
# Input: past (B, P, C)     Output: forecast (B, F, C)
# M = down-sampling steps, L = PDM blocks  [§3.1] [app. A, tab. 7]

def timemixer(past):
    # 1. Build M+1 scales by average pooling; each step halves the
    #    length  [§3.1]
    scales = [past]
    for m in 1 to M:
        scales.append(avg_pool(scales[m - 1], window=2))  # (B, P/2^m, C)

    # 2. Per-scale instance normalisation. Code only, not in the paper
    #    [code: models/TimeMixer.py#L213-L218]
    stats = [normalise_stats(s) for s in scales]

    # 3. Channel independence: each variate becomes its own univariate
    #    series  [code: models/TimeMixer.py#L349-L350]
    series = [reshape_per_variate(s) for s in normed]  # (B*C, P/2^m, 1)

    # 4. Embed every scale into d_model channels  [§3.1]
    #    [code: layers/Embed.py#L176-L194]
    hidden = [embed(s) for s in series]  # (B*C, P/2^m, d_model)

    # 5. L PDM blocks mix seasonal and trend parts across scales
    #    [eq. 1] [eq. 3]
    for block in pdm_blocks:
        hidden = pdm_block(hidden)

    # 6. Per-scale linear predictor over time, then a projection to one
    #    channel  [§3.3]
    preds = []
    for m in 0 to M:
        y = predict_layers[m](hidden[m])  # (B*C, d_model, F)
        preds.append(projection(y))       # (B*C, F, 1)

    # 7. Sum the scale predictions  [eq. 6]
    out = sum(preds)

    # 8. Undo normalisation with scale-0 statistics
    #    [code: models/TimeMixer.py#L382]
    return reshape_back(denormalise(out, stats[0]))  # (B, F, C)

# One Past-Decomposable-Mixing block  [§3.2] [eq. 3]
# Input: list of M+1 scales, each (B*C, T_m, d_model)

def pdm_block(hidden):
    # 1. Split each scale into seasonal and trend parts with a moving
    #    average, kernel 25 by default  [eq. 3]
    #    [code: layers/Autoformer_EncDec.py#L21-L53]
    seasonal = []
    trend = []
    for h in hidden:
        s, t = moving_average_decomp(h)
        seasonal.append(s)
        trend.append(t)

    # 2. Bottom-up seasonal mixing: fine to coarse, residual  [eq. 4]
    #    The code uses the already-updated finer scale
    #    [code: models/TimeMixer.py#L61-L67]
    for m in 1 to M:
        seasonal[m] = seasonal[m] + bottom_up(seasonal[m - 1])

    # 3. Top-down trend mixing: coarse to fine, residual  [eq. 5]
    #    The code uses the already-updated coarser scale
    #    [code: models/TimeMixer.py#L105-L111]
    for m in M - 1 down to 0:
        trend[m] = trend[m] + top_down(trend[m + 1])

    # 4. Recombine each scale, add the feed-forward block (d_model to
    #    d_ff and back), keep the residual  [eq. 3]
    out = []
    for m in 0 to M:
        mixed = seasonal[m] + trend[m]
        out.append(hidden[m] + feed_forward(mixed))  # (B*C, T_m, d_model)
    return out

Key equations, as printed in the paper:

\mathcal{X}^{l} = \operatorname{PDM}(\mathcal{X}^{l-1}) \qquad [\text{eq. 1}]

\widehat{\mathbf{x}} = \operatorname{FMM}(\mathcal{X}^{L}) \qquad [\text{eq. 2}]

\mathcal{X}^{l} = \mathcal{X}^{l-1} + \operatorname{FeedForward}\!\big(\operatorname{S\text{-}Mix}(\{\mathbf{s}_m^l\}) + \operatorname{T\text{-}Mix}(\{\mathbf{t}_m^l\})\big) \qquad [\text{eq. 3}]

\mathbf{s}_m^l \leftarrow \mathbf{s}_m^l + \operatorname{Bottom\text{-}Up\text{-}Mixing}(\mathbf{s}_{m-1}^l), \quad m = 1..M \qquad [\text{eq. 4}]

\mathbf{t}_m^l \leftarrow \mathbf{t}_m^l + \operatorname{Top\text{-}Down\text{-}Mixing}(\mathbf{t}_{m+1}^l), \quad m = M-1..0 \qquad [\text{eq. 5}]

\widehat{\mathbf{x}} = \sum_{m=0}^{M} \widehat{\mathbf{x}}_m, \qquad \widehat{\mathbf{x}}_m = \operatorname{Predictor}_m(\mathbf{x}_m^L) \qquad [\text{eq. 6}]

Hyperparameters

Paper configuration, from the appendix table app. A, tab. 7. "M" is the number of down-sampling steps, which the code calls down_sampling_layers. "Layers" is the number of PDM blocks, which the code calls e_layers. LR is the initial learning rate, which the code passes as max_lr to OneCycleLR. The scripts in the repository do not always match this table (see Paper vs code).

Settings the paper does not state. Values are the code defaults, or what the run scripts pass. Each is cited to the code.

Benchmark data sizes and forecastability, from app. A, tab. 6. "Series Length" in that table is the prediction horizon for M4, not the input length.

Results

Author-reported. Setup: input length 96 for every model, results averaged over the 4 prediction lengths, averaged over 3 runs, one NVIDIA A100 80GB §4 tab. 2 app. A. Baselines are re-run with a unified setting, not copied from their papers §4. The hyperparameter-searched upper bounds are in app. E and are not reproduced here.

Long-term forecasting (MSE / MAE, averaged over four horizons) tab. 2.

Headline numbers from the text §4.1: the Weather MSE is 9.4% lower than PatchTST's, and the Solar-Energy MSE is 24.7% lower. The authors also note good performance on low-forecastability datasets such as Solar-Energy and ETT §4.1.

Short-term forecasting, PEMS (input 96, horizon 12) tab. 3. Bold is TimeMixer's row.

Short-term forecasting, M4 (univariate, single variate per series) tab. 4. Bold is TimeMixer's column. The table's header abbreviations are kept: FED. is FEDformer, Auto. is Autoformer, Pyra. is Pyraformer, In. is Informer. N-BEATS was run without its ensemble tab. 4 caption.

Ablations tab. 5. Run on M4, PEMS04 and the ETTm1 predict-336 setting. The official design is case 172. ↗ is bottom-up mixing and ↙ is top-down. A check mark means the component is present and × means absent.

Other §4.2 claims, reported in text and figures only: the seasonal mixing weights are periodic and the trend mixing weights are local fig. 3. The predictions from fine scales track detail and those from coarse scales track trend fig. 4. The number of scales M is set to 3 for long-term and 1 for short-term forecasting, to trade off accuracy and cost §4.2 app. D.

Efficiency, training memory and time app. B, tab. 8. Batch size is not given in the table; fig. 5 says batch size 16 on ETTh1, averaged over 100 iterations.

The authors attribute the speed gap to the fully MLP design app. B.

Paper vs code

Official code: https://github.com/kwuking/TimeMixer, Apache-2.0, cloned at commit e24610583b36fdd8c76cc17a8df4e65759a5f460. The paper links it §7. The code is the reference for every item below; the paper is silent on most of them.

• Normalisation is missing from the paper. The paper never mentions it. The code wraps every scale in an instance normalisation with learnable affine parameters, and undoes it on the output with scale-0 statistics TimeMixer.py TimeMixer.py TimeMixer.py StandardNorm.py. Turned off by use_norm 0, which the Solar script sets TimeMixer_unify.sh.
• Residual and feed-forward differ for channel-mixing. Eq. 3 adds a residual and a feed-forward block to every PDM block eq. 3. The code adds that residual only when channel_independence is 1 TimeMixer.py. For channel-mixing it applies a feed-forward (cross_layer) to the seasonal and trend parts before decomposition, and returns their sum with no residual TimeMixer.py TimeMixer.py.
• Extra prediction path for channel-mixing. The paper's FMM is only a sum of per-scale predictors eq. 6. For channel-mixing the code adds a second regression path with its own per-scale linear layers, applied to the moving-average component of the input TimeMixer.py TimeMixer.py TimeMixer.py.
• Projection layer shared across scales. The paper describes one predictor per scale that also projects to C variates §3.3. The code uses one projection module for all scales; only the time-axis linear layer is per scale TimeMixer.py.
• Unused layer norm. PastDecomposableMixing defines a LayerNorm, but forward never calls it TimeMixer.py.
• Mixer intermediate width is not stated. The paper gives the input and output lengths of each mixer, but not the width between its two linear layers. The code uses the output length P/2^m there, in both mixers TimeMixer.py TimeMixer.py.
• Hyperparameters in the scripts differ from Table 7. Checked against app. A, tab. 7:
  • Solar: script d_model 512, d_ff 2048, LR 1e-3, 3 layers, 10 epochs, M=2, channel_independence 0 TimeMixer_unify.sh TimeMixer_unify.sh. Table: d_model 128, LR 1e-2, 2 layers, 20 epochs, M=3.
  • Weather and Electricity: script 3 layers TimeMixer_unify.sh TimeMixer_unify.sh. Table: 2 layers.
  • Traffic: script 3 layers and no epochs flag TimeMixer_unify.sh. Table: 2 layers, 20 epochs.
  • PEMS: script LR 3e-3 TimeMixer.sh. Table: 1e-3.
  • ETTh1, M4, PEMS (layers, d_model, batch) and Weather/ECL/Traffic batch sizes match the table.
  • The text says "By default, TimeMixer contains 2 PDM blocks" app. A. Table 7 and the scripts use 4 to 5 for M4 and PEMS.
• Metric formulas. The appendix writes MAE and RMSE as sums over the horizon app. A, metric details. The code uses means: MAE and MSE are np.mean, and RMSE is the square root of MSE metrics.py.
• Stated M is down-sampling steps. Table 7 "M" equals the code's down_sampling_layers run.py. The number of scales is M+1.

Pitfalls

• Normalisation is the first thing to add. Without the per-scale instance normalisation the model will not match reported numbers, and the paper gives no hint of it. Decide which statistics to use for the output (scale 0, as in the code).
• Channel independence changes the architecture. With the default channel_independence 1, each variate is an independent univariate series and the model is shared across variates TimeMixer.py. The paper's Eq. 3 with its residual and feed-forward matches this default TimeMixer.py. For channel-mixing (channel_independence 0, used by the Solar script) the code's block differs from Eq. 3 (see Paper vs code).
• Mixer intermediate width is underspecified. The paper gives the mixer's input and output lengths only. The code's intermediate width is the output length P/2^m, and the second linear layer maps it to itself TimeMixer.py TimeMixer.py.
• Mixing order matters and is sequential. Bottom-up mixing uses the already updated finer scale, and top-down mixing uses the already updated coarser scale TimeMixer.py TimeMixer.py. A parallel implementation will differ.
• Ablation text and table disagree on cases 176 to 178. The appendix says cases 176 to 178 reverse the directions app. F. Table 5 shows 176 uses top-down on both parts and 177 bottom-up on both. Only 178 swaps the two directions. Implement the official design (172) and use the table as the reference.
• Forecastability values disagree. Table 1 gives ETT 0.46 and Solar 0.33 tab. 1. The appendix gives ETTh1 0.38, ETTh2 0.45, ETTm1 0.46, ETTm2 0.55 app. A, tab. 6. The PEMS08 test size of 265 looks like a typo next to validation 3548 app. A, tab. 6. The M4 "Series Length" column is the horizon.
• Hyperparameter search changes the picture. Under per-model search the gap to PatchTST shrinks, which the authors state app. E. Use the unified numbers for comparisons, and do not compare them with searched numbers from other papers.
• Appendix efficiency table has a typo. One DLinear entry reads "0.0.004" app. B, tab. 8. Read it as 0.004 only if you can check it another way.
• Error bars and full searched tables are not in this card. The standard deviations and significance tests app. C and the full searched long-term results app. E are in the paper. The full ablation grids app. F are in the paper's LaTeX and not reproduced here.
• Costs the abstract hides. Efficiency depends on series length. At length 3072, TimeMixer takes 0.016 s per iteration and 1411 MiB, against 0.094 s and 16119 MiB for PatchTST app. B, tab. 8. The batch size of that table is not stated. The mixing layers are linear in input length, and the authors name the resulting parameter growth as a limitation for mobile use app. J.
• Only the temporal dimension is mixed. The authors name variate mixing as future work app. J.
• Scale choice is a dataset decision. M=3 suits long series and M=1 suits short ones §4.2 app. A. Test M on your own data.

Lineage

• Builds on: series decomposition from Autoformer, used for the seasonal and trend split §3.2.
• Related decomposition model: DLinear, which uses decomposition before linear regression §2.
• Related multiscale designs: Pyraformer (pyramidal attention) and SCINet (bifurcate downsampling tree), which the authors say do not use past information at all scales together for the future prediction §2.
• Mixing ideas: MLP-Mixer (channel and patch mixing) and FNet (Fourier token mixing) §2.
• Compared against: PatchTST, TimesNet, SCINet, Crossformer, MICN, FiLM, DLinear, LightTS, FEDformer, Stationary, Pyraformer, Autoformer, Informer, N-HiTS, N-BEATS §4 tab. 2 tab. 3 tab. 4. Two concurrent MLP models (MTSMixer and TSMixer) and Scaleformer are in app. G, not in the main tables app. G.
• Superseded by: not stated in this paper.