Long-term Forecasting with TiDE: Time-series Dense Encoder

In plain words

Long-range forecasting means: given a long stretch of past values for many related series, plus side information that is known in advance (calendar features, promotions, and similar), predict many steps into the future. Transformer models were the usual choice for this, but their cost grows fast with the length of the history. Earlier work also found that a single linear map from history to future could beat them on standard benchmarks.

TiDE keeps the simplicity of that linear approach and adds non-linear layers. It flattens the past values and the known-in-advance side information into one vector, passes it through small neural blocks that each include a straight-through linear shortcut, and then produces the whole forecast in one pass. A plain linear path from the history to the forecast is added at the end, so the linear model is a special case of TiDE.

The authors also prove that a linear version of this design is near-optimal for a simple linear dynamical system. Their main claim is that the model matches or beats strong forecasters on common benchmarks while running much faster.

TL;DR

• An MLP-only encoder-decoder for multivariate long-horizon forecasting with optional covariates. It has no attention, recurrence or convolution §1 §4.
• Look-back is 720 for all horizons and the loss is MSE §5.1 §4.2.
• Author-reported: it matches or beats PatchTST/64 on most of seven benchmarks, beats it on Traffic at every horizon in MSE, and is 5x faster in inference and over 10x faster in training than the best Transformer §1 tab. 2.
• The released code differs from the paper in several ways that change the model (see Paper vs code). The biggest are a learned series-ID embedding and a block count of layers minus one.

Problem

Long-term multivariate forecasting: N time series, a look-back of length L, a horizon of length H, dynamic covariates of dimension r that are known up to L+H, and static attributes §3.2. Accuracy is measured by MSE on the standardized data §5.1 eq. 2.

Transformer forecasters approximate full self-attention with sub-quadratic variants, and the paper argues this hurts accuracy on long horizons §2 §5.1. The vanilla Transformer used by PatchTST needs memory quadratic in L, which is the efficiency issue the paper targets §5.3. Linear models do well on these benchmarks but cannot model non-linear dependencies or covariates §4 §2.

Method

Overview. TiDE is channel-independent: each series goes through the model on its own, and the weights are shared across all series §4. The input for series i is its past values y(1:L), its covariates x(1:L+H) and its static attributes a(i). The output is the H-step forecast §4 §3.2.

Residual block. The basic layer is a residual MLP with one hidden layer and a ReLU. It has a fully linear skip connection. Dropout is applied only on the output linear layer, and layer norm is applied to the sum of the output and the skip §4. Whether layer norm and dropout are on is set per dataset tab. 8.

Steps.

1. Feature projection. Each time step's covariate vector (dimension r) goes through a residual block into a small width of 4, the temporal width §4.1 eq. 3 app. B.3.
2. Dense encoder. The projected covariates for the past and future steps are flattened and concatenated with the past values and the static attributes. A stack of residual blocks of width hiddenSize maps this vector to an embedding §4.1 eq. 4.
3. Dense decoder. Residual blocks map the embedding to a vector of size H times p, where p is decoderOutputDim. It is reshaped into a p-by-H matrix whose t-th column is the decoded vector for horizon step t §4.2.
4. Temporal decoder. For each horizon step, a residual block with a single output combines the decoded vector with the projected covariates of that same step. This gives a direct path from known future covariates to the prediction §4.2.
5. Global linear residual. A linear map from the look-back to the horizon is added to the output. A purely linear model is therefore always a special case of TiDE §4.2.

Training uses mini-batches of series, where each batch holds batchSize series and their windows, with MSE as the loss §4.2. The paper says each epoch covers all look-back and horizon pairs in the training range, and that two mini-batches may overlap in time points §4.2.

# Residual block: one ReLU hidden layer, linear skip, dropout, layer norm  [§4]
# Input: inputs (..., d_in)   Output: (..., d_out)
def residual_block(inputs):
    # Hidden layer with ReLU  [§4]
    hidden = relu(linear_a(inputs))
    # Output layer; dropout is applied to this layer only  [§4]
    output = dropout(linear_b(hidden))
    # Fully linear skip connection, added to the output  [§4]
    output = output + linear_skip(inputs)
    # Optional layer norm on the sum; on or off per dataset  [§4] [tab. 8]
    return layer_norm(output)

# TiDE forward pass for one series  [§4.1] [§4.2] [eq. 3] [eq. 4]
# Inputs: past_values (L,), past_covariates (L, r), future_covariates (H, r)
# Output: forecast (H,)
def tide_forward(past_values, past_covariates, future_covariates):
    # Project each step's covariates to a small width  [§4.1] [eq. 3]
    past_projected = residual_block(past_covariates)        # (L, 4)
    future_projected = residual_block(future_covariates)    # (H, 4)
    # Flatten everything into one vector and encode it  [§4.1] [eq. 4]
    encoder_input = concat(
        past_values,
        flatten(past_projected),
        flatten(future_projected),
        static_attributes,
    )
    embedding = encoder(encoder_input)                      # (hidden,)
    # Decode one vector per future step  [§4.2]
    decoded = reshape(decoder(embedding), (p, H))           # (p, H)
    # Per step, join the decoded vector with that step's covariates  [§4.2]
    temporal_inputs = concat(decoded, future_projected.T)   # (p + 4, H)
    forecast = temporal_decoder(temporal_inputs.T)          # (H, 1)
    # Add the global linear shortcut from the look-back  [§4.2]
    return squeeze(forecast) + linear(past_values)          # (H,)

Key equations. The feature projection and the encoder are numbered in the paper; the decoders are written without numbers §4.1 §4.2.

\tilde{\mathbf{x}}^{(i)}_t = \mathrm{ResidualBlock}\left(\mathbf{x}^{(i)}_t\right) eq. 3

\mathbf{e}^{(i)} = \mathrm{Encoder}\left(\mathbf{y}^{(i)}_{1:L};\; \tilde{\mathbf{x}}^{(i)}_{1:L+H};\; \mathbf{a}^{(i)}\right) eq. 4

\mathrm{MSE} = \frac{1}{NH} \sum_{i=1}^{N} \left\lVert \mathbf{y}^{(i)}_{L+1:L+H} - \hat{\mathbf{y}}^{(i)}_{L+1:L+H} \right\rVert_2^2 eq. 2

Theory. The authors study the linear analogue of the model, which is a linear map from a finite context (the last k values plus covariates) to the horizon. For data generated by a linear dynamical system with bounded norms and a stable transition matrix (0 ⪯ A ⪯ γI with γ < 1), they prove a generalization bound. The excess risk over the best system predictor is at most ε plus a term of order log(1/ε)·√(log(1/δ))/√N, with probability 1−δ, when k = Θ(log(1/ε)) app. A.1. The bound is for this linear autoregressive class, not for the full TiDE network app. A.1.

Ablations and checks. The temporal decoder helps the model adapt to future covariates more quickly, shown on a semi-synthetic electricity variant §5.4 fig. 3. Removing all residual connections lowers accuracy at horizons 96 to 336 on Electricity and leaves horizon 720 unchanged §5.4.

Hyperparameters

Per-dataset settings for the published results tab. 8. The paper sets numEncoderLayers and numDecoderLayers separately in its tables, but every row has them equal. Read "layers" against the code before you implement, see Paper vs code.

Settings that apply to all experiments, and what the paper leaves open:

Results

Author-reported. Setup: seven benchmarks (Weather, Traffic, Electricity, four ETT datasets), multivariate, standardized data, MSE and MAE, four horizons, and look-back 720 §5.1 tab. 1. TiDE numbers are means over 5 runs tab. 2. Transformer and N-HiTS baseline numbers come from the PatchTST paper, and the PatchTST numbers are corrected for a test data-loader bug §5.1 tab. 2. DLinear numbers come from the DLinear paper app. B.1. The table below shows three baselines. The full table, with FEDformer, Autoformer, Informer, Pyraformer, LogTrans and N-HiTS, is in tab. 2.

The paper groups TiDE with PatchTST, N-HiTS and DLinear as the clear leaders over the Transformer baselines §5.1. TiDE beats PatchTST on Traffic at all horizons by a clear margin §5.1 tab. 2. The paper reports that, apart from Weather, it either outperforms PatchTST or stays within statistical significance for most horizons §5.1. The table above shows the cases where PatchTST has the lower MSE, such as ETTh2 at horizon 720.

Demand forecasting (M5). Test WRMSSE on the private leaderboard, three runs. TiDE with static and dynamic covariates: 0.611 ± 0.009. TiDE with date features only: 0.637 ± 0.005. DeepAR with static and dynamic covariates: 0.789 ± 0.025. PatchTST (no covariates): 0.976 ± 0.014. The authors trained DeepAR and TiDE with a zero-inflated negative binomial loss, following the GluonTS template §5.2 tab. 3 app. B.1.

S4 comparison. On four datasets at horizons 336 and 720, TiDE has lower MSE than S4 in all eight pairs. The S4 numbers come from the S4 paper tab. 6 app. B.1.

Efficiency. The paper gives no numeric timing table in the text. The claims are: inference is about 5x faster and training more than 10x faster than the best Transformer §1. The Electricity timing plots show an order of magnitude gap in inference, and PatchTST runs out of memory for look-back 1440 and longer §5.3 fig. 2. The hardware was one NVIDIA T4 GPU §5.3.

Synthetic linear dynamical system. A 30-dimensional system with a Wishart-based transition matrix, look-back and horizon both 320, and 3 runs. MSE: linear 0.510 ± 0.001, LSTM 1.455 ± 0.455, Transformer 0.731 ± 0.041 tab. 4 app. A.2.

Paper vs code

Official code: the tide/ directory of google-research/google-research at commit 3ef52111c1fabc301880818f5a547570e926a76b, Apache-2.0. Its README names this arXiv paper and its authors. The code is TensorFlow. The paper text has no code link and no project page. A GitHub repository search for "tide" returned only third-party implementations, and a search inside the google-research organization found tide/. The release has scripts for seven datasets and none for M5.

The discrepancies below were found by reading the code and the paper. Each one is a place where the code and the paper disagree or where the paper is silent and the code decides.

1. Learned series-ID embedding. The code adds a 16-dimensional embedding per series to the encoder input. The paper's encoder input contains only the past values, projected covariates and static attributes §4.1 eq. 4. models.py models.py. This means a trained model needs the series index at inference time.
2. Residual block count is layers minus one. num_layers = n builds n - 1 residual blocks in the encoder and in the decoder. For n = 1, the encoder is one plain linear layer with no activation, no skip, no dropout and no layer norm. The decoder is then also one plain linear layer models.py models.py. The paper describes the encoder as "multiple residual blocks" §4.1 and tunes numEncoderLayers over {1, 2, 3} tab. 7. The published Traffic, Weather and ETTm1 settings use 1 tab. 8 and so use no residual block in the encoder or decoder traffic.sh weather.sh ettm1.sh.
3. Encoder and decoder depth are tied. The code has one num_layers flag for both train.py. The paper tunes them separately tab. 7. All published rows set them equal, so this does not change the published configs.
4. Projection hidden width. The feature-projection block is hard-coded to hidden width 64 and output width 4 train.py. The paper gives only the output width app. B.3.
5. Data splits differ for the ETT datasets. Electricity, Traffic and Weather use 70/10/20 splits, as stated in the paper §5.1. The ETT datasets use fixed 60/20/20 boundaries (for example 8640, 11520, 14400 for ETTh) train.py. The paper says 7:1:2 for all datasets §5.1.
6. Dataset length is truncated. The code uses only the data up to the end of the test range. For ETTh that is 14400 points, while the paper's Table 1 lists 17420 train.py tab. 1. For ETTm it is 57600 points, against 69680 in Table 1 train.py tab. 1.
7. ETTh1 look-back for horizons 336 and 720. The script sets --hist_len=720, then sets --hist_len=336 on the same command. Absl flags normally keep the last value, so the effective look-back is 336. The paper says 720 for all horizons §5.1 etth1.sh etth1.sh. This is not verified by running the code.
8. ETTh1 has two configurations. Horizons 96 and 192 use hidden size 512, decoder output 32, final hidden 16, dropout 0.5, learning rate 9.85e-4, and holiday=true etth1.sh. The row in Table 8 matches only the horizon 336 and 720 block tab. 8.
9. Holiday features. Most scripts set holiday=true. Electricity and Weather set it to false traffic.sh electricity.sh. The paper describes only time-derived features app. B.2.
10. Count of time features. The paper says there are 8 time features app. B.2. The code has 7 base features (minute, hour, day of month, day of week, day of year, month, week of year), plus holiday columns when holiday is on time_features.py.
11. Model selection. The code keeps the epoch with the best validation RMSE, not validation MSE or loss, and only after min_num_epochs train.py. Test metrics are recorded at that epoch train.py. The paper says the validation loss was used for tuning app. B.3.
12. Early stopping. The code stops once more than 40 epochs pass without improvement (patience 40), with a maximum of 100 epochs train.py train.py. The main-text results do not state this app. B.1.
13. Learning-rate schedule. CosineDecay runs over 30 x training length optimizer steps, with the training length measured in time steps, and no warm-up. Adam also has a value clip of 1000 train.py. The paper says only "cosine decay" app. B.3.
14. Batch composition per epoch. For each window, the loader draws N // B + 1 random batches of B series. Series are sampled without replacement inside a batch but can repeat across batches [code: tide/data_loader.py#L157-L160]. The paper says each epoch covers every look-back and horizon pair §4.2. For N = 862 and B = 512, some pairs appear twice in an epoch and some not at all.
15. Reversible normalization. The code uses a learned per-series scale and bias, plus the per-window mean and std of the look-back, and inverts both on the output models.py models.py models.py. The paper mentions only "reversible instance normalization" app. B.3.
16. Categorical covariates. The code embeds categorical covariates into 4-dimensional vectors per time step, not as static attributes [code: tide/models.py#L163-L169]. The paper's static attributes are used for M5 app. B.1.
17. Things the code matches. The residual block (ReLU hidden layer, linear skip, dropout on the output layer, layer norm on the sum), the temporal decoder (per-step block on decoded vector plus projected future covariates), the global linear residual from the look-back, the MSE loss and the Adam optimizer all match the paper models.py models.py §4 §4.2.

Pitfalls

• Match the published numbers or match the paper text, not both. Following the paper's wording ("n residual blocks") gives a deeper model than the code's n - 1. The published Traffic, Weather and ETTm1 settings have no residual block in the encoder or decoder, as the code builds them.
• The series-ID embedding limits use. A model trained with it can only forecast series it saw in training, and it needs the series index. The paper does not mention this models.py.
• ETT comparisons need the code's splits. Use 60/20/20 for ETT, not 7:1:2, or the numbers are not comparable with Table 2. Use the code's truncation as well train.py.
• Look-back for ETTh1 is ambiguous. The paper says 720 for all horizons. The script's effective value at horizons 336 and 720 is 336 [code: tide/scripts/etth1.sh#L46-L54].
• Standardized metrics. With normalize=True, MSE and MAE are in units of the training-period standard deviation. They are not comparable with raw-scale errors on the same data §5.1 data_loader.py.
• The learning-rate schedule does not depend on epochs. The decay horizon is 30 x training time steps, counted in optimizer steps. Stopping early leaves the rate high. Adam's value clip is 1000, which the paper does not mention train.py.
• Dropout and layer norm vanish in the one-layer case. For num_layers = 1 the encoder and decoder are plain linear layers, so dropout and layer norm apply only in the projection and temporal decoder blocks [code: tide/models.py#L64-L70].
• Theory is for a linear class. The bound in app. A.1 is for a linear autoregressive predictor on bounded LDS data. It does not prove anything about TiDE's non-linear blocks. The constants (c, γ and the O(·) factor) are not given numerical values app. A.1.
• Costs the abstract hides. Training uses batches of 512 series with hidden sizes up to 1024 tab. 8. The efficiency gain is measured on one T4 GPU with a batch size of 8 §5.3. Memory grows with the encoder input width, which is L + 4L + 4H + 16 in the code, so long look-backs raise both cost and memory models.py.
• Baseline numbers. PatchTST numbers are the PatchTST paper's, corrected by the TiDE authors for a data-loader bug that mainly affected ETTh1 and ETTh2 §5.1 tab. 2. Re-running the baseline yourself will differ.

Lineage

• Builds on: the finding that a linear model from history to horizon is strong on long-term benchmarks, which the paper attributes to DLinear §2 §4. The paper also cites N-BEATS and N-HiTS as earlier MLP forecasters §2.
• Compared against: FEDformer, Autoformer, Informer, Pyraformer, LogTrans, N-HiTS, PatchTST/64, DLinear, DeepAR (on M5) and S4 §5.1 §5.2 app. B.1.
• Superseded by: not stated in this wiki's content.