Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis

In plain words

Many papers propose new models for forecasting measurements taken over time from many sensors, such as traffic counts or electricity use. Their reported numbers often disagree, and the reason is rarely the model. Each paper tends to clean the data, train and score it in its own way, so the numbers cannot be compared directly.

This paper first builds one shared pipeline that trains and scores every model the same way, and it scores predictions on the original data scale so the errors can be read against the values themselves. It then looks at the data. Some datasets have clear, repeating patterns, some drift over time, and some have almost no pattern. Some datasets also contain many sensors whose recent histories look alike even though their futures differ. The authors argue that the best kind of model depends on which of these cases a dataset is. Fancy models help where the pattern is clear. Simple linear models do about as well where it is not. Graph-based models of sensor relationships help only where sensors are hard to tell apart.

The authors run more than 45 published models through the shared pipeline. Their main message is that a single benchmark number is not enough: check the kind of data before you trust a conclusion about which method is better.

TL;DR

BasicTS+ is a benchmark for multivariate time series forecasting. It fixes the data processing, training and evaluation so that models are compared fairly, and it reports errors on re-normalized data §4. The paper classifies datasets by temporal pattern and by spatial indistinguishability §5. In its experiments, advanced neural networks beat linear models only on datasets with clear patterns, and GCN-style spatial modelling helps only on datasets whose samples are hard to tell apart §6.2 §6.3. Numbers are author-reported.

Problem

Reported results for the same model differ across papers, even on the same dataset and window settings §4.1. The authors attribute this to three sources: data processing (normalization), training configuration (loss and tricks) and evaluation implementation §4.1. Long-term forecasting papers also often report errors on normalized data, which makes errors look small §4.2.

Conclusions also conflict. Linear models were reported to beat Transformers on long-term forecasting, while later papers reported the opposite §1. The authors argue that the conflict comes from dataset heterogeneity, not from the models alone §5.

Method

The paper has two parts: a benchmark and a dataset analysis. There is no new forecasting model.

Benchmark (BasicTS+) §4.1. A unified dataloader, runner and evaluator (architecture figure, §4.1):

1. Normalize inputs and targets with z-score statistics. Z-score is the default choice; min-max is available for comparison §4.1.
2. Add external temporal features such as time of day and day of week §4.1.
3. Train with masked MAE by default, which ignores null target values. Masked MAE is preferred over naive MAE and MSE §4.1.
4. Optionally apply curriculum learning and gradient clipping inside the runner §4.1.
5. Evaluate on re-normalized data with MAE, RMSE, MAPE and WAPE §4.2 eq. 1.

# BasicTS+ unified pipeline  [§4.1] [§4.2]
# Input:  series X (T, N), model f, history length P, horizon F
# Output: test metrics on the original data scale

def run_benchmark(X, model, P, F):
    # 1. Fit the scaler on the training split  [§4.1]
    #    [code: src/basicts/runners/basicts_runner.py#L219-L220]
    scaler = fit_zscore(X[train])
    # 2. Cut sliding windows of P history and F future steps  [§6.1]
    history, future = sliding_windows(X, P, F)
    # 3. Mark nulls; normalize inputs and targets; zero-fill nulls
    #    [code: src/basicts/runners/taskflow/forecasting_taskflow.py#L16-L33]
    history = scaler.transform(history)
    future = scaler.transform(future)
    # 4. Add time-of-day and day-of-week features  [§4.1]
    history = add_time_features(history)
    # 5. Train with masked MAE, ignoring null targets  [§4.1]
    prediction = model(history)
    loss = masked_mae(prediction, future)
    # 6. Undo the scaling, then compute metrics  [§4.2] [eq. 1]
    prediction = scaler.inverse_transform(prediction)
    future = scaler.inverse_transform(future)
    return metrics(prediction, future)

Metrics, with Ω the set of observed samples eq. 1:

\text{MAE} = \frac{1}{|\Omega|}\sum_{i\in\Omega}|y_i-\hat y_i|

\text{RMSE} = \sqrt{\frac{1}{|\Omega|}\sum_{i\in\Omega}(y_i-\hat y_i)^2}

\text{MAPE} = \frac{1}{|\Omega|}\sum_{i\in\Omega}\left|\frac{y_i-\hat y_i}{y_i}\right|

\text{WAPE} = \frac{\sum_{i\in\Omega}|y_i-\hat y_i|}{\sum_{i\in\Omega}|y_i|}

For M4, the paper uses sMAPE, MASE and OWA with their original definitions and does not restate them §4.2.

Temporal heterogeneity §5.1. Datasets are placed in three groups: clear and stable patterns, significant distribution drift, and unclear patterns. The groups are ordered by how predictable the authors consider them. The paper says quantifying predictability is unsolved, so the grouping is made by inspecting plots and t-SNE/kernel-density views of train and test samples §5.1 (t-SNE figure).

Spatial indistinguishability §5.2 eq. 2. A sample is "indistinguishable" when two series look alike in their history but diverge in their future. The paper measures this with cosine similarity between all series pairs, using two thresholds, e_u = 0.9 for the history and e_l = 0.5 for the future.

# Spatial indistinguishability  [§5.2] [eq. 2]
# Input:  series X (T, N), window lengths P (history) and F (future)
# Output: r1 and r2

def indistinguishability(X, P, F, e_u=0.9, e_l=0.5):
    # 1. Cosine similarity of every series pair in each history window
    history_sim = pairwise_cosine(history_windows(X, P))  # (T, N, N)
    # 2. Cosine similarity of the same pairs in the matching future window
    future_sim = pairwise_cosine(future_windows(X, F))    # (T, N, N)
    # 3. Count pairs that look alike in history but not in the future
    hits = sum((history_sim > e_u) & (future_sim < e_l))
    similar = sum(history_sim > e_u)
    # 4. r1 divides by all triples, r2 by the similar-history triples  [§5.2]
    r1 = hits / (T * N * N)
    r2 = hits / similar
    return r1, r2

The paper states that r1 says whether indistinguishability is a major obstacle, and r2 says how strong it is among similar-history samples §5.2.

Hypotheses tested §5. H1.1: advanced networks beat basic networks on datasets with clear patterns, where basic networks under-fit. H1.2: basic networks beat advanced ones on datasets with drift or unclear patterns, where advanced networks over-fit. H2.1: modelling spatial dependencies helps on datasets with significant spatial indistinguishability. H2.2: forcing spatial modelling can hurt on datasets without it.

Hyperparameters

Only settings stated in the paper are listed. Per-model optimizer settings are not given in the paper body.

Defaults in the released code, for comparison (not paper values): batch size 64 tsf_config.py, null value NaN tsf_config.py, seed 42 tsf_config.py.

Results

Author-reported. Setup: each model is trained on the datasets listed in §6.1 (datasets) using the shared pipeline above. Figures below are MAE, as printed in the paper. RMSE, WAPE, MAPE, parameter counts and training speed are in the same tables and are omitted here for length. Table numbers are not given, since they could not be recovered from the LaTeX source. Each table is cited by section.

Headline claims §1 §4.1 §6.4.

• Graph WaveNet's MAE on PEMS04 reproduced with the shared pipeline is up to 33% lower than the values other papers report for it §4.1.
• In the STF table, D2STGNN has the lowest MAE on METR-LA (2.88) and PEMS07 (19.49). STEP is lowest on PEMS-BAY (1.48), and STEP and D2STGNN tie on PEMS04 (18.32) §6.4 (STF table).
• On LTSF datasets without a clear pattern, advanced Transformers and linear models perform comparably §6.4.
• On M4, NBeats has the best weighted average in sMAPE, MASE and OWA §6.4 (M4 table). See Pitfalls for the text claim about STID and PatchTST.

STF, MAE, horizon 12 §6.4 (STF table). Lower is better. The paper marks the best values in bold and underline; those markings are not reproduced here.

LTSF, MAE, horizon 336 §6.4 (LTSF table). The paper reports the best of the four history lengths §6.1. The table does not say which length each cell uses.

Advanced vs basic models §6.2 (effectiveness table). MAE, horizon not restated in the table. Advanced: Informer, Autoformer, FEDformer. Basic: Linear, DLinear, NLinear. The "Gap" row is as printed; the text describes it as the performance gap between the best advanced and the best basic model.

Reading the Gap row against the values: on PEMS04 and PEMS08 the advanced model is better, and on ETTh2 and ETTm2 the basic model is better, which matches the text §6.2.

Spatial identity and GCN ablation §6.3 (GCN table). MAE, with and without the spatial component (STID*, AGCRN*). The "Gap" column is as printed.

M4 §6.4 (M4 table). Weighted average over all subsets. The paper also reports the Yearly, Quarterly, Monthly and Others subsets.

Normalized vs re-normalized evaluation §4.2 (evaluation table). ETTh1 and ETTh2. The table does not state the horizon:

Datasets §6.1 (datasets table, appendix). Samples, variates, frequency, span, graph.

Paper vs code

Code: GestaltCogTeam/BasicTS at c2bb6e31e591167e84459775a21a62e70a5893ce (Apache-2.0). The snapshot is later than the paper, and it contains model definitions and demo scripts, but not the per-model configurations behind the paper's tables. So the paper's exact optimizer settings could not be checked.

• Evaluation scale. The paper evaluates on re-normalized data §4.2. In the code, rescale defaults to False tsf_config.py, and inverse transform runs only when rescale is set forecasting_taskflow.py. The loss is computed before the inverse transform, on normalized values, in both training and validation basicts_runner.py, basicts_runner.py. Users must set rescale=True to match the paper's numbers.
• Loss. Paper: masked MAE by default §4.1. Code: default loss is MAE tsf_config.py, which is masked by the null value NaN tsf_config.py. Consistent in effect.
• WAPE. Paper: one global ratio, sum of absolute errors over sum of absolute targets eq. 1. Code: a per-batch-element ratio, summed over dimension 1 with a small epsilon, then averaged wape.py. This can give different values from the paper's formula.
• MAPE. Paper: plain formula eq. 1. Code: targets with absolute value below 5e-5 are dropped mape.py, and the mean is taken over the rest. Results on data with many near-zero values are therefore not comparable to a literal reading of eq. 1.
• Indistinguishability. Paper: e_l = 0.5 §5.2. The repository notebook uses e_l = 0.4 indistinguishability.ipynb, computes only the ratio r2 indistinguishability.ipynb, and uses channel 0 only indistinguishability.ipynb. The notebook does not compute r1.
• Gradient clipping and curriculum learning. The paper says the runner integrates both §4.1. In the code, both are optional callbacks, and nothing enables them by default clip_grad.py, curriculum_learrning.py.
• Seeds and demo settings. The paper states no seed. Code default seed is 42 tsf_config.py. The forecasting demo uses seed 233, 100 epochs, learning rate 5e-4 and a MultiStepLR schedule, but for iTransformer on ETTh1, not for the paper's models forecasting_demo.py.

Pitfalls

• Re-normalized evaluation is off by default in the code. With default settings the code reports metrics on normalized values. Set rescale=True to match the paper's evaluation.
• Text vs table disagreement on D2STGNN. The text says its METR-LA MAE is "only 6% higher" than Graph WaveNet's §6.4. The table gives 2.88 vs 3.03, so D2STGNN is about 5% lower §6.4 (STF table). Treat the text claim as likely misprinted.
• Text vs table disagreement on M4. The text says PatchTST and STID perform worse than classic algorithms on M4 §6.4. In the weighted sMAPE row, PatchTST (12.324) and STID (11.755) are ahead of LGBM (13.430) and DeepAR (13.323), and both trail NBeats (11.508). The claim holds against NBeats only.
• Dataset count. The text says 20 datasets and 45+ models §1 and the abstract, but the experiments use 14 listed datasets §6.1. ETTh2 and ETTm2 appear only in the advanced-vs-basic table §6.2. The experiment section says not all datasets are shown.
• Selection of history length. The paper reports the best of four history lengths per model §6.1. It does not say whether this choice used a validation split. Reported LTSF numbers may therefore be optimistic.
• Tuning budget. Each baseline gets a grid search per dataset to reach at least its published performance §6.1. The grid is not stated, so the tuning effort behind each row is unknown.
• Seeds and variance. The paper does not state seeds, repeats or variance. Small differences in the tables cannot be judged as significant from the paper alone.
• Heterogeneity labels are visual. The temporal groups come from plots and t-SNE views, not from a metric §5.1. The paper says a metric for predictability is unsolved. Use the labels as guidance for dataset choice.
• Spatial indistinguishability thresholds are fixed. e_u = 0.9 and e_l = 0.5 are chosen by the authors §5.2, and the code uses 0.4 for e_l. Results depend on these numbers, and the paper gives no sensitivity analysis.
• Normalization changes the scale of the numbers. For the same ETT models, the normalized MAE is about 0.4 to 0.5 and the re-normalized MAE is 1.6 to 3.7, with MAPE of 55% to 70% §4.2 (evaluation table). Always state the scale with a number.
• Hidden costs. The efficiency columns give seconds per epoch. STEP takes 497.26 s per epoch on METR-LA and 1406.90 s on PEMS-BAY §6.4 (STF table). Accuracy-only rankings hide this.
• Undefined items for re-implementers. The paper does not state per-model learning rates, schedules, epochs, early-stopping rules, clipping norms, curriculum parameters or seeds §6.1. To reproduce the tables, use the released code and its configs, and treat any missing value as unknown.

Lineage

• Builds on: the existing MTS forecasting baselines it re-runs, including DCRNN, Graph WaveNet, STID, AGCRN, Informer, Autoformer, FEDformer, PatchTST, Crossformer and the linear models (DLinear, NLinear) §6.1. The paper adds no new model.
• Compared against: more than 45 MTS forecasting solutions [abstract]. The printed tables show 15 STF models, 13 LTSF models and 5 M4 models §6.4.
• Superseded by: not known from content in this wiki.