MSGNet: Learning Multi-Scale Inter-Series Correlations for Multivariate Time Series Forecasting

In plain words

Many measurements arrive as several time series at once, such as flights per airport, or power use per household. Each series rises and falls on its own, and series also move together. How strongly two series move together can depend on the time scale being looked at. Many forecasting models treat each series alone, or assume one fixed pattern of which series relate to which.

MSGNet first looks for the rhythms in the data, such as a daily or a half-day cycle. For each rhythm it asks a separate question: which series move together at this speed? Each rhythm gets its own learned map of relationships between series, and the network mixes information along those links. Within each rhythm it also looks at how a series changes from one cycle to the next. The results from the different rhythms are then combined, with the strongest rhythms counting most.

The authors test on eight real datasets, one of them flight counts from the COVID-19 period, and report that the model does better than six comparison methods on average. These are the authors' own numbers, and they come from their own experiment setup.

TL;DR

• MSGNet predicts several series at once. It picks the k strongest periods of the input with an FFT and handles each period as one scale §4.2 eq. 3.
• Each scale gets its own learned adjacency matrix and a Mixhop graph convolution, followed by self-attention within the period, and the scales are summed with softmax weights from their amplitudes §4.3 §4.4 eq. 9.
• Author-reported: best average rank on the 8 datasets (1.813 against 2.750 for TimesNet), and the best average MSE on 5 datasets tab. 1 §5.4.
• The official code differs from the paper in several places that change the computation: the scale graph convolutions are chained, the within-period attention is causally masked, and the scales are re-estimated in every block. See Paper vs code before implementing.

Problem

• A multivariate forecast has to use both the series' own history and how series relate to each other §1 §3.
• Existing deep models were said to capture intra-series patterns well but to learn inter-series relations poorly. Graph models were said to use one fixed relation for all times §1 §2.2 app. §2.
• The paper argues that inter-series correlations differ between time scales, so one graph is not enough §1 §3 fig. 1.
• The Flight data cover the COVID-19 period. The authors use it as an out-of-distribution test of robustness §5.1 §5.7.
• Assumptions: input window L = 96 and horizons T in {96, 192, 336, 720} §5.3 tab. 4. Training uses MSE §5.3 tab. 5.

Method

Numbering note: the paper's body sections print without numbers (secnumdepth is 0). The § numbers here count them in order: 1 Introduction; 2 Related Works (2.1, 2.2); 3 Problem Formulation; 4 Methodology (4.1 Input embedding and residual connection, 4.2 Scale identification, 4.3 Multi-scale adaptive graph convolution, 4.4 Multi-head attention and scale aggregation, 4.5 Output layer); 5 Experiments (5.1 Datasets, 5.2 Baselines, 5.3 Experimental setups, 5.4 Results and analysis, 5.5 Visualization of learned inter-series correlation, 5.6 Ablation analysis, 5.7 Generalization capabilities); 6 Conclusion. The appendix numbers its sections in the paper itself, so app. §3.2 is unambiguous.

Each of the k ScaleGraph blocks does four things in order §4:

1. Scale identification. The FFT of the embedded input gives an amplitude per frequency. The paper averages these over the model dimension, takes the top k frequencies f_i, and sets the period s_i = L / f_i §4.2 eq. 3.
2. Reshape. The series is zero-padded and reshaped into a grid of shape (d_model, s_i, f_i) for each scale §4.2 eq. 4.
3. Adaptive graph convolution. A linear map W^i takes the d_model channels to N variables §4.3 eq. 5. The adjacency is A^i = softmax(ReLU(E1 E2^T)), with two trainable N x h matrices §4.3 eq. 6. Mixhop convolution over the powers of A^i is concatenated and passed through an MLP back to (d_model, s_i, f_i) §4.3 eq. 7.
4. Within-scale attention and aggregation. Multi-head self-attention runs over the s_i positions of each scale §4.4 eq. 8. The k results are reshaped back to length L and summed with softmax weights over the amplitudes of the k frequencies §4.4 eq. 9.

The network is a stack of these blocks with residual links §4.1 eq. 2. The input is embedded with a 1-D convolution of kernel 3 plus a positional term and a time-stamp term §4.1 eq. 1. The output layer maps the d_model channels to N variables and then the L time steps to T steps §4.5 eq. 10. The input is normalised before embedding §4.1. The paper does not describe the de-normalisation step; the code has one (see Paper vs code).

Equations as printed in the paper:

\mathbf{X}_{emb} = \alpha\,\mathrm{Conv1D}(\hat{\mathbf{X}}) + \mathbf{PE} + \sum_{p=1}^{P} \mathbf{SE}_p eq. 1

\mathbf{X}^{l} = \mathrm{ScaleGraphBlock}(\mathbf{X}^{l-1}) + \mathbf{X}^{l-1} eq. 2

\mathbf{F} = \mathrm{Avg}(\mathrm{Amp}(\mathrm{FFT}(\mathbf{X}_{emb}))),\quad f_1,\dots,f_k = \mathrm{argTopk}(\mathbf{F}),\quad s_i = L / f_i eq. 3

\mathcal{X}^i = \mathrm{Reshape}_{s_i, f_i}(\mathrm{Padding}(\mathbf{X}_{in})) eq. 4

\mathcal{H}^i = \mathbf{W}^i \mathcal{X}^i eq. 5

\mathbf{A}^i = \mathrm{SoftMax}(\mathrm{ReLU}(\mathbf{E}^i_1 (\mathbf{E}^i_2)^T)) eq. 6

\mathcal{H}^i_{out} = \sigma\Big(\big\Vert_{j \in \mathcal{P}} (\mathbf{A}^i)^j \mathcal{H}^i\Big) eq. 7

\hat{\mathcal{X}}^i_{out} = \mathrm{MHA}_s(\hat{\mathcal{X}}^i) eq. 8

\hat{a}_1,\dots,\hat{a}_k = \mathrm{SoftMax}(\mathbf{F}_{f_1},\dots,\mathbf{F}_{f_k}),\quad \hat{\mathbf{X}}_{out} = \sum_{i=1}^{k} \hat{a}_i \hat{\mathbf{X}}^i_{out} eq. 9

\hat{\mathbf{X}}_{t:t+T} = \mathbf{W}_s \hat{\mathbf{X}}_{out} \mathbf{W}_t + \mathbf{b} eq. 10

The appendix gives a mixture-of-experts reading of the scale aggregation app. §1.2, and a two-hop-delta argument for why graph convolution is needed app. §2. Both are analysis, not extra steps.

# MSGNet forward pass, as the paper describes it  [§4] [eq. 1]-[eq. 10]
# Input:  x_enc (B, L, N), L = 96 past steps of N variables
# Output: forecast (B, T, N)

def msgnet_forward(x_enc):
    # Normalise the input (details not stated in the paper)  [§4.1]
    x_norm = normalise(x_enc)                       # (B, L, N)
    # Embed the variables and add time terms  [§4.1] [eq. 1]
    x = embed(x_norm)                               # (B, L, d_model)
    # Stack the ScaleGraph blocks with residual links  [§4.1] [eq. 2]
    for layer in range(num_blocks):
        x = scale_graph_block(x) + x                # (B, L, d_model)
    # Map channels to N variables, then steps to T  [§4.5] [eq. 10]
    out = linear_over_channels(x)                   # (B, L, N)
    out = linear_over_time(out)                     # (B, T, N)
    return out


def scale_graph_block(x):
    # Pick the k strongest periods of the input  [§4.2] [eq. 3]
    periods, amps = top_k_periods(x, k)             # (k,), (k,)
    # Softmax over the k amplitudes gives the scale weights  [§4.4] [eq. 9]
    weights = softmax(amps)                         # (k,)
    outputs = []
    for i in range(k):
        # Reshape into a grid with period periods[i]  [§4.2] [eq. 4]
        grid = reshape_with_padding(x, periods[i])  # (d_model, s_i, f_i)
        # Project d_model channels to N variables  [§4.3] [eq. 5]
        nodes = W[i] @ grid                         # (N, s_i, f_i)
        # Learn the adjacency for this scale  [§4.3] [eq. 6]
        adj = softmax(relu(E1[i] @ E2[i].T))       # (N, N)
        # Mix neighbour information over adjacency powers  [§4.3] [eq. 7]
        mixed = concat_powers(adj, nodes, P)       # (len(P) * N, s_i, f_i)
        # Map back to d_model channels with an MLP  [§4.3]
        grid = mlp(mixed)                           # (d_model, s_i, f_i)
        # Self-attention over the s_i positions of each period  [§4.4] [eq. 8]
        grid = attention_over_scale(grid)           # (d_model, s_i, f_i)
        # Undo the reshape and keep the first L steps  [§4.4]
        outputs.append(unreshape(grid, L))          # (B, L, d_model)
    # Sum the scales, weighted by amplitude softmax  [§4.4] [eq. 9]
    return sum(weights[i] * outputs[i] for i in range(k))

Hyperparameters

Values are as printed in the paper. Where the code sets a value the paper does not state, the code value is shown and marked.

Results

Author-reported. Setup: 8 datasets, input length 96, horizons {96, 192, 336, 720}, MSE training loss, RTX 3090 §5.3 tab. 4. Baselines: Informer, Autoformer, MTGNN, DLinear, NLinear, TimesNet §5.2. The paper states that MSGNet is best on average MSE on 5 datasets and second-best on 2, and that it has the best average rank across all datasets §5.4 tab. 1.

Main results, MSE and MAE, against TimesNet, the second-ranked model tab. 1. The table lists all six baselines; this card includes only TimesNet.

Average rank across all datasets, MSGNet 1.813, TimesNet 2.750, NLinear 2.813, DLinear 3.563, Autoformer 4.750, MTGNN 5.313, Informer 7.000 tab. 1.

On Flight, the paper reports the average over horizons reduced from 0.265 to 0.208 MSE (21.5%) and from 0.372 to 0.321 MAE (13.7%) against TimesNet §5.4.

Ablation on Flight, Weather and ETTm2, averaged over horizons 96 and 336 tab. 2:

Generalization, Flight under two partitions tab. 3. MSE and MAE are averaged over horizons. "Decrease" is the relative error increase when the training set is limited to before the pandemic:

Efficiency on Electricity, batch size 32 tab. 7. Memory is in GB, time per epoch in seconds, and the MSE rank among the four models in the table:

Other results are described in words only. The learned scales on Flight are said to correspond to patterns of one day, half a day and a morning fig. 8 app. §4. The paper reports that MSGNet improves on ETT with longer review windows fig. 10 app. §5. Flight prediction plots are in fig. 3 and fig. 11.

Paper vs code

Official code: https://github.com/YoZhibo/MSGNet, commit 953b8330a2ca469dab4955e804b46a61eb08a9c2 (default branch, shallow clone). The paper does not link it. The repository's README cites this paper README.md README.md README.md, and the GitHub profile of the owner account (YoZhibo) shows the first author's name and Sichuan University as the company field. I attribute the repository to the authors on that basis; the paper's own affiliation was not checked. The commit read here is the default branch at clone time, which may differ from the code used for the paper's numbers.

The repository has no LICENSE file, so no code is quoted and every reference is by path and line. The code was read but not run: Flight.csv and the other datasets are not in the repository Flight.sh.

Discrepancies that change the computation:

• Scale graph convolutions are chained. Paper: each scale's graph convolution starts from the same input, X_in §4.2 §4.3 eq. 4. Code: the loop reassigns x after each graph convolution, so scale i works on the output of scale i-1 MSGNet.py.
• Residual base. Paper: X^l = block(X^{l-1}) + X^{l-1} §4.1 eq. 2. Code: the residual adds the tensor left after all k graph convolutions, not the block input MSGNet.py. A LayerNorm is also applied after every block MSGNet.py. The paper has none, and the same LayerNorm module is reused for all blocks.
• Scale identification happens in every block. Paper: the scales are chosen once, from X_emb §4.2 eq. 3. Code: each block runs the FFT on its own input MSGNet.py.
• Scale selection. Paper: amplitudes averaged over the model dimension §4.2. Code: averaged over the batch and the model dimension, and the DC bin is set to zero before the top-k MSGNet.py. The softmax weights are per sample MSGNet.py MSGNet.py.
• Within-scale attention is a full transformer layer. Paper: multi-head self-attention only §4.4 eq. 8. Code: attention, residual, LayerNorm, a two-layer 1x1 convolution feed-forward with GELU, and a second LayerNorm MSGBlock.py, followed by a GELU before aggregation MSGNet.py.
• Within-scale attention is causally masked. The paper states no mask. FullAttention defaults to mask_flag=True and builds an upper-triangular mask when none is passed MSGBlock.py. The block calls it with no mask MSGBlock.py MSGNet.py.
• Node projection. Paper: a dense W^i of shape N x d_model §4.3 eq. 5. Code: a 2-D convolution whose kernel spans d_model - N + 1 channels, so the same weights are shared along the channel axis MSGBlock.py. This needs d_model >= N.
• Mixhop. Paper: the powers (A^i)^j for j in a set P, concatenated §4.3 eq. 7. Code: a propagation h_k = alpha x + (1 - alpha) A h_{k-1} with alpha = 0.3 by default, after adding a self-loop and row-normalising A, then the concatenation of x and the h_k MSGBlock.py MSGBlock.py. The set P is not stated in the paper.
• Graph block residual. The graph block ends with its own residual and LayerNorm MSGBlock.py. The paper does not mention either.
• Adjacency. Matches eq. 6, with a softmax over the rows [code: layers/MSGBlock.py#L145]. Each scale has its own node vectors [code: layers/MSGBlock.py#L133-L134].
• Embedding. Paper: alpha-scaled convolution, a positional term, and a sum of learned time-stamp embeddings with a vocabulary of 60 minutes §4.1 eq. 1. Code: a kernel-3 circular convolution without bias Embed.py, a fixed sinusoidal positional term Embed.py, and a linear map on time features Embed.py. The sum is in Embed.py. No alpha is applied.
• Normalisation. Paper: input normalised §4.1. Code: instance normalisation with mean and standard deviation over time, reversed on the output MSGNet.py MSGNet.py.
• Output. Matches eq. 10 in form MSGNet.py. The predict_linear layer is defined and never used MSGNet.py.
• Dropout. gcn_dropout is parsed with default 0.3 run_longExp.py but no code path reads it. The Mixhop module stores a dropout value and never applies it MSGBlock.py MSGBlock.py.
• Hyperparameters. The paper's text gives E as {10, 100}, its table as {10, 30, 100} app. §3.2 tab. 5. The scripts use 30 for Exchange exchange.sh and 100 for Flight and Electricity. Read the table as the search range and the scripts as the values used.
• Learning rate. Paper: 1e-4 §5.3. Code: halved every epoch tools.py exp_main.py.
• Metrics on the standardised scale. The test function computes MSE and MAE on the model output with no inverse transform exp_main.py. The paper does not say which scale its numbers use. The data are standardised with training statistics data_loader.py.
• Model selection. The test loader is evaluated every epoch [code: exp/exp_main.py#L186]. The paper does not say how its reported checkpoint was chosen.
• Consistent with the paper: Adam exp_main.py, MSE loss exp_main.py, learning rate 1e-4 run_longExp.py, batch size 32 run_longExp.py, 10 epochs run_longExp.py, eight heads by default run_longExp.py, Mixhop depth 2 run_longExp.py, and sequence length 96 run_longExp.py.

Pitfalls

• Causal mask. The within-period attention is causal in the code and the paper says nothing. Decide explicitly. An unmasked version will not match the reported numbers if the code produced them MSGBlock.py.
• Chained scales. Implementing the scales in parallel, as the equations read, gives a different model from the released one MSGNet.py.
• Flight data is not in the repository. The Flight dataset comes from OpenSky data for seven European airports, but the preprocessing that turns it into Flight.csv is not described in the repository §5.1 app. §3.1 Flight.sh. Results on Flight cannot be reproduced from the repository alone.
• Split labels in the dataset table. Table 4 lists sizes as train / test / valid, but the numbers follow train / valid / test at 70:10:20 for Flight, Weather, Electricity and Exchange tab. 4 [code: data_provider/data_loader.py#L154-L158].
• Efficiency claim. The paper says MSGNet is more efficient than TimesNet app. §6. Table 7 shows lower time per epoch at every horizon, but higher GPU memory than TimesNet at 96, 192 and 336 tab. 7.
• Figure references. The appendix text refers to "Figure 1 and 2 of Appendix" for the flight data. In the compiled order these are fig. 5 and fig. 6. The caption of fig. 12 says ETTm2 while the file is named for ETTh2.
• Ablation averaging. The ablation table averages horizons 96 and 336 only tab. 2. The text says multi-head attention gives "marginal" gains, and w/o-A is only slightly worse than full MSGNet on Flight §5.6 tab. 2.
• Generalization claim. Table 3 has two partitions and one Flight dataset. MSGNet has the smallest relative increase in MSE and MAE, but the margin over TimesNet is small (21.29% against 26.47% in MSE) tab. 3.
• Longer review windows. On Flight with a 336-step horizon, the text says MSGNet is slightly worse than the linear model at long windows, but better than the other models app. §5 fig. 9. The ETT result with longer windows is the one the paper points to for the scale transformation app. §5 fig. 10.
• Costs the abstract hides. Memory grows with horizon: 13.04 GB at 96 and 16.98 GB at 720 on Electricity, against 8.81 and 20.22 GB for TimesNet tab. 7. Time per epoch at horizon 96 is 320 s for MSGNet, 749 s for TimesNet and 4.4 s for DLinear tab. 7.
• Unstated values. Dropout, alpha, the Mixhop power set P, the feed-forward width, the number of runs and the seed are not stated in the paper. The code defaults are listed in Hyperparameters and should not be read as the paper's values.

Lineage

• Builds on: Mixhop graph convolution §4.3 [/papers/1905.00067]; the FFT-based period detection of TimesNet §4.2 (TimesNet has no card in this wiki); the uniform input embedding of Informer §4.1 (no card); multi-head attention from the Transformer §4.4 (no card); a normalisation method the paper cites with bibliography key liu2022non §4.1 (no card).
• Compared against: Informer, Autoformer, MTGNN, DLinear, NLinear and TimesNet §5.2 tab. 1. None has a card in this wiki, so nothing from them is used.
• Superseded by: not in this wiki.