Gated Transformer Networks for Multivariate Time Series Classification

In plain words

Many recordings come from several sensors at once and change over time, such as physiological signals, financial records or machine readings. Classifying such a recording into one label is harder than classifying a single signal, because the sensors are related to each other as well as to their own past.

The paper adapts the Transformer, an attention-based network, to this task. The model runs two attention stacks side by side: one compares moments in time, the other compares sensors. A small learned gate decides how much weight each of the two views gets, and the weighted features go to a classifier. The authors report accuracy that is comparable to convolutional and recurrent baselines on 13 benchmark datasets, though not better on all of them. Those numbers are the authors' own, and the Pitfalls section explains how they were selected.

TL;DR

Gated Transformer Networks (GTN) classify a multivariate series with two Transformer-encoder towers: a step-wise tower that attends across time steps and a channel-wise tower that attends across channels. A softmax gate computed from both towers weights them before a linear classifier. The paper reports test accuracy comparable to FCN and ResNet on 13 datasets. The released code selects checkpoints on test accuracy and differs from the paper in several details (see Paper vs code), so treat the numbers as optimistic and compare on a fixed split.

Problem

Multivariate series classification must map a whole series, with several channels per time step, to one label. Physiological, financial and sensor data are cited as examples §1. Distance measures with nearest neighbours, such as dynamic time warping, were long the reference §1. Deep models that reached strong results were convolutional (FCN, ResNet, MCNN and others) or recurrent §2. Transformers had been studied for time series mainly for forecasting and regression, with only a few classification studies §2. The paper asks whether a Transformer extended to multivariate input can match those baselines §1.

Method

The input is a series of T time steps, each with C channel values §3. Series are zero-padded to a common length before the model sees them [code: dataset_process.py#L86-L95]. The series passes through two towers §3.2 fig. 1:

• Step-wise tower. Each time step is embedded as a d-dimensional token. A fixed sinusoidal positional encoding is added, and N encoder blocks attend across the T tokens §3.1 §3.2. This tower uses a causal mask during training multiHeadAttention.py.
• Channel-wise tower. The series is transposed so that each channel is one token of length T. Each channel token is embedded to d dimensions, and N encoder blocks attend across the C tokens. The paper gives no positional encoding here §3.2.

Each tower's output is flattened. The paper combines the two flattened vectors with a gate §3.3 eq. 1: a linear layer on their concatenation gives two logits, a softmax turns them into weights g1 and g2, and each tower is scaled by its weight before the two are concatenated. A final linear layer produces the class scores transformer.py.

Each encoder block is a post-norm Transformer block §3.2 [code: encoder.py#L126-L134]: multi-head scaled dot-product self-attention, dropout, residual addition and layer normalisation, then a position-wise feed-forward network, dropout, residual addition and layer normalisation. The paper names these parts but not the feed-forward activation; the code uses ReLU [code: feedForward.py#L66-L71].

Gating, as the paper's only numbered equation block eq. 1:

h = W \cdot \mathrm{Concat}(C, S) + b g_1, g_2 = \mathrm{Softmax}(h) y = \mathrm{Concat}(C \cdot g_1, S \cdot g_2)

The equation names the tower outputs C and S without saying which is which. The code applies the first weight to the step-wise tower transformer.py.

Training uses cross-entropy on the classifier output, with Adagrad at learning rate 1e-4 and dropout 0.2 §4.1. The released training loop is shown below.

# Training loop  [§4.1] [code: run.py#L117-L146]
# Input: train loader (series, label), test loader
# Output: test accuracy at the best evaluation point

def train(net, train_loader, test_loader, epochs):
    # Adagrad, constant learning rate 1e-4  [code: run.py#L45, L59, L83]
    optimizer = Adagrad(net.parameters(), lr=1e-4)
    best = 0
    for epoch in range(1, epochs + 1):
        for series, label in train_loader:
            # Cross-entropy on the raw class scores  [code: loss.py#L79-L85]
            loss = cross_entropy(net(series, stage="train"), label)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        # Every 5 epochs: evaluate, keep best test accuracy
        # [code: run.py#L137-L144]
        if epoch % 5 == 0:
            acc = accuracy(net, test_loader)
            best = max(best, acc)
    return best

Only the step tower sees time order: through the sinusoidal encoding and, in training, the causal mask. The channel tower has neither §3.2.

Pseudocode for the forward pass of the two towers and the gate:

# Gated Transformer forward pass  [§3] [fig. 1] [eq. 1]
# Input: series (B, T, C): T time steps, C channels
# Output: class scores (B, K)

def gtn_forward(series):
    # Step-wise tower: one token per time step
    # [§3.1] [code: transformer.py#L62]
    steps = linear_embed_channels(series)                  # (B, T, d)
    # Add a fixed sinusoidal positional encoding, step tower only  [§3.1]
    steps = steps + sinusoidal_encoding(T, d)              # (B, T, d)
    for block in step_blocks:                              # N blocks  [§3.2]
        # Causal mask in training only  [code: multiHeadAttention.py#L40]
        steps = block(steps, causal=is_training)           # (B, T, d)
    # Channel-wise tower: one token per channel
    # [§3.2] [code: transformer.py#L82]
    channels = linear_embed_steps(transpose(series))       # (B, C, d)
    for block in channel_blocks:                           # N blocks, no mask
        channels = block(channels, causal=False)           # (B, C, d)
    # Flatten each tower  [code: transformer.py#L89-L90]
    step_vec = reshape(steps, (B, T * d))
    channel_vec = reshape(channels, (B, C * d))
    # Gate: one softmax over the two towers
    # [§3.3] [eq. 1] [code: transformer.py#L93]
    both = concat(step_vec, channel_vec)                   # (B, (T + C) * d)
    gate = softmax(linear_gate(both))                      # (B, 2)
    # Scale each tower by its gate weight, then concatenate
    # [code: transformer.py#L94]
    gated = concat(step_vec * gate[:, 0:1], channel_vec * gate[:, 1:2])
    # Linear classifier on the gated features  [code: transformer.py#L97]
    return linear_out(gated)                               # (B, K)

Multi-head attention inside each block, from the released code:

# Multi-head scaled dot-product self-attention
# [code: multiHeadAttention.py#L18-L51]
# Input: x (B, L, d); h heads; per-head query/key size q; value size v
# Output: (B, L, d)

def attention(x, causal):
    # Linear projections; heads stacked on the batch axis
    # [code: multiHeadAttention.py#L33-L35]
    queries = split_heads(W_q(x))                          # (B*h, L, q)
    keys = split_heads(W_k(x))                             # (B*h, L, q)
    values = split_heads(W_v(x))                           # (B*h, L, v)
    # Scaled dot-product scores, divided by sqrt(q)
    # [code: multiHeadAttention.py#L37]
    scores = queries @ transpose(keys) / sqrt(q)           # (B*h, L, L)
    if causal:
        # Keep only scores at or before the current position
        # [code: multiHeadAttention.py#L40-L43]
        scores = where(lower_triangular(L), scores, -2**32 + 1)
    weights = softmax(scores)                              # (B*h, L, L)
    heads = weights @ values                               # (B*h, L, v)
    # Undo the head stacking and project back
    # [code: multiHeadAttention.py#L49-L51]
    return W_o(merge_heads(heads))                         # (B, L, d)

Hyperparameters

Paper values are marked §4.1. Code values come from the official repository at code_commit; the paper states none of the architecture sizes.

Results

Author-reported test accuracy (%) tab. 1. Setup: 13 datasets from a multivariate archive, default train/test split, no preprocessing §4.1. Baselines: MLP, FCN, ResNet, Encoder, MCNN, t-LeNet, MCDCNN, Time-CNN and TWIESN §4.1. Hardware, number of runs and variance are not stated. The paper does not name a significance test §4.2. Table 1 labels the dataset "CMUsubject1"; the ablation and the text write "CMUsubject16" tab. 2 §4.2.1.

Ablation tab. 2, test accuracy (%). Columns: step-wise tower only; step-wise with mask; channel-wise only; channel-wise with mask; both towers concatenated; GTN (both towers with gating).

Other reported results, all author-reported:

• GTN is described as comparable to FCN and ResNet, with no significant difference among the three, and as better on NetFlow and KickvsPunch §4.2.
• On AUSLAN the average gate weights are about 0.78 for the step-wise tower and 0.22 for the channel-wise tower, varying per sample §4.4.
• Attention maps on JapaneseVowels are discussed qualitatively. The authors find the link to DTW or Euclidean distance weak and leave it to future work §4.3 fig. 2 fig. 3.
• t-SNE of the embedding layer on AUSLAN forms clusters that the authors map to shapelet-like patterns §4.5 fig. 4 fig. 5. The features after gating on JapaneseVowels look more separable in t-SNE §4.6 fig. 6.

Paper vs code

Official repository ZZUFaceBookDL/GTN at code_commit, linked from the paper footnote §4.1. The repository has no LICENSE file, so it is cited only, not copied. Differences, each with both citations:

1. Embedding activation. Paper: a non-linear tanh follows the embedding §3.1. Code: plain linear layers with no activation [code: transformer.py#L43-L44, L62, L82].
2. Pre-gate layer. Paper: each tower output goes through a fully connected layer with a non-linear activation before the gate §3.3. Code: the flattened encoder outputs go straight into the gate [code: transformer.py#L88-L94].
3. Channel masking. Paper's method text says the channel encoder uses masking §3.2. Code: channel encoders are built without a mask, and the mask flag applies to the step tower only [code: transformer.py#L31, L35-L41, L81; run.py#L57].
4. Learning-rate schedule. Paper: a reduce-on-plateau schedule is applied §4.1. Code: the optimiser is created with a fixed rate and no scheduler run.py. No scheduler appears anywhere in the repository.
5. Model selection. Paper: the best test results are recorded, and the reported accuracy is taken from the model with the best training loss §4.1. Code: evaluation every 5 epochs on the test set, saving the checkpoint when test accuracy improves and naming the file with the best test accuracy [code: run.py#L137-L149]. The paper's two statements disagree with each other.
6. Dropout stays off after the first evaluation. test() calls net.eval() run.py, and train() calls net.train() only once, before the epoch loop run.py. After the first evaluation at epoch 5, the loop trains with dropout disabled. The paper does not mention this.
7. Masking at evaluation. Paper credits the mask with keeping attention away from padding §4.2.1. Code: the causal mask is applied only when stage == 'train' multiHeadAttention.py, and evaluation runs net(x, 'test') without it run.py.
8. Preprocessing. Paper: no preprocessing §4.1. Code: zero-padding to the longest series, computed over train and test [code: dataset_process.py#L86-L95, L107-L117].
9. Baseline values. The README's table differs from paper Table 1 in places, for example UWave Time-CNN is 8.9 there and 85.9 in the paper [code: README.md#L17] tab. 1.

Pitfalls

• Reported accuracy is test-selected. The released code takes the best of the test accuracies it logs every 5 epochs run.py. A reimplementation needs a validation split for selection, or it cannot compare fairly with these numbers. The paper's "best training loss" wording §4.1 does not match the code.
• Dropout is disabled after epoch 5 in the released loop (Paper vs code, item 6). Replicating the paper's dropout 0.2 literally means calling net.train() before each epoch.
• Train and evaluation differ in attention. Training uses the causal mask and evaluation does not multiHeadAttention.py. At test time, real time steps can attend to the trailing zero padding of shorter series. The paper does not discuss this.
• Fixed input shape. The channel tower's embedding maps length T to d [code: transformer.py#L44], and the gate and output layers take (T + C) × d inputs transformer.py. With the code's defaults (d = 512), AUSLAN's T = 136 and C = 22 give 158 × 512 = 80,896 inputs to each of those layers (derived from the code). Changing the length means retraining.
• Padding length depends on the test set. The common length is the maximum over train and test together dataset_process.py.
• The paper's dropout description is incomplete. The attention-weight dropout is defined multiHeadAttention.py but never applied, so dropout appears only after each sub-layer encoder.py.
• Gate granularity. The gate is one scalar per tower and sample, applied to every feature of that tower transformer.py.
• Attention cost. The step tower builds a (B·h, T, T) score matrix and the channel tower a (B·h, C, C) matrix at each block [code: multiHeadAttention.py#L37]. Memory grows with the square of T and of C.
• Single seed. The released run sets seed 30 once [code: run.py#L20; random_seed.py#L136-L140]. Numbers of runs or variance are not reported.
• Other configurations. The repository's result-figure file names record runs with Adam, batch sizes 3, 4 and 20, 10 to 100 epochs, and dropout 0 or 0.2 . The paper states Adagrad only, so which configuration produced Table 1 is not stated.
• NetFlow labels. The loader subtracts 1 from every label [code: dataset_process.py#L32], but the code comments say NetFlow's labels (1 and 13) need a change run_with_saved_model.py. Check the label mapping before using NetFlow.
• Windows-specific code. The training script hard-codes a Windows dataset path and splits file names on backslashes run.py. It also writes into saved_model/, which is not in the repository [code: run.py#L144]. Create the directory and edit paths before running on another system.
• Statistical claim. "No significant difference" is stated without a named test §4.2.

Lineage

• Builds on: the Transformer encoder with sinusoidal positional encoding, as cited in the paper's introduction and method §1 §3.1. Not compiled here.
• Own contribution: the two-tower design (step-wise and channel-wise) and the softmax gate §3.2 §3.3.
• Compared against: MLP, FCN, ResNet, Encoder, MCNN, t-LeNet, MCDCNN, Time-CNN and TWIESN §4.1. Related transformer work for time series, including classification on satellite and clinical data, is discussed in §2.