TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data

In plain words

Many industrial systems produce many sensor readings at once, and someone needs to notice when they stop behaving normally. Labelled failures are rare, so the usual approach is to learn what normal looks like and flag anything the model cannot reproduce. This paper applies that idea with a transformer, a neural network that looks at a whole stretch of the signal at once rather than one step at a time.

The model takes a short window of recent readings and tries to rebuild it. A reading that cannot be rebuilt well is suspicious. The model runs twice. The first pass makes a rough rebuild. The second pass is given the rough rebuild's error as an extra hint, so it concentrates on the parts that looked wrong. The second pass also sees the longer history of the signal, which helps it judge whether a short spike is part of a trend.

Training uses two output heads that push against each other, so that small deviations stand out more clearly. The authors also train the model with a meta-learning method intended to help when data is limited. They report that their model trains much faster than the baselines they compare with, and that it scores higher on F1 and AUC on the public benchmarks they use. Those comparisons are the authors' own.

TL;DR

• TranAD is a transformer-based detector and diagnoser for multivariate time series. It scores each timestamp by how badly the window around it can be reconstructed §3.5.
• Inference runs in two phases: a first reconstruction gives an error signal (the focus score) that conditions a second reconstruction §3.4 eq. 4 eq. 5.
• Training uses two decoders with a reconstruction loss and an adversarial term, plus MAML, with an offline and an online stage §3.4 alg. 1.
• Author-reported: average F1 of 0.8802 on the full training data (dataset count is inconsistent in the paper, see Pitfalls), up to 17.06% higher F1 than the state-of-the-art baselines, and training time 75%–99% lower §4.3 §5.2 tab. 2 tab. 5.
• An official BSD-3-Clause repository exists, but the released TranAD training loop omits the adversarial sign and the meta-learning step, and uses different encoder sizes and a fixed 5-epoch schedule. Read "Paper vs code" before reimplementing models.py.

Problem

Anomaly detection in multivariate time series is hard for four reasons: anomaly labels are scarce, the data is volatile, the detector must run with very low latency, and the detector must also say which dimensions caused an alarm §1. Recurrent models (LSTM-based) are slow to train and to run, and their fixed-size input windows limit how much long-term context they see §1 §2. Simple transformer encoder-decoders can miss anomalies whose deviation from normal is small §1 §3.4.

Training uses only the training series; the labels are used for evaluation on the test series §3.1 §4.1.

Method

The model is reconstruction-based. Each timestamp gets an anomaly score §3.5.

Problem definition §3.1. A multivariate series has m dimensions. Detection gives one 0/1 label per timestamp. Diagnosis gives one 0/1 label per dimension and timestamp §3.1.

Preprocessing §3.2.

1. Normalise each dimension with the minimum and maximum of the training series, plus a small constant, so the values fall roughly in [0, 1) §3.2 eq. 1.
2. Turn the series into sliding windows of length K ending at each timestamp, so each timestamp keeps its local context §3.2.
3. Pad the first windows by repeating a constant vector; the text repeats x_t §3.2 (see Pitfalls).
4. Train and test on windows, not single vectors §3.2.

Architecture §3.3. Dot-product attention scales the scores by \sqrt{m} §3.3 eq. 2. Multi-head attention uses h heads, which the paper sets equal to m §3.3 eq. 3 §4. The first encoder takes the window concatenated with the focus score, adds position encoding, and encodes the sequence up to the current timestamp §3.3 eq. 4. The window encoder applies masked self-attention to the window and then attends over the first encoder's output §3.3 eq. 5. Two identical decoders, each ending in a sigmoid, reconstruct the window §3.3 eq. 6. fig. 1 shows the architecture: the encoders, the window encoder and the two decoders.

Two-phase inference §3.4.

• Phase 1 runs with a zero focus score. Decoder 1 gives O_1 and decoder 2 gives O_2 §3.4 alg. 1 line 5.
• The focus score is the deviation between the input window and O_1, measured by an L2 norm. Phase 2 feeds this focus score back into the encoder and decoder 2 produces \hat{O}_2 §3.4 alg. 1 line 6. The paper calls this self-conditioning §3.4.

Adversarial two-decoder loss §3.4. Phase 1 losses are L_1 = \|O_1 - W\| and L_2 = \|O_2 - W\| §3.4 eq. 7. The adversarial objective is a minimax: decoder 1 minimises the phase-2 error and decoder 2 maximises it §3.4 eq. 8 eq. 9. The combined losses are weighted by an evolving factor \epsilon^{-n}, where \epsilon is a training parameter close to one, so the weight on the reconstruction term changes over training §3.4 eq. 10. The paper says the reconstruction term is weighted high at first and the adversarial term is raised as training proceeds §3.4.

Meta-learning §3.4. To help with limited data, the parameters are also updated with MAML after each pass over the windows: an inner step \theta' = \theta - \alpha \nabla_\theta L and an outer step \theta \leftarrow \theta - \beta \nabla_\theta L(\theta') §3.4 eq. 11 eq. 12 alg. 1 line 11. The paper does not state the value of \alpha §4 (see Hyperparameters).

Online inference and thresholding §3.5. At test time, the model runs in the same two phases, using only data up to the current timestamp §3.5 alg. 2 lines 2–3. The anomaly score is half the L2 norm of the phase-1 error plus half the L2 norm of the phase-2 error §3.5 eq. 13. A timestamp is flagged for dimension i when the score s_i reaches a threshold chosen by Peak-Over-Threshold (POT), an extreme-value method (Siffer et al., 2017) §3.5. The timestamp is flagged for detection when any dimension is flagged §3.5 eq. 14 alg. 2 lines 5–6. The paper reports that POT gave about 7.2% higher average F1 than the annual-maximum method for TranAD §3.5.

Training schedule. The training series is split 80% for training and 20% for validation, and training stops when validation accuracy starts to fall §4. The authors call the training loop "offline" and the test loop "online" §3.4 §3.5.

Algorithm 1 (training), in Python-like form:

# TranAD offline training  [alg. 1] [§3.4]
# Input:  windows (T, K, m) from the training series  [§3.2]
# Output: encoder E, decoders D1 and D2

def train(windows, encoder, decoder_1, decoder_2, epsilon, n_limit):
    # n counts processed windows in Alg. 1, not epochs  [alg. 1 line 10]
    n = 0
    while True:
        for window in windows:
            # Phase 1: reconstruct with a zero focus score  [alg. 1 line 5]
            focus = zeros_like(window)
            out_1 = decoder_1(encoder(window, focus))
            out_2 = decoder_2(encoder(window, focus))
            # Phase 2: focus = norm of phase-1 error  [alg. 1 line 6]
            focus = norm(out_1 - window)
            out_hat_2 = decoder_2(encoder(window, focus))
            # Evolving weight of the reconstruction term  [eq. 10]
            weight = epsilon ** (-n)
            # Decoder 1 minimises phase-2 error; decoder 2 maximises it  [eq. 9]
            loss_1 = weight * norm(out_1 - window)
            loss_1 += (1 - weight) * norm(out_hat_2 - window)
            loss_2 = weight * norm(out_2 - window)
            loss_2 -= (1 - weight) * norm(out_hat_2 - window)
            update(encoder, decoder_1, decoder_2, loss_1, loss_2)
            n = n + 1
        # Once per pass: MAML step on a random batch  [eq. 11-12]
        meta_learn(encoder, decoder_1, decoder_2, random_batch(windows))
        if not n < n_limit:
            break

Algorithm 2 (online scoring), in Python-like form, one timestamp at a time:

# TranAD online scoring  [alg. 2] [§3.5]
# Input:  trained encoder, decoder_1, decoder_2; one test window per step
# Output: anomaly score per timestamp (and per dimension, see Pitfalls)

def score(window):
    # Phase 1 with a zero focus score  [alg. 2 line 2]
    out_1 = decoder_1(encoder(window, zeros_like(window)))
    out_2 = decoder_2(encoder(window, zeros_like(window)))
    # Phase 2, conditioned on the phase-1 deviation  [alg. 2 line 3]
    focus = norm(out_1 - window)
    out_hat_2 = decoder_2(encoder(window, focus))
    # Half phase-1 error plus half phase-2 error  [eq. 13]
    return 0.5 * norm(out_1 - window) + 0.5 * norm(out_hat_2 - window)

Threshold and labels, applied to the score of each dimension:

# Labels from scores  [§3.5] [eq. 14] [alg. 2 lines 5-6]
# threshold_i is POT fitted to the anomaly scores of the training windows
def label(score_i, threshold_i):
    # Dimension i is flagged when its score reaches its threshold  [eq. 14]
    flag_i = score_i >= threshold_i
    # Timestamp is anomalous if any dimension is flagged  [eq. 14]
    return any(flag_i)

Key equations, copied in form from the paper (numbers as they appear in the compiled paper):

x_t \leftarrow \frac{x_t - \min(\mathcal{T})}{\max(\mathcal{T}) - \min(\mathcal{T}) + \epsilon'} eq. 1

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{m}}\right)V eq. 2

s = \tfrac{1}{2} \lVert O_1 - \hat{W} \rVert_2 + \tfrac{1}{2} \lVert \hat{O}_2 - \hat{W} \rVert_2 eq. 13

y_i = \mathbf{1}(s_i \geq \mathrm{POT}(s_i)), \quad y = \bigvee_i y_i eq. 14

Hyperparameters

Values stated in the paper, with the official code's value where the code sets one. "not stated" means the paper gives no value.

Dataset statistics, copied from the paper's table with its numbers tab. 1. Values in parentheses are the number of sequences in each dataset; results are averaged over sequences §4.1:

Results

Author-reported. Setup: the nine datasets in tab. 1, the baselines listed in §4, and one NVIDIA RTX 3080 machine §4. The paper excludes the Yahoo benchmark and keeps only the non-trivial sequences of MSL and SMD §4.1.

Headline numbers §4.3. Average F1 across datasets is 0.8802 with full training data and 0.8012 with 20% of training data (F1*) §4.3. Against the state-of-the-art baselines, the paper reports gains of up to 17.06% in F1, 14.64% in F1*, 11.69% in AUC and 11.06% in AUC* §4.3. Training time is 75%–99% lower than the baselines §5.2 tab. 5. Anomaly diagnosis on SMD and MSDS identifies between 46.3% and 75.3% of the root-cause dimensions §4.3 tab. 4.

Full detection results (F1 and AUC on full training data) tab. 2. Bold is the paper's bold.

NAB, UCR and MBA (P = precision, R = recall, AUC = ROC AUC):

SMAP, MSL and SWaT:

WADI, SMD and MSDS:

Results with 20% of the training data (AUC*, F1*) tab. 3. The paper uses a centred 20% of the training series (see Paper vs code). Bold is the paper's bold.

NAB, UCR and MBA:

SMAP, MSL and SWaT:

WADI, SMD and MSDS:

Anomaly diagnosis on SMD and MSDS tab. 4. H = HitRate@P%, N = NDCG@P%, with P = 100 or 150 §4.2. Bold is the paper's bold.

Training time per epoch, in seconds tab. 5. The paper says MERLIN's entry is discord discovery time on the test data rather than training time §5.2. Bold is the paper's bold.

Ablations, F1 and F1 tab. 6 §5.1.* Bold is the paper's bold.

NAB, UCR and MBA:

SMAP, MSL and SWaT:

WADI, SMD and MSDS:

Baseline re-implementation check tab. 7 app. A. Appendix A compares the MERLIN baseline in the original MATLAB code with the authors' Python port, on the same benchmarks. The paper reports that the Python version gives scores close to the original, with training-time gains of up to 96%. The full table has precision, recall, AUC, F1 and time for each dataset; the paper gives the MinL and MaxL values it used per dataset app. A.

Figures, in words.

• fig. 1 shows the TranAD architecture: the window encoder feeding two decoders §3.3.
• fig. 2 plots the anomaly predictions against the ground truth for a sample series §3.5.
• fig. 3 shows, on SMD, that focus scores rise at peaks and sudden changes and that attention weights concentrate on dimensions with larger deviations §3.5.
• fig. 4 gives critical-difference diagrams for F1 and AUC, based on a Friedman test followed by Wilcoxon signed-rank tests at \alpha = 0.05. TranAD ranks first §4.3.
• fig. 5 plots predicted and ground-truth labels on the MSDS test set §4.3.
• fig. 6 varies the fraction of training data from 20% to 100%, and shows F1, AUC and training time, with 90% confidence bounds. TranAD has the highest F1 and the lowest training time at every fraction §5.3.
• fig. 7 varies the window size and shows the same three metrics §5.3.

Paper vs code

Official repository: imperial-qore/TranAD, commit 7ffb98d0c18189cc3d9ab732b4cb0278200a0af0 (cloned at depth 1), BSD-3-Clause, copyright Shreshth Tuli LICENSE. The paper links it in its Software Availability section. The repository is marked "refactored" and says it is not the code used for the baselines README.md.

Each item cites the paper and the code.

1. Adversarial sign is not implemented. The paper gives decoder 2 a negative loss on the phase-2 error §3.4 eq. 9. The code's TranAD branch takes one loss, the weighted sum of both decoders' errors with positive signs: loss = mean(l1) where l1 = (1/n) * l(z[0]) + (1 - 1/n) * l(z[1]) models.py. Both decoders are updated by the same gradient, so the minimax game in eq. 8 is absent.
2. Weight schedule differs. The paper uses \epsilon^{-n} with n counted per window §3.4 alg. 1 line 10. The code uses 1/n and 1 - 1/n, with n equal to the epoch index plus one models.py main.py.
3. Meta-learning is absent. The paper has a MAML step once per pass §3.4 alg. 1 line 11 eq. 11 eq. 12. The released code has no inner or outer MAML update. A search for meta and maml finds only class-name comments models.py models.py models.py models.py.
4. Focus score is squared, not a norm, and not detached. The paper defines the focus score as the L2 deviation of O_1 from W §3.4 alg. 1 line 6. The code uses elementwise squared error c = (x1 - src) ** 2 and does not detach it, so gradients flow through the focus into the phase-1 decoder models.py.
5. Decoder input is the last timestep only. The paper's window encoder attends over the full window with a causal mask §3.3 eq. 5. The code's decoder takes tgt = elem, the last row of the window, repeated twice, as its query, and passes no mask models.py models.py. With one query step, the causal mask is not needed.
6. Architecture count. The paper says the model has two transformer encoders and two decoders §3.3. The code has one encoder and two decoders models.py.
7. Feed-forward size. The paper gives 64 hidden units per encoder layer §4. The code uses dim_feedforward=16 for the encoder and both decoders, with d_{\text{model}} = 2m models.py.
8. Encoder block has no LayerNorm and uses LeakyReLU. The paper's equations 4 and 5 use LayerNorm after each residual §3.3 eq. 4 eq. 5. The custom encoder and decoder layers have residuals but no normalisation, and use LeakyReLU as the activation dlutils.py dlutils.py.
9. Attention scale. The paper scales by \sqrt{m} §3.3 eq. 2. The code uses PyTorch's multi-head attention, which scales by the square root of the head dimension. With d_{\text{model}} = 2m and h = m, the head dimension is 2 dlutils.py models.py.
10. Input scaling is not in the paper. The code multiplies the concatenated input by \sqrt{m} before position encoding models.py.
11. Positional encoding differs from the cited form. The paper cites the standard sinusoidal encoding §3.3. The code adds both sine and cosine of the same angle to every dimension dlutils.py.
12. Anomaly score. The paper scores with half the L2 norm of each phase's error §3.5 eq. 13. The code scores with the squared error of the phase-2 output only, per dimension models.py. Phase 1 is not used at test time.
13. Detection rule. The paper labels a timestamp when any dimension passes its own POT threshold §3.5 eq. 14. The code's reported detection result runs one POT fit on the mean score over dimensions, labelled anomalous when any dimension is labelled main.py. Per-dimension POT is used only for diagnosis main.py.
14. POT parameters. The paper gives coefficient 10^{-4} and low quantiles §4. The code's pot_eval uses q = 10^{-5} pot.py, starts from a per-dataset level constants.py, and multiplies the mean threshold by a per-dataset factor pot.py. The paper's quantiles do not appear in the code.
15. Training duration and split. The paper uses an 80/20 split with early stopping §4. The code trains on the full training series for 5 epochs main.py.
16. Optimiser and scheduler. The paper gives one initial learning rate, 0.01, and a meta rate, 0.02 §4. The code uses per-dataset rates constants.py, AdamW with weight decay 10^{-5} main.py, and StepLR(optimizer, 5, 0.9) main.py. Batch size is 128 models.py.
17. Window padding. The text repeats x_t §3.2. The code pads with the first point of the series, data[0] main.py.
18. Window alignment. The window for index i is data[i-w:i], which ends at x_{i-1}, not x_i main.py. Its last row is the one scored models.py. The labels are not shifted in the scoring loop. The code shifts the ground truth by one step only for plotting main.py. This reading comes from the code and was not verified by running it.
19. Only one trace per dataset is evaluated. The paper reports averages over all sequences §4.1. load_dataset loads a single hard-coded trace, such as P-1_ for SMAP or 136_ for UCR main.py. The README's sample SMAP output has precision 0.8043, recall 0.9999 and F1 0.8915 README.md. These match the paper's SMAP row in tab. 2. Whether the table row is one trace or an average is unresolved.
20. Data preprocessing differs from the paper's description. The paper normalises with training-series minima and maxima §3.2 eq. 1. The code:
    • for NAB, uses the same series as both train and test preprocess.py;
    • for WADI, subsamples every tenth row and normalises train and test each with their own minimum and maximum preprocess.py;
    • for MSDS, subsamples every fifth row and takes the minimum and maximum over train and test together preprocess.py;
    • for SMD, writes the raw values with no normalisation preprocess.py;
    • for UCR, normalises the whole series with its own minimum and maximum before splitting preprocess.py;
    • for SWaT, keeps rows 3000–6000 for training and 7000–12000 for test preprocess.py, which does not match the sizes in tab. 1. The repository's data/SWaT/series.json is empty, so this cannot be checked.
21. 20% of the training data is a centred slice. The paper's sensitivity study uses randomly sampled subsequences §5.3. The code keeps a central contiguous 20% of the series utils.py main.py.
22. Diagnosis rounding. The paper defines the top-candidate count as P% of the ground-truth dimensions, with 100% and 150% as examples §4.2. The code computes round(p * len / 100), which in Python rounds half to even diagnosis.py diagnosis.py. Cases that land on .5 may differ from a round-half-up reading.
23. POT initialisation uses training scores. The code fits POT on the training windows' scores main.py pot.py. The paper does not state this in the algorithm, though it describes a threshold computed from past scores §3.5.
24. Baselines. The README says the baseline implementations in the repository are not the ones used in the paper README.md. The paper says some baselines were re-implemented and some use the original code §4 (footnote).

Pitfalls

• Focus score is ambiguous. The paper writes F = L_1 §3.4, a scalar norm, but also calls it a deviation matrix with the window's shape §3.3. The code uses an elementwise squared error (see item 4 in Paper vs code). Pick one and state it.
• Evolving weight direction depends on \epsilon. For \epsilon < 1, \epsilon^{-n} grows without bound, so 1 - \epsilon^{-n} becomes large and negative. That reverses the schedule the text describes §3.4 eq. 10. The text wants the reconstruction weight high first and falling. For that, \epsilon > 1 is needed. The value of \epsilon is not stated.
• Counter n counts windows. Algorithm 1 increments n inside the per-window loop alg. 1 line 10, so \epsilon^{-n} changes per window. The text calls n the training epoch §3.4. The code uses the epoch count (see item 2 in Paper vs code).
• Which decoder produces \hat{O}_2 at test time. Algorithm 2 assigns \hat{O}_2 from both D_1 and D_2 on one line alg. 2 line 3, while training uses only D_2 alg. 1 line 6. The score in eq. 13 needs one \hat{O}_2. The text refers to "the pair (O_1, \hat{O}_2)" §3.5.
• Per-dimension scores. eq. 13 uses vector norms, giving one number per window. The text then talks about per-dimension scores s_i §3.5 eq. 14. Use a per-dimension error if you want diagnosis; the paper does not write this out.
• Normalisation can push test values outside [0, 1). Eq. 1 uses training minima and maxima §3.2. Test values outside the training range fall outside the sigmoid's output range (0, 1), so the decoder cannot match them and the error grows.
• Padding. Repeating x_1 (code) and repeating x_t (text) differ for early windows. Test windows are affected too.
• Early stopping on "accuracy". Reconstruction has no accuracy. The paper's stopping rule §4 cannot be implemented as written. The code runs five epochs.
• Learning-rate scale. Learning rates differ by 90× across datasets in the code (0.0001 to 0.009) constants.py. None of the code's per-dataset values equals the paper's single 0.01 §4 constants.py.
• Meta-learning is a claimed component with no code. The ablation without MAML shows about 1% F1 change but about 12% F1* change on average §5.1. Without the released code path, the MAML contribution cannot be checked.
• Diagnosis and detection can disagree. Detection in the code is a single threshold on the mean score (item 13). Diagnosis thresholds each dimension. A timestamp can be detected but have no dimension flagged, or the reverse.
• WADI text and table disagree on the best F1*. The text says OmniAnomaly has the highest WADI F1* at 0.1017 §4.3. The table bolds MERLIN at 0.1174 tab. 3. Check before quoting.
• Diagnosis percentages at small counts. The code's rounding (item 22) can change the top-candidate count, and if one NDCG call raises an exception, the function returns an empty result for all NDCG keys diagnosis.py. Either can change the reported averages.
• The training-time comparison is one machine. The paper reports per-epoch time on one RTX 3080 workstation §4 tab. 5. The code's epoch count (5) and the paper's early-stopping schedule are different, so per-epoch time is the only measure they share.
• Relative gains are per dataset. "Up to 17%" §1 §4.3 is the largest relative increase over the state-of-the-art baselines on any one dataset, not an average.
• POT parameters are hard to reproduce. The paper's quantiles and coefficient §4 and the code's level and multiplier constants.py pot.py give different thresholds. Report which one you use.
• The Python MERLIN port is not the original. The authors grid-searched MinL and MaxL per dataset to maximise F1 app. A. The paper does not say which split the search used, so it may have used the evaluation labels. Treat the MERLIN row with that in mind.
• Gains are on public benchmarks with known quality concerns. The paper drops NAB's nyc-taxi trace and the Yahoo benchmark because of label problems §4.1, and says it shares concerns about benchmark quality in general §4.1. Comparisons inherit those concerns.
• Dataset count is inconsistent. The abstract says six public datasets [abstract]. Section 4.1 says seven §4.1. Table 1 and Table 2 list nine tab. 1 tab. 2. Check which set an average covers before comparing it with another paper.
• Test-time cost. Phase 2 repeats the encoder and decoders for each timestamp alg. 2, so each score costs about two forward passes. The paper does not report memory use.

Lineage

• Builds on: the transformer encoder-decoder with scaled dot-product attention and positional encoding §3.3 (Vaswani et al., 2017); MAML for meta-learning §3.4 (Finn et al., 2017); the adversarial two-decoder idea from USAD §2 (Audibert et al., 2020); and POT thresholding from OmniAnomaly §3.5 (Su et al., 2019) §2.
• Compared against: GDN, CAE-M, MTAD-GAT, USAD, MAD-GAN, MSCRED, OmniAnomaly, DAGMM, LSTM-NDT and MERLIN §4 tab. 2.
• Superseded by: not known from content already in this wiki.