Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals

In plain words

Sensors such as wrist trackers, EEG caps or wearable skin sensors produce many signals at once. The task is to learn what "normal" looks like from recordings without labels, then flag any stretch that does not fit. Simple methods miss how the signals relate to each other over time. Plain autoencoders can also learn to rebuild bad data too well, which hides the anomalies.

CAE-M combines three parts. A small convolutional network compresses all sensor channels into a short code and tries to rebuild the input. A penalty pushes the spread of those codes toward a bell-shaped target, so odd or noisy samples do not stand out as a free-for-all. Two predictors then guess the next step of the signal from the past: one is a recurrent network with attention over time, the other a simple linear model. All three parts are trained together.

At test time, a sample is scored by how badly the rebuild and the two guesses miss. Scores above a threshold learned from normal training data are called anomalies.

TL;DR

• Unsupervised detector for multi-sensor time series; trained on normal data only §4.3.
• Convolutional autoencoder with an MMD penalty, plus an attention BiLSTM and an autoregressive predictor, optimised end-to-end §3.3–§3.5.
• Anomaly score is the summed loss; the threshold is the training-loss mean plus one standard deviation §3.6.
• Author-reported F1 beats ten baselines on PAMAP2, CAP and a private fatigue dataset tab. 2. No code is public.

Problem

Multi-sensor data has spatial structure (across sensors) and temporal structure (over time). The authors argue that three things fail together: classical methods (PCA, k-means, OCSVM) ignore temporal memory §1 §2.1; plain autoencoders can reconstruct abnormal or noisy training data well, so they generalise poorly §1 §3.3.2; and multi-stage pipelines that separate feature learning from the predictor can get stuck in poor local optima §3.5. The training set is assumed to contain some noise and some anomalies, and no labels are used §3.1, Theorem box "Unsupervised anomaly detection".

Method

Overview §3.2 fig. 1. The characterization network (a convolutional autoencoder) produces, for each time window, a low-dimensional code and a reconstruction error. The memory network predicts the current window's features from earlier windows, using a non-linear branch and a linear branch. The three losses (reconstruction, MMD, and the two prediction losses) are summed with weights eq. 20.

Notation §3.1. There are n sensors with m signals each, so N = nm signals of common length T. One sample is an N × T matrix. Each sample is split into h time steps (windows) before the memory network sees it (Algorithm 1 transforms each sample into an h × N × t array) alg. 1.

Step 1, deep feature extraction §3.3.1 eq. 1 eq. 2 eq. 3.

1. Encode the sample x into a hidden code z_f with convolution and pooling layers eq. 1.
2. Decode z_f back to x' with transposed convolutions eq. 2.
3. The reconstruction error z_r is the squared l2 distance between x and x' eq. 3.

Step 2, MMD penalty on the codes §3.3.2 eq. 4 eq. 5. 4. Collect the codes of all time steps in a batch: each batch of M samples and h steps gives M·h codes §3.3.2. 5. Measure the maximum mean discrepancy between those codes and samples from a Gaussian target P_z, using a Gaussian kernel k(u, v) = exp(-‖u−v‖² / 2σ²) eq. 4 eq. 5. The authors write the kernel MMD in the three-term form of eq. 5, without the i = j terms.

Step 3, memory network inputs §3.4 eq. 6. 6. For each time step h, form z_h = [z_f, z_r]_h: the code concatenated with the reconstruction error eq. 6.

Step 4, non-linear predictor §3.4.1 eq. 7 to eq. 17. 7. Run a bidirectional LSTM over the steps; each direction is a standard LSTM with input, forget, output gates and a cell state eq. 7 to eq. 12. The merged output is one value per step eq. 13. 8. Attention scores each step's output with a tanh and sigmoid layer, then a softmax over steps eq. 14 to eq. 17. 9. The context vector is the attention-weighted sum of the step outputs, taken as the prediction of z_h from z_1 … z_{h−1} eq. 17.

Step 5, linear predictor §3.4.2 eq. 18. 10. An autoregressive model: a dense layer combines the past values z_{h−i} with learned weights and a constant term eq. 18.

Step 6, prediction loss §3.4.2 eq. 19. 11. The two predictions are scored against the true z_h with squared Frobenius error eq. 19.

Step 7, joint objective §3.5 eq. 20. 12. Total loss J = L_MSE + λ1·L_MMD + λ2·L_lp + λ3·L_np, where L_MSE is the reconstruction error, L_lp the linear prediction error and L_np the non-linear prediction error eq. 20. All of it is trained end-to-end.

# One training step of CAE-M  [alg. 1] [§3.3] [§3.4] [§3.5]
# Input:  batch of normal samples, (M, N, T); weights lambda_1..3
# Output: loss to minimise

def training_step(batch, lambda_1, lambda_2, lambda_3):
    # Split each sample into h time steps  [alg. 1]
    windows = split_time(batch, h)                # (M, h, N, t)

    # Encode and decode each window  [§3.3.1] [eq. 1] [eq. 2]
    latent = encode(windows)                      # z_f
    recon = decode(latent)                        # x'

    # Reconstruction error per step  [§3.3.1] [eq. 3]
    recon_error = squared_error(windows, recon)   # z_r

    # Penalty pulling the codes toward a Gaussian  [§3.3.2] [eq. 4] [eq. 5]
    mmd = mmd_to_gaussian(latent)

    # Join code and error into one feature per step  [§3.4] [eq. 6]
    features = concat(latent, recon_error)        # z_h

    # Predict each step from the steps before it  [§3.4.1] [§3.4.2]
    # [eq. 7-18]
    past = features[:, :-1]
    target = features[:, -1]
    nonlinear_pred = attention_bilstm(past)       # Y_h
    linear_pred = autoregressive(past)            # z_hat_h

    # Compound objective  [§3.5] [eq. 20]
    loss = mean(recon_error) + lambda_1 * mmd
    loss += lambda_2 * mean(sq_err(nonlinear_pred, target))
    loss += lambda_3 * mean(sq_err(linear_pred, target))
    return loss

Decision threshold §3.6 eq. 21. After training, compute the summed loss for every training sample, take the mean μ and the standard deviation σ (divided by D, not D−1), and set THR = μ + σ eq. 21.

# Threshold from training losses  [§3.6] [eq. 21]
# Input:  err, one summed loss per training sample (normal data)
# Output: threshold

def threshold(err):
    # Mean of the training losses  [eq. 21]
    mu = mean(err)
    # Population standard deviation of the training losses  [eq. 21]
    sigma = std(err, ddof=0)
    return mu + sigma

Inference §3.6 alg. 1. For each test sample, compute the same summed loss Err(x). Label it anomaly if Err(x) > THR, otherwise normal §3.6 alg. 1.

# Label one test sample  [§3.6] [alg. 1]
# Input:  sample; threshold from training losses
# Output: "anomaly" or "normal"

def predict(sample, threshold):
    # Same combined loss as in training  [§3.5] [eq. 20]
    err = combined_loss(sample)
    if err > threshold:
        return "anomaly"
    return "normal"

Equations are cited by the number they carry in the compiled paper. The attention equations eq. 14 to eq. 17 are each unlabelled in the source but are numbered in order.

Hyperparameters

Datasets tab. 1 §4.1:

Anomaly definition per dataset §4.1: PAMAP2 uses running, ascending stairs, descending stairs and rope jumping as anomalies, with all other activities normal. CAP uses sleep-disordered breathing as anomaly and healthy subjects as normal. Fatigue data is anomalous (fatigue), and is about a fifth of the total.

Results

Author-reported. Setup: three datasets (PAMAP2, CAP, private mental fatigue), ten baselines, mean precision, recall and F1 averaged over normal and anomalous classes §4.4 tab. 2. Each cell is the mean over the test protocol described in §4.3.

Main comparison tab. 2. The paper's table, reproduced with its numbers.

The paper also prints an "Improvement" row (7.96 %, 6.48 %, 7.96 % for PAMAP2; 6.72 %, 6.93 %, 6.98 % for CAP; 5.89 %, 6.13 %, 6.16 % for fatigue) tab. 2. Its basis is not stated and it does not match the table (see Pitfalls).

Per-sleep-stage mF1 tab. 3. Only mF1 is reproduced here; mPre and mRec for each stage are in tab. 3. The paper marks p = 0.0074 for this table.

Leave-one-subject-out on PAMAP2, 8 subjects tab. 4.

Repeated measures for subject 1 tab. 5 (mean ± spread as printed).

Ablation tab. 6 fig. 2b. Each row removes or changes one part. Loss weights for each variant are in §4.3.

Other reported results, all author-reported:

• Robustness to injected noise (PAMAP2, three methods only): CAE-M's F1, precision and recall show no significant decline as the noisy fraction grows from 1 % to 30 % §4.7 fig. 3. No table is given.
• Parameter sensitivity: F1 tends to fall as any λ grows; the paper reports robustness over a wide range §4.8.1 fig. 4.
• Convergence: the three datasets' losses level off within fewer than 40 iterations §4.8.2 fig. 5.

Paper vs code

No official code.

Searches made, as required for this card:

• The paper source (raw/src/2107.12626) has no code, project or repository link. The arXiv metadata has no comments field entry with a link.
• GitHub repository search: CAE-M, CAE-M anomaly, Convolutional Autoencoding Memory anomaly, multi-sensor anomaly detection autoencoder attention. The only hit with "CAE-M" in the name is pranyawadhwa/CAE-MVTec-Anomaly-Detection, a third-party project on a different dataset; it is not used.
• GitHub user search for the authors: Zhiwen Pan returned an unrelated account; a repository search restricted to user:jindongwang for "anomaly" returned nothing.

No discrepancies with code can be checked. The internal inconsistencies in the paper are listed under Pitfalls.

Pitfalls

• No code, so every detail is from the paper. Several details an implementer needs are missing: the learning rate, the Hyperopt search space and final values, the MMD bandwidth σ, the latent dimension, the number of training epochs, the BiLSTM hidden size per direction, the attention scoring details, and the padding and stride of the convolutions. The only tuned values given are λ and h = 5 or 10 §4.3.
• Layer description contradicts itself. The text says each convolution is followed by max-pooling §3.3.1, but the listed chain has two max-pool layers and three transposed convolutions with none §4.3. The stack also has 5 conv layers but is referred to as Conv1–Conv5 with 32, 64, 64, 32, 1 filters §4.3. Build what the chain says, and check the output shape against N × T.
• Input shape is ambiguous. Section 3.3 packs each sample as N × T §3.3.1, but Algorithm 1 reshapes it to h × N × t across time alg. 1. The window length t is never given for CAE-M.
• λ1 is written "e-04" throughout. The leading digit is missing in the source. Read it as 10^-4 §3.5 §4.3 §4.8.1.
• The reconstruction term has no weight. The objective writes L_MSE with no λ eq. 20. The "w/o Rec+MMD" ablation says it removes the reconstruction error, but setting λ1 = 0 only removes MMD. The reconstruction term is still in the objective. Implement the ablation as "reconstruction term dropped" only if you have checked the code, which is not available here.
• Bi-LSTM merge mode. The text says outputs of the two directions are merged by sum §3.4.1. Equation 13 writes concatenation eq. 13. Choose one and say which.
• Typo in the candidate cell. Equation 10 uses h_{h−1} where the rest of the BiLSTM uses y_{h−1} eq. 10. Read it as y_{h−1}.
• Prediction loss defined twice. Equation 19 sums the squared errors over the batch, while equation 20 uses a mean over the batch of the same two terms eq. 19 eq. 20. Use the mean, since that is what the objective uses.
• Attention sums over all steps in one place and h−1 in another. Equation 17 sums over h eq. 17. The implementation details say attention over h−1 steps §4.3. Predicting z_h from z_1 … z_{h−1} is consistent only with h−1.
• Threshold is on training data. The threshold is the mean plus one standard deviation of training losses §3.6 eq. 21. It is not tuned on the validation split, although the split is described as used for selection §4.3. The text's wording about the normal distribution is loose; the formula in eq. 21 is the operative definition.
• Improvement row does not match its table. The "Improvement" row reports 7.96 % on PAMAP2 F1 tab. 2. The ratio of CAE-M's mF1 to the best baseline in the same table is 0.9639 / 0.8843 − 1 ≈ 9.0 % (ConvLSTM-COMP), not 7.96 %. CAP and fatigue give ≈ 7.5 % and ≈ 6.6 % against the same comparison. The paper does not say how the row is computed.
• Dataset numbers disagree. Section 4.3 says PAMAP2 has 36 dimensions §4.3, but Table 1 gives 27 tab. 1. Section 4.1 says 9 subjects §4.1, and the LOSO study uses 8 §4.6.1. Use the table only after noting this.
• Per-stage results are not uniformly best. For S3, OCSVM (0.9605) and ConvLSTM-COMP (0.9217) are above CAE-M (0.9063) tab. 3. The text does not discuss this.
• Baseline set-up affects the comparison. The paper reports MSCRED with batch size 1 in the public code, so it did not converge §4.4. Baseline numbers depend on re-implementations (the paper uses PyOD and a third-party MSCRED repo) and on feature choices §4.3. Treat the comparisons as author-reported, not reproduced.
• Evaluation protocol. Precision, recall and F1 are averaged over normal and anomalous classes §4.3. The threshold depends on the training set, which has normal samples only. Anomalous samples appear only in the test set §4.3, and the test-set class balance is not stated, which affects precision.
• Noise and convergence claims are narrow. Noise injection is tested on PAMAP2 with three methods only §4.7. Convergence is shown on three datasets, with "fewer than 40 iterations" and no unit given §4.8.2.
• The "steady state" is not a training budget. Do not read "fewer than 40 iterations" as a stopping rule without the training code.

Lineage

• Builds on: the composite-model idea of Zong et al. (a reconstruction autoencoder plus a density model in the latent space) and the Gaussian-mixture regularisation they use. The paper adds MMD in place of the mixture model §1 §3.2.
• Uses: bidirectional LSTM with attention §3.4.1; autoregressive prediction §3.4.2; Maximum Mean Discrepancy §3.3.2.
• Compared against: KPCA, ABOD, OCSVM, HMM, CNN-LSTM, LSTM-AE, UODA, MSCRED, ConvLSTM-AE and ConvLSTM-COMPOSITE §4.2 tab. 2.
• Superseded by: not known from this wiki.