Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection

In plain words

Spotting unusual moments in a long video is hard when you only know whether a whole clip contains something unusual, not when it happens. Most systems learn mainly from the unusual clips, and that leaves them weak on what normal footage looks like. Ordinary actions, such as running in exercise videos, can look like arrests, so the system raises false alarms.

This paper's model keeps two lists of learned patterns. One list holds typical normal moments, and the other holds the unusual ones. A new moment is compared with both lists, and the model is trained to match each moment to the right list and to keep the two apart. Normal moments are also stored in a smooth, spread-out space, so that harmless variation, such as a camera switch, is less likely to look abnormal. An attention block lets each moment take context from both nearby and distant moments in the video.

The approach matters because it targets the false alarms that weakly supervised systems produce. The authors report better results than the baselines they compare with on two standard video datasets. These numbers are theirs and have not been checked independently.

TL;DR

• Task: frame-level anomaly detection in untrimmed videos, trained with video-level labels only §1.
• Core ideas: a global-plus-local attention block §3.2, two memory banks that store normal and abnormal prototypes §3.3, and a Gaussian latent space for normal features with an uncertainty term §3.4.
• Author-reported: 86.97% frame AUC on UCF-Crime and 81.66% AP on XD-Violence with I3D RGB features tab. 1 tab. 2.
• An official MIT-licensed repo exists. Its shipped best-record files show the headline numbers, but those checkpoints were selected on the test split, and the code differs from the paper in several places (see Paper vs code). I did not re-run training.

Problem

Weakly supervised video anomaly detection (WS-VAD) gets one label per video, normal or abnormal, and must localise anomalies in time §1. Multiple-instance methods treat abnormal videos as bags with at least one abnormal snippet. The authors argue that focusing on anomalous patterns without a normal contrast overfits, and that ignoring normal data yields a higher false-alarm rate §1 §2. A single memory of normal prototypes still fails on hard samples, such as running that occurs both in exercise videos and in arrest videos §1. Feature-magnitude methods such as RTFM assume normal snippets have low magnitude, but normal features also vary with camera switches and similar noise §1 §3.4.

Method

Overview §3. The model works on pre-extracted I3D features, one vector per 16-frame snippet §3.1. Three modules follow:

1. GL-MHSA (global and local multi-head self-attention) turns the snippet features into context-aware embeddings. It adds a second, local branch with a distance-decaying mask to the usual attention §3.2.
2. Dual memory units (DMU): an abnormal bank and a normal bank of learned prototypes. Each snippet is matched to both banks, and a loss shapes the banks §3.3.
3. Normal data uncertainty learning (NUL): the normal bank's output is mapped to a Gaussian over latent normal features, and a distance term keeps abnormal features away from it §3.4.

Training uses the video-level labels, sampled in pairs of normal and abnormal videos §3.5. Testing uses only the mean encoder, so no random sampling is involved §3.5.

Steps, following the paper's order:

1. Extract I3D RGB features (Kinetics-400 pretraining) for each video, one vector per 16-frame snippet §3.1 §4.2.
2. During training, split each video into N = 200 snippets and average the features inside each snippet span §4.2 dataset_loader.py.
3. Apply a global self-attention and a local self-attention, combine them, and add the feed-forward layer and normalisation §3.2 eq. 1 eq. 2.
4. Read each memory bank: compute the sigmoid match of every snippet to every slot, then the augmented feature as the match-weighted sum of slots §3.3 eq. 3.
5. For each bank, take the top-K memory matches per snippet and average them, then keep the K snippets with the highest match §3.3 eq. 4.
6. Train the banks with the dual memory loss, which asks each video to match the bank it belongs to §3.3 eq. 5.
7. Pull the anchor (normal snippets, normal bank) and the positive (abnormal video, normal bank) towards each other and the negative (abnormal video, abnormal bank) away, with a triplet loss §3.3 eq. 6.
8. Sample the normal latent with the reparameterisation trick, keep it close to a standard normal with a KL term, and push abnormal means away with a margin term §3.4 eq. 7 eq. 8 eq. 9 eq. 10.
9. Combine the video-level BCE loss with the four weighted terms §3.5 eq. 11.
10. At test time, use the mean encoder for the normal bank (no sampling), score every snippet, and assign each snippet's score to its 16 frames §3.5 ucf_test.py.

Pseudocode for the global and local attention, with the local mask §3.2 eq. 1 eq. 2 translayer.py:

# GL-MHSA for one batch of snippet tokens  [§3.2]
# Input:  tokens (B, N, D); N snippets, D = model width
# Output: contextual features (B, N, D)
def gl_mhsa(tokens):
    # Query, key, value and a separate local value, per head  [§3.2]
    queries, keys, values, local_values = project_four(tokens)
    # Global branch: standard scaled dot-product attention  [eq. 2]
    global_scores = queries @ keys.T / scale  # scale: see Pitfalls
    global_out = softmax(global_scores) @ values
    # Local branch: softmax over minus the snippet distance, divided by e^tau
    # Mask is the same for every head and batch item  [eq. 1]
    local_mask = softmax(-abs(i - j) / exp(tau))  # (N, N), tau = 1 in code
    local_out = local_mask @ local_values
    # Concatenate both branches per head, then project back to D  [eq. 2]
    return project_out(concat(global_out, local_out))

Key equations, as written in the paper §3.2 §3.3 §3.4:

\textbf{T}_{m}(i,j)=-\frac{|i-j|}{e^{\tau}} eq. 1

\textbf{X}=\text{MH}\big([\text{softmax}(\tfrac{\textbf{Q}\textbf{K}^T}{\sqrt{D}})\textbf{V}_g;\ \text{softmax}(\textbf{T}_{m})\textbf{V}_L]\big), \quad \textbf{X}=\text{LN}(\text{MLP}(\text{LN}(\textbf{X}))+\textbf{X}) eq. 2

\textbf{S}=\sigma\Big(\frac{\textbf{X}\textbf{M}^T}{\sqrt{D}}\Big), \quad \textbf{M}_{aug}=\textbf{S}\textbf{M} eq. 3

\textbf{S}_{k;ind},\ \textbf{S}_{k;sc}=\text{topK}(\textbf{S},K), \quad \textbf{S}_{k;i}=\frac{\sum_{j=1}^{K} S_{k;sc}(i,j)}{K} eq. 4

L_{dm}=\text{BCE}(\textbf{S}^n_{k;n},\textbf{y}^n_n)+\text{BCE}(\textbf{S}^n_{k;a},\textbf{y}^n_a)+\text{BCE}(S^a_{k;n;k},y^a_n)+\text{BCE}(S^a_{k;a;k},y^a_a) eq. 5

L_{trip}=\text{Triplet}(f_a,f_p,f_n),\quad f_a=\text{topK}(\textbf{S}^n_{k;n};\textbf{X}^n),\ f_p=\text{topK}(\textbf{S}^a_{k;n};\textbf{X}^a),\ f_n=\text{topK}(\textbf{S}^a_{k;a};\textbf{X}^a) eq. 6

p(z^n_i\mid x^n_i)=\mathcal{N}(z^n_i;\mu_i,\sigma_i^2 I) eq. 7

z^n_i=\mu^n_i+\sigma^n_i\varepsilon,\quad \varepsilon\sim\mathcal{N}(0,1) eq. 8

L_{kl}=-\frac{1}{2D}\sum_{i=1}^{D}\big(1+\log(\sigma^n_i)^2-(\mu^n_i)^2-(\sigma^n_i)^2\big) eq. 9

L_{dis}=\max\big(0,\ d-(\lVert\mu^a_k\rVert_2^2-\lVert z^n_k\rVert_2^2)\big) eq. 10

L=L_{cls}+\lambda_1 L_{dm}+\lambda_2 L_{trip}+\lambda_3 L_{kl}+\lambda_4 L_{dis} eq. 11

Pseudocode for the memory read and the dual memory loss §3.3 eq. 3-5:

# Read one memory bank and score each snippet against it  [§3.3]
# Input:  tokens (N, D); bank (M, D), M = 60 slots
# Output: per-snippet match (N,), augmented feature (N, D)
def read_memory(tokens, bank):
    # Sigmoid match of each snippet to each slot, not a softmax  [eq. 3]
    scores = sigmoid(tokens @ bank.T / sqrt(D))  # (N, M)
    # Augmented feature: match-weighted sum of the slots  [eq. 3]
    augmented = scores @ bank  # (N, D)
    # Per snippet: mean of its top-K slot matches, K = floor(M/16) + 1  [eq. 4]
    k_slots = M // 16 + 1
    match = mean(topk(scores, k_slots), axis=-1)  # (N,)
    return match, augmented

# Dual memory loss for one batch of normal and abnormal videos  [§3.3] [eq. 5]
# Inputs: per-snippet matches (N,) for each video and bank, from read_memory
# Labels are the targets stated in the paper
def dual_memory_loss(normal_on_normal, anomaly_on_normal,
                     normal_on_abnormal, anomaly_on_abnormal):
    # Normal bank should fire on every snippet of a normal video  [eq. 5]
    loss = bce(normal_on_normal, ones_like(normal_on_normal))
    # Abnormal bank should stay silent on normal snippets  [eq. 5]
    loss += bce(anomaly_on_normal, zeros_like(anomaly_on_normal))
    # Normal bank on an abnormal video, labelled 1 as the paper states  [eq. 5]
    loss += bce(mean_topk(normal_on_abnormal), 1)
    # Abnormal bank fires on the abnormal video  [eq. 5]
    loss += bce(mean_topk(anomaly_on_abnormal), 1)
    return loss

Latent normal space, KL and margin, and the total loss §3.4 §3.5 eq. 8-11:

# Losses for the normal latent space  [§3.4]
# Inputs: mu, logvar: encoder outputs for normal snippets (N, D)
#         anomaly_mu: mean-encoder output for abnormal snippets (N, D)
# Output: kl and margin terms (scalars)
def latent_losses(mu, logvar, anomaly_mu):
    # Reparameterised sample for training only  [eq. 8]
    # Code reads logvar as log-variance  [code: model.py#L46]
    std = exp(0.5 * logvar)
    latent = mu + std * random_normal_like(std)  # (N, D)
    # KL to a standard normal, scaled by 1/(2D)  [eq. 9]
    kl = -sum(1 + logvar - mu**2 - exp(logvar)) / (2 * D)
    # Margin between abnormal means and normal latents  [eq. 10]
    gap = squared_norm(anomaly_mu) - squared_norm(latent)
    margin = relu(100 - gap)  # d = 100  [§4.2]
    return kl, margin

# Total loss with lambda = (0.1, 0.1, 0.001, 0.0001)  [§4.2] [eq. 11]
# Input: the video-level BCE and the four auxiliary terms
def total_loss(video_bce, dual_mem, triplet, kl, margin):
    return (video_bce
            + 0.1 * dual_mem
            + 0.1 * triplet
            + 0.001 * kl
            + 0.0001 * margin)

The video score is pooled over snippets, and the paper does not say how. The code uses the mean of the top-K snippet scores before the BCE [code: train.py#L29-L30]. Pooling and the four auxiliary terms are described in the loss pseudocode above; the code's pooling is in Paper vs code.

Hyperparameters

Results

Author-reported. Setup §4.1: UCF-Crime has 1,900 videos over 13 anomaly types, with 800 normal and 810 abnormal training videos and 140 normal and 150 abnormal test videos. XD-Violence has 4,754 videos, with 3,954 for training and 800 for testing. Metrics: frame-level AUC for UCF-Crime and frame-level precision-recall AP for XD-Violence §4.1. Hardware is not stated. The baselines are the methods listed in each table.

Headline numbers §4.3 §4.4:

• UCF-Crime frame AUC: 86.97% with 10-crop I3D features. That is +1.35 points over MSL with VideoSwin (85.62%) and +2.67 points over RTFM (84.30%) tab. 1.
• XD-Violence AP, RGB only: 81.66%, +3.85 points over RTFM (77.81%) tab. 2.
• XD-Violence AP, RGB and audio by feature concatenation: 81.77% tab. 2. The audio features are VGGish from the XD-Violence release §4.4.

Table 1: frame-level AUC on the UCF-Crime test set tab. 1. Rows are grouped as unsupervised and weakly supervised. The second column names the input features.

Table 2: frame-level AP on the XD-Violence validation set tab. 2. The single-feature rows are RGB, and the rest use audio as well.

The MSL RGB entry appears twice in the source table with different numbers. Both are kept as printed tab. 2.

Table 3: memory-unit ablation tab. 3. "Single A-Mem" is the abnormal bank alone, "Single N-Mem" is the normal bank alone, and "Dual Mems" is the full model. FAR is the false-alarm rate on normal videos, as the paper defines it.

Table 4: number of memory slots (M_A, M_N) tab. 4.

Table 5: loss-term ablation tab. 5. Check marks show which terms are on.

Table 6: module ablation tab. 6. AUC_sub and AP_sub are scored on the abnormal videos only. FAR is measured on normal videos only.

Qualitative results fig. 4 show eight videos, four per dataset. The authors compare their frame scores with RTFM's on the same videos, and report fewer false alarms on normal footage §4.6.

Paper vs code

Official repository: henrryzh1/UR-DMU, MIT license, commit 40cfdf5f8bebbc3b59373935f9d6ca00e2f32bbc. Its README calls it the implementation of this paper. The owner's handle (henrryzh1) resembles the local part of the first author's HUST address in the TeX source (henrryzh); this is a resemblance, not confirmation. I checked the code by reading the model, memory, loss, data, training and test files.

• Fourth dual-memory term. Paper: the normal bank on an abnormal video is labelled 1 §3.3 eq. 5. Code: the target is 1 on 1 - N_Aatt, which pushes the normal bank's score on abnormal clips towards 0 train.py. The other three terms match the paper.
• KL normalisation. Paper: 1/(2D) eq. 9. Code: the sum over D without 1/D, then the batch mean model.py. The code term is D times the paper's, which is 512 at the default width. The same lambda of 0.001 therefore weights it 512 times more strongly in code.
• Distance margin. Paper: squared L2 norms eq. 10. Code: plain L2 norms model.py.
• Top-K size. Paper: K = floor(M/16)+1 = 4 for every top-K §4.2. Code: the memory-level top-K uses M//16+1 = 4 memory.py, but the snippet-level selection and the video pooling use T//16+1, where T is the number of snippets. With 200 training snippets that is 13 model.py model.py model.py train.py train.py.
• Attention scale. Paper: 1/sqrt(D) in eq. 2 and eq. 3. Code: 1/sqrt(128), the per-head width translayer.py. Memory reads (eq. 3) do use sqrt(512) memory.py.
• Attention block layout. Paper: one block with LN after the feed-forward residual eq. 2. Code: two stacked pre-norm blocks, GELU feed-forward and dropout 0.5 model.py translayer.py.
• Embedding layer. Code applies a 1D convolution, kernel 3, from 1024 to 512 with ReLU before the transformer model.py [code: model.py#L36]. The paper does not describe this layer.
• Log-variance. Code treats the variance head's output as a log-variance, with standard deviation exp(logvar/2) model.py. The paper calls sigma a variance §3.4 but uses it as a standard deviation in eq. 8 and as a squared term in eq. 9.
• Sampling. Code samples the normal latent only during training, and uses the mean at test model.py model.py. This matches the paper §3.5.
• Test-time length. Code does not resample test videos to N snippets; it feeds full-length features dataset_loader.py. The paper does not say what length is used.
• Number of auxiliary losses. The text says "five auxiliary losses" §3.5, but eq. 11 has four weighted terms plus the classification loss. Code uses the four weighted terms with lambdas 0.1, 0.1, 0.001 and 0.0001 train.py.
• Audio. The repo has no audio loader. The XD loader reads one feature directory, and the RGB path is the only one wired up dataset_loader.py dataset_loader.py. The RGB+Audio row cannot be reproduced from this code.
• FAR. The paper reports FAR §4.1 tab. 3. The repo's test loop computes video-level accuracy at threshold 0.5 ucf_test.py xd_test.py and no FAR. The FAR definition in the paper is the one used for the tables.
• Model selection. The paper does not say how a checkpoint was chosen. The code saves the checkpoint with the best test AUC for UCF-Crime and the best test AP for XD-Violence, checked every 10 steps ucf_main.py xd_main.py. The shipped records are UCF step 1430 (AUC 0.8697, AP 0.3559) and XD step 840 (AUC 0.9402, AP 0.8166) ucf_best_record_2022.txt xd_best_record_2022.txt. These match the paper's headline numbers, but they are test-selected.

No discrepancies found for the loss weights, the margin d = 100, the 60-slot banks, the 200-snippet training length, the learning rate or the 3000 steps.

Pitfalls

• Many details are not in the paper. The optimizer, the weight decay, the classifier head, the triplet margin, the L2 normalisation before the triplet, the memory initialisation, the dropout, the feed-forward width, the transformer depth, the feature width D and tau are all not stated. The code settles each of them (see the table above). Implementations that follow only the paper will differ on these.
• Top-K size is the largest source of disagreement. Following the paper's K = 4 for snippet selection gives a very different selection from code's 13. Check which one you implement, and record it.
• KL weight depends on the normalisation. At lambda = 0.001 the paper's 1/(2D) form is 512 times weaker than the code's sum, so copying the lambda alone changes the balance of the loss.
• Top-K indices carry no gradient. Snippets are picked with top-K and then gathered, so gradients reach only the gathered features, not the choice of snippet model.py. The paper does not discuss this.
• Gaussian sampling and scale. The paper warns that a large sigma moves the sample far from its mean §3.4. The code does not clip sigma or the log-variance model.py.
• Memory reads are sigmoid, not softmax. Each slot's match is independent, and M_aug = S M is an unnormalised sum eq. 3, so its scale depends on how many slots fire. Table 4 peaks at 60 slots for both datasets, not at the largest bank.
• Attention scale is ambiguous. The paper writes sqrt(D) with D undefined for the attention block eq. 2. Code divides by sqrt(128), the per-head width translayer.py. Dot-product magnitudes, and so the softmax sharpness, differ by a factor of about 2 between the two readings.
• Cost grows with video length. Attention is quadratic in the number of snippets, and test videos are not resampled. Long videos at test are much more expensive than the 200-snippet training videos. The code also computes the local mask on the GPU, so CUDA is required translayer.py.
• Test-set selection. The UCF and XD numbers in the shipped records are the best checkpoints picked on the test split (see Paper vs code). They are not held-out results.
• No variance reported. The paper gives no run-to-run variance or seed spread. Small gaps in the ablations, such as 86.60 against 86.97 for UCF AUC in Table 5, cannot be judged without it.
• Frame labels come from 16-frame snippets. Each snippet score is copied to its 16 frames before computing AUC and AP ucf_test.py xd_test.py. Frame alignment errors shift the curves.
• Stated losses versus the code's loss. The paper's total loss includes the classification term L_cls without saying how snippet scores are pooled into a video score. Code uses the mean of the top-K scores, and a different pooling will change the classification term.

Lineage

• Builds on: RTFM's feature-magnitude learning §1 §3.4; the data uncertainty learning scheme of DUL, used in face recognition §2; the memory-augmented autoencoders MemAE and MNAD §2; and the global and local structure of graph convolutional networks, transferred to self-attention §3.2. The paper cites these by author name; it gives no arXiv ids.
• Compared against: Hasan et al., Lu et al., GCL, Sultani et al., MA, GCN, IBL, HL-Net, CLAWS, MIST, RTFM, CRFD, MSL, Pang et al. and ACF tab. 1 tab. 2.
• Superseded by: not stated in this wiki.