Noise residual features

Overview

A noise residual is what remains of an image after a high-pass filter removes its smooth, colour-dominated content. The authors' claim is that a manipulated region can leave statistical traces in this residual that the RGB image does not show, so a detector that reads only colour can latch onto the wrong cues 2103.12376 §3.1 2103.12376 §3.3. The filters are SRM kernels, borrowed from steganalysis, applied to the input image and again to intermediate feature maps 2103.12376 §4.1. The kernel set is not stated 2103.12376 §4.1.

The shared idea across the design options is the same: keep a colour stream for low-frequency texture and add a residual stream for high-frequency traces, then decide how the two interact 2103.12376 §4. The residual can also steer the colour stream with attention 2103.12376 §4.2. For the neighbouring task of locating tampered regions, see Image manipulation detection.

Variants

The one paper card here ablates its design one module at a time, so the variants below are its ablation rows, trained on FF++ F2F and tested on other FF++ methods 2103.12376 tab. 3. Their paper does not name them as variants.

SRM noise stream alone

Only the high-pass branch is used. On the F2F-trained ablation it scores AUC 0.758 on DF, 0.913 on FS and 0.858 on NT, against 0.803, 0.762 and 0.696 for the RGB Xception baseline 2103.12376 tab. 3. Card: Generalizing Face Forgery Detection with High-frequency Features.

Two-stream fusion

An RGB stream and a noise stream are combined before classification. The ablation gives AUC 0.810 on DF, 0.922 on FS and 0.894 on NT [arxiv:2103.12376 tab. 3]. Card: Generalizing Face Forgery Detection with High-frequency Features.

Fusion with residual-guided spatial attention

A spatial attention map computed from the residual gates the RGB features before fusion 2103.12376 §4.2 2103.12376 eq. 2. The ablation gives AUC 0.819 on DF, 0.966 on FS and 0.927 on NT 2103.12376 tab. 3. Card: Generalizing Face Forgery Detection with High-frequency Features.

Fusion with dual cross-modality attention

The two streams exchange information through attention over all positions. The correlation matrix between them is HW × HW 2103.12376 §4.3. The ablation gives AUC 0.801 on DF, 0.975 on FS and 0.957 on NT [arxiv:2103.12376 tab. 3]. Card: Generalizing Face Forgery Detection with High-frequency Features.

Full model: multi-scale noise features

The noise features are taken at several scales inside the network, not only from the input residual 2103.12376 §4.1 2103.12376 eq. 1. With RSA and DCMA added, the ablation gives AUC 0.837 on DF, 0.987 on FS and 0.984 on NT 2103.12376 tab. 3. Card: Generalizing Face Forgery Detection with High-frequency Features.

When to use which

Implementation sketch

Common core of the two-stream design, in our own words. Steps 1 and 2 are the noise variant; step 3 is the RSA variant; step 4 is the fusion. The paper's DCMA would replace the plain concatenation in step 4 with cross-stream attention, applied twice 2103.12376 §4.3.

# Forward pass of a two-stream noise-residual detector  [§4]
# Input:  face crop rgb (B, 3, 256, 256)  [§5.1]
# Output: manipulation score  [§4.3]

def detect(rgb):
    # 1. High-pass residual of the input via fixed SRM kernels  [§4.1]
    residual = srm_filter(rgb)
    # 2. Colour and noise features; the paper takes these at several scales  [§4.1]
    rgb_feat = rgb_encoder(rgb)
    noise_feat = noise_encoder(residual)
    # 3. Residual-guided attention gates the colour features  [§4.2]
    rgb_feat = spatial_attention(residual) * rgb_feat
    # 4. Fuse both streams with channel attention, then score  [§4.3]
    fused = channel_attention(concat(rgb_feat, noise_feat))
    return classifier(fused)  # trained with AM-Softmax [§4.3]

Not stated and not to be filled from elsewhere: the SRM kernel set, the number of scales, whether the filters are fixed or learned §4.1, the attention reduction ratio §4.3, and the margin and scale of AM-Softmax §4.3.

Contradictions

Only one paper card lists this concept, so there is no cross-paper disagreement to report. Within the paper:

• The ablation text says performance "gradually improves" with each module 2103.12376 §5.2. On DF, adding DCMA lowers AUC from 0.819 to 0.801 2103.12376 tab. 3.
• The Xception baseline differs between two tables: the DF-trained FS cell is 0.485 in 2103.12376 tab. 1 and 0.490 in 2103.12376 tab. 5, and the NT-trained DF cell is 0.800 and 0.799 respectively. The text does not explain the difference.

Papers

• Generalizing Face Forgery Detection with High-frequency Features: a two-stream detector with multi-scale SRM noise features, residual-guided attention and cross-stream attention; author-reported cross-database AUC gains over Xception and Face X-ray.