Generalizing Face Forgery Detection with High-frequency Features

In plain words

A face-swap detector that learns the look of one forgery method often fails on the next method, because it has latched onto that method's colour texture. The authors show this for a standard CNN classifier: it does well on the forgeries it was trained on and poorly on forgeries made with other tools.

Their fix is to look at what remains once the colour is removed. A high-pass filter keeps only the fine, noisy variation in an image, and the seam where a fake face was pasted in can stand out there. The model therefore runs two streams side by side: the usual colour stream and a noise stream. Each stream is computed inside the network at several scales, and the two streams are wired together so that the noise guides attention in the colour stream and the two streams exchange information.

The authors report that this generalizes better than the baselines across several public deepfake datasets. Those numbers are their own, measured under their protocol, so check them before relying on them.

TL;DR

A two-stream face forgery detector: a modified Xception RGB stream and an SRM high-pass noise stream computed at several scales §4.1. The streams are linked by residual-guided spatial attention on the RGB side §4.2 and dual cross-modality attention between the streams §4.3. Trained on FF++ (HQ) and tested cross-database, the authors report higher AUC than Xception and Face X-ray on DFD, DFDC, CelebDF and DF1.0 tab. 4. Relevant if you need a cross-dataset deepfake detector; many training details (training length, SRM kernels, attention sizes) are not stated §5.1.

Problem

Detectors that reach high accuracy on the forgery method they were trained on lose much of it on forgeries from other methods. The authors call this the generalization problem §1. Xception trained on one FF++ method reaches AUC between 0.485 and 0.888 on the other three methods' test sets tab. 1.

The authors argue the cause is texture bias. Grad-CAM maps of an Xception trained on Face2Face concentrate on the mouth in its own forgeries, but on unseen forgeries the maps resemble those on real faces, and the model calls them real §3.1 fig. 2. Their analysis rests on this one training method and one CNN, so the generality of the bias claim is the authors' inference §3.1.

Noise residuals remove colour content and expose statistical traces of manipulation, which the authors say are often left in noise space by manipulation §3.3 fig. 4.

The authors also note that every manipulation has a generation stage, which varies by method, and a blending stage, where the face is warped back into the source image and the background stays from the source. They focus on the blending stage, which they argue violates the original data distribution because the background stays from the source §3.2 fig. 3. Face X-ray is cited as a method that targets the same blending step by detecting the boundary §3.2 §5.3. The authors argue that boundary patterns change with post-processing, so they look at noise instead §3.2.

Method

Overview. The input is an aligned face crop resized to 256 × 256 §5.1. An RGB stream carries low-frequency texture. A noise stream carries high-frequency residuals obtained with SRM filters, both at the input and at intermediate feature maps §4.1 fig. 5. The network has an entry flow, a middle flow and an exit flow fig. 5 caption. Three modules sit in these flows: multi-scale high-frequency feature extraction (entry flow, §4.1), residual-guided spatial attention (entry flow, §4.2) and dual cross-modality attention (middle flow, §4.3). A channel-attention fusion and an AM-Softmax classifier finish the model §4.3.

1. Multi-scale high-frequency feature extraction §4.1 eq. 1. The input image X is turned into a residual image X_h with SRM filters §4.1. Regular convolutions on X and X_h produce F^1 and F_h^1. The SRM filters are then applied to F^1, followed by a 1×1 convolution, to give F̃_h^1. The downsampled RGB map F^2 and the downsampled sum F_h^1 + F̃_h^1 (giving F_h^2) are passed on, and the same steps repeat. The output is the multi-scale noise feature F_h, which the authors say carries more high-frequency signal than the direct residual X_h §4.1. Eq. 1 defines the entry flow outputs F and F_h from X and X_h eq. 1.

2. Residual-guided spatial attention (RSA) §4.2 eq. 2. Inspired by CBAM, a spatial attention map M is computed from the residual image X_h, not from the RGB image §4.2 eq. 2. M is multiplied element-wise with the RGB map F^2, and the product goes into further convolutions and downsampling to give the RGB features F §4.2. The authors place several RSA modules in the entry flow, because the spatial correspondence between residuals and low-level RGB features is kept §4.2. The attention maps highlight abnormal facial boundaries in manipulated faces and are spread out over real ones §4.2 fig. 8.

3. Dual cross-modality attention (DCMA) §4.3 eq. 3. For each stream, the input T (RGB) or T_h (noise) gives a value V (V_h for noise), with C channels, and a key K with C/r channels. The key of each stream is built from two convolutions. The first convolution is separate for each stream; the second is shared, which puts both keys in one space §4.3. The correlation between the two streams is C = flt(K)^T ⊗ flt(K_h), a HW × HW matrix, where flt flattens the spatial dimensions §4.3. The RGB attention is A = softmax(C ⊗ W) and the noise attention is A_h = softmax(C^T ⊗ W_h) §4.3. The RGB output is T' = T + R with R = flt(V_h) ⊗ A, and the noise output is computed the same way with the roles swapped §4.3 eq. 3. DCMA is applied N = 2 times at different scales §4.3.

4. Fusion and loss §4.3 fig. 5. High-level features of the two streams are concatenated at the end of the exit flow. Channel-wise attention, cited to the squeeze-and-excitation work, is applied to the concatenation before the prediction §4.3. The loss is AM-Softmax, which the authors choose because it gives smaller intra-class and larger inter-class variation than cross-entropy §4.3.

Key equations.

\mathbf{F}, \mathbf{F}_h = f_{\text{entry}}(\mathbf{X}, \mathbf{X}_h) eq. 1

\mathbf{M} = f_{\text{att}}(\mathbf{X}_h) eq. 2

\mathbf{T}', \mathbf{T}_h' = f_{\text{DCMA}}(\mathbf{T}, \mathbf{T}_h) eq. 3

Pseudocode. Shapes assume H = W = 256 §5.1; everything else is the paper's description, and names are ours.

# Forward pass  [fig. 5] [§4]
# Input:  rgb_image (B, 3, 256, 256)  [§5.1]
# Output: class logits (AM-Softmax)  [§4.3]

def forward(rgb_image):
    # High-pass residual of the input image via SRM filters  [§4.1]
    residual = srm_filters(rgb_image)
    # Entry flow: RGB features and multi-scale noise features  [eq. 1]
    rgb_feat, noise_feat = entry_flow(rgb_image, residual)
    # Middle flow: DCMA applied N = 2 times, at different scales  [§4.3]
    for dcma_index in range(2):
        rgb_feat, noise_feat = dcma(rgb_feat, noise_feat)
    # Exit flow: channel-wise attention on the concatenated streams  [§4.3]
    fused = channel_attention(concat(rgb_feat, noise_feat))
    return classifier(fused)

# Entry flow: multi-scale noise features and RSA  [§4.1] [§4.2]
# Input:  rgb_image (B, 3, H, W); residual (B, C_s, H, W)  [§4.1]
# Output: rgb_feat (RGB stream); noise_feat (multi-scale noise stream)

def entry_flow(rgb_image, residual):
    # Regular convolutions on the image and on its residual  [§4.1]
    rgb_map = conv_rgb(rgb_image)
    noise_map = conv_residual(residual)
    # SRM filters on the RGB map, then a 1x1 conv to match channels  [§4.1]
    srm_map = conv_1x1(srm_filters(rgb_map))
    # Downsample the RGB map and the sum of the two noise maps  [§4.1]
    noise_map = downsample(noise_map + srm_map)
    rgb_map = downsample(rgb_map)
    # Repeat the two steps above for further scales (count not stated)  [§4.1]
    # Residual-guided attention gates the RGB map  [§4.2] [eq. 2]
    attention = spatial_attention(residual)
    rgb_map = attention * rgb_map
    # Convolutions and downsampling on the gated RGB map  [§4.2]
    rgb_feat = downsample(conv_rgb_out(rgb_map))
    return rgb_feat, noise_map

# Dual cross-modality attention  [§4.3] [eq. 3]
# Input:  rgb_feat T (C, H, W); noise_feat T_h (C, H, W)  [§4.3]
# Output: refined rgb_feat T' and noise_feat T_h'  [§4.3]

def dcma(rgb_feat, noise_feat):
    # Value (domain-specific information) for each stream  [§4.3]
    value_rgb = conv_value_rgb(rgb_feat)
    value_noise = conv_value_noise(noise_feat)
    # Key: first conv per stream, second conv shared  [§4.3]
    key_rgb = shared_key_conv(first_key_conv_rgb(rgb_feat))
    key_noise = shared_key_conv(first_key_conv_noise(noise_feat))
    # Correlation between the two streams over all positions  [§4.3]
    correlation = flatten(key_rgb).T @ flatten(key_noise)  # (HW, HW)
    # Attention maps; the weight matrices W and W_h are not defined  [§4.3]
    attention_rgb = softmax(correlation @ weight_rgb)
    attention_noise = softmax(correlation.T @ weight_noise)
    # Each stream receives the other's values, weighted by attention  [§4.3]
    # Restoring R to the input shape is not described  [§4.3]
    refined_rgb = rgb_feat + restore(flatten(value_noise) @ attention_rgb)
    refined_noise = noise_feat + restore(
        flatten(value_rgb) @ attention_noise
    )
    return refined_rgb, refined_noise

Hyperparameters

Results

Author-reported. Setup: five benchmark sets tab. 2; a modified Xception with SRM noise stream; baselines are Xception and Face X-ray. The authors implemented Face X-ray themselves, following its paper's instructions §5.3.

Cross-database: trained on FF++ (HQ), AUC tab. 4

Cross-method inside FF++, AUC tab. 5. Rows give the training method; columns give the test method. Gray cells are within-method.

The authors' own Xception row for DF-trained FS and NT-trained DF differs slightly from tab. 1 (see Pitfalls).

Ablation, trained on F2F, AUC tab. 3. The paper marks within-method (F2F) results in gray; the F2F column is the within-method column.

CelebDF, AUC tab. 8

Low-quality FF++ vs high-frequency methods, accuracy tab. 6. Video-level detection on LQ FF++, following each competitor's setup; the model is trained at image level §5.4.

Multi-task methods, accuracy, trained on F2F (HQ) tab. 7. Competitor numbers are copied from their papers, not rerun §5.4.

Paper vs code

No official code. The paper does not link a repository, and the LaTeX source contains no repository URL.

Pitfalls

• Missing architecture and training details. The two-stream structure and training settings are said to be in the supplementary material §5.1; that file is not in the source archive. Layer sizes, the number of SRM filters, the number of scales, the attention reduction ratio, training length, learning-rate schedule and weight decay are therefore not stated. Don't fill them from another paper.
• Unstated shapes in DCMA. The paper does not say how R is restored to the input shape, what shape W and W_h have, or whether they are learned §4.3. The HW × HW correlation makes memory grow with the square of the spatial size at each DCMA scale. The paper does not say at which scales DCMA sits §4.3.
• AM-Softmax parameters. Scale and margin are not stated §4.3. Use the loss from the cited AM-Softmax paper only as a starting point, and label it as such.
• Possible typo in the multi-scale step. The text writes "We down-sample F_1" where it means F^1 §4.1. Read it as the RGB map at scale 1.
• RSA scale matching. The residual X_h has full resolution, but RSA multiplies it into feature maps at lower resolutions. How M is resized to match is not described §4.2 eq. 2.
• Gradual-improvement claim. The ablation text says performance "gradually improves" with each module §5.2. It does on FS and NT, but on DF the DCMA row drops from 0.819 to 0.801 tab. 3.
• Xception baseline numbers differ between tables. The DF-trained FS cell is 0.485 in tab. 1 and 0.490 in tab. 5. The NT-trained DF cell is 0.800 in tab. 1 and 0.799 in tab. 5. The text does not explain the difference.
• CelebDF margin. The paper says the model beats the competitors "by more than 15%" in AUC §5.4 tab. 8. The gap to FFD is 0.150 (0.794 − 0.644), which is exactly 15 points, not more. The gap to FWA is 0.256.
• Cross-paper comparisons. The multi-task and high-frequency comparisons copy numbers from other papers, under different splits and metrics (accuracy rather than AUC) §5.4 tab. 6 tab. 7. Treat these as indicative, not as a like-for-like ranking.
• Face X-ray is reimplemented. The authors implemented Face X-ray themselves, following its paper §5.3. Their numbers are not the original paper's, and the details of their reimplementation are not stated.
• Compression. HQ is the default; the LQ comparison uses a different, video-level protocol with 25 frames per video §5.1 §5.4.
• Table and figure numbers come from source order. The compiled PDF is not in the source archive, so the numbers used here follow the order of LaTeX captions. Check against the PDF if a reader will compare.
• Blending-boundary claim. The argument that noise is better than boundary patterns rests on the authors' reasoning and on the comparisons above; the paper gives no test that isolates the boundary cue §3.2 §5.3.

Lineage

• Builds on: SRM high-pass filters from steganalysis [fridrich2012rich], cited for both the input residual and the feature-level residuals §4.1; spatial attention from CBAM, adapted so the residual drives the attention §4.2; cross-modal attention from Ye et al. and Hou et al., extended to two streams of one image §4.3; channel-wise attention from squeeze-and-excitation §4.3; AM-Softmax §4.3; Xception as the backbone §5.1.
• Compared against: Xception §5.3; Face X-ray, re-implemented by the authors §5.3; SRMNet, Bayar Conv, SSTNet and F3Net, which use high-frequency features §5.4; LAE, ClassNSeg and ForensicTrans, which use multi-task learning §5.4; FWA and FFD on CelebDF §5.4.
• Superseded by: not stated in this wiki.