Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection

In plain words

A plain classifier that learns to label real and fake faces only has to find a line between the two groups. It does not have to keep the real faces tightly grouped, and the fakes can scatter in many different ways. This paper adds a second training rule. It pulls the real faces toward one central point and only asks the fake faces to sit farther from that point than the real ones, without squeezing the fakes together. That fits how fakes vary, since each faking method leaves its own traces.

The paper also gives the network a second view of each image: a block-based transform of the colours and brightness, the same kind of step used in JPEG compression. A small learned module reads this view and picks out the frequency patterns that separate real from fake, instead of relying on fixed hand-made filters. Both ideas together give better scores on a standard face-forgery video benchmark. The authors say the method may not handle forgery methods that it was not trained on.

TL;DR

• Adds a single-center loss to the usual softmax loss for face forgery detection: real faces are pulled to one center, fake faces are pushed beyond a margin.
• Adds a learned frequency branch on YCbCr 8x8 DCT coefficients, fused into an ImageNet-pretrained Xception §3.1 §3.2.
• Author-reported on FF++ c40: video AUC 0.924 and pAUC(0.1) 0.810, against 0.861 and 0.652 for their Xception baseline tab. 4.
• Training details beyond the architecture and loss (optimizer, learning rate, batch size, epochs) are not in the published text §4.1.
• No official code was found.

Problem

Face forgery detectors usually train with softmax loss over real and fake labels. The paper argues that softmax does not explicitly make features compact within a class or separated between classes §1 §3.3. Metric losses such as triplet and center loss do enforce compactness, but they also compress the fake faces from many manipulation methods into one cluster, and the paper argues these fakes differ too much from each other for that to work §1 §3.3. Separately, detectors that use frequency information often rely on fixed filter banks or hand-crafted features, which the paper argues cannot capture fake patterns across varied backgrounds, ages, skin colours and manipulation methods §1 §3.2.

Method

Overview §3.1 fig. 2. Two inputs are processed at the same time: the RGB image and a frequency representation produced by the data preprocessing below. The two streams are fused early, between the entry flow and the middle flow of Xception §4.1. The fused features go through the rest of Xception to an embedding, and a two-node fully connected layer classifies real or fake §4.1. Training minimises the softmax loss plus the single-center loss weighted by a factor λ eq. 6.

Data preprocessing (part of AFFGM) §3.2 fig. 4.

1. Convert the RGB image to YCbCr §3.2.
2. Apply a 2D DCT to each 8x8 block of each channel §3.2. The paper notes that these two steps are also used in JPEG-style compression and says this should make the method faster to run and more compatible with compression traces. The compatibility claim is not tested in the paper.
3. Group the coefficients of the same frequency index across all blocks into one channel, keeping their spatial layout §3.2 fig. 3. An input of H x W x 3 becomes H/8 x W/8 x 192 §3.2.
4. Normalise each of the 192 channels with the mean and variance of the training set §3.2. The paper says most energy sits in low frequencies, while middle and high bands matter more for detection §3.2.

Adaptive frequency information mining block (AFIMB) §3.2 fig. 2.

1. 3x3 convolution with three groups, so Y, Cb and Cr are processed separately §3.2.
2. Ordinary 3x3 convolution, then max pooling §3.2. This mixes information across the Y, Cb and Cr channels §3.2.
3. Channel attention: a max-pooling layer followed by two linear layers §3.2.
4. A final 1x1 convolution §3.2.

Fusion §4.2.2 tab. 2. The RGB and frequency features are fused with a 1x1 convolution, groups = 1, which gives the best AUC and pAUC in the paper's fusion ablation tab. 2. The paper also states that plain concatenation or sum gives good results §4.2.2.

Single-center loss (SCL) §3.3 eq. 1 eq. 2 eq. 3. Let D be the embedding dimension and f_i the embedding of sample i. There is one parametric center C for the real faces, and the paper defines M_nat and M_man as mean Euclidean distances from the real and fake embeddings to C within a batch eq. 2 eq. 3.

L_sc = M_nat + max(M_nat - M_man + m*sqrt(D), 0) eq. 1

The margin is m*sqrt(D) because the paper designs m as a scale factor relative to the square root of the feature dimension §3.3 §4.2.1.

The center C is randomly initialised and updated from mini-batches at each iteration §3.3. The paper gives the gradients with respect to f_i eq. 4 and with respect to C eq. 5, with an indicator 1[L > 0] that turns the margin term on or off, where L = M_nat - M_man + m*sqrt(D) §3.3. The paper gives these gradients for label 0 (real) and label 1 (fake) samples.

Joint loss §3.3 eq. 6. L_total = L_softmax + λ * L_sc. The softmax loss is defined as the last fully connected layer, the softmax and cross-entropy together §3.1 §1 footnote. The paper says the softmax loss also guides the update of the center, because a center updated only from mini-batches is unstable §3.3.

# Preprocessing: RGB image to frequency tensor  [§3.2]
# Input:  rgb (B, 3, H, W)
# Output: freq (B, 192, H/8, W/8)
def preprocess(rgb):
    # Convert RGB to YCbCr  [§3.2]
    ycc = rgb_to_ycbcr(rgb)                         # (B, 3, H, W)
    # 2D DCT on each 8x8 block of each channel  [§3.2]
    coeffs = block_dct_8x8(ycc)                     # (B, 3, 64, H/8, W/8)
    # Group equal-frequency coefficients across blocks into channels  [§3.2]
    grouped = reshape(coeffs, (B, 3 * 64, H // 8, W // 8))
    # Normalise each channel with training-set mean and variance  [§3.2]
    return (grouped - train_mean) / train_std

# Adaptive frequency information mining block  [§3.2]
# Input:  freq (B, 192, H/8, W/8)
# Output: frequency features (B, C, h, w); C, h, w not stated
def afimb(freq):
    # 3x3 convolution in three groups: Y, Cb and Cr separately  [§3.2]
    x = conv3x3_groups3(freq)
    # Ordinary 3x3 convolution, then max pooling  [§3.2]
    x = max_pool(conv3x3(x))
    # Channel attention: max pooling, then two linear layers  [§3.2]
    attention = linear2(linear1(max_pool(x)))
    # How attention is applied to x is not stated  [§3.2]
    x = combine(x, attention)
    # Final 1x1 convolution  [§3.2]
    return conv1x1(x)

# Single-center loss  [§3.3] [eq. 1] [eq. 2] [eq. 3]
# Input:  emb (B, D); labels (B,), 0 = real, 1 = fake
# center: learnable vector (D,), updated from mini-batches  [§3.3]
def single_center_loss(emb, labels, center, m, D):
    # Euclidean distance of each embedding to the center  [§3.3]
    dist = norm(emb - center, axis=1)               # (B,)
    # Mean distance of real samples  [eq. 2]
    m_nat = mean(dist[labels == 0])
    # Mean distance of fake samples  [eq. 3]
    m_man = mean(dist[labels == 1])
    # Margin scales with the square root of the embedding dimension  [§3.3]
    margin = m * sqrt(D)
    # Pull real samples to the center; fake samples at least margin farther  [eq. 1]
    return m_nat + max(m_nat - m_man + margin, 0)

# Total training loss  [§3.1] [§3.3] [eq. 6]
# Input:  logits (B, 2), emb (B, D), labels (B,), center, lam, m
def total_loss(logits, emb, labels, center, lam, m, D):
    # Softmax loss: two-node FC, softmax and cross-entropy  [§3.1]
    ce = cross_entropy(logits, labels)
    # L_total = L_softmax + lambda * L_sc  [eq. 6]
    return ce + lam * single_center_loss(emb, labels, center, m, D)

Hyperparameters

Results

Author-reported, FF++ dataset, 20 frames per manipulated video and 80 per original video §4.1. The Xception baseline is the authors' own run; Two-branch is video-based tab. 4.

Main comparison tab. 4:

* Differs from the reported value in the fourth decimal; the paper's supplementary material has exact values tab. 4. Bold marks the best result in the paper's table, and FDFL is bold in every column tab. 4.

Ablation, c40 tab. 3:

Loss comparison, c40 tab. 1, all with softmax loss:

Fusion module, c40, softmax only tab. 2:

Other reported results: the paper's t-SNE plots show that the SCL and softmax combination gives compact real-face clusters and looser fake-face clusters, and that the center loss on its own overfits to some extent §4.2.1 fig. 6. Face generalisation to unseen manipulation methods is weaker, according to the paper's supplementary material, which is not in the source §5.

Paper vs code

No official code. The arXiv LaTeX source has no repository link (the only URL is a CVPR template comment). GitHub repository search with the method name, the title words and the author-group terms found no repository from the authors. One third-party PyTorch reimplementation of the single-center loss exists (Amadeus-AI/pytorch-single-center-loss, MIT); it is not an author repository, so it was not used. No web search was run.

Pitfalls

• Training recipe missing from the published text. Optimizer, learning rate, schedule, batch size, epochs, weight decay, seed and augmentation are all deferred to the supplementary material §4.1. That material is not in the LaTeX source, so an implementer must choose them and report the choice. The TeX source also contains commented-out draft text with optimizer settings; they are not in the published paper and are not used on this card.
• Margin depends on the embedding dimension. The margin is m * sqrt(D) eq. 1, so it is about 9.49 for m = 0.3 at D = 1000. The swept values 0.05 to 0.35 correspond to margins of about 1.58 to 11.07 (derived). Changing the backbone changes D and therefore the margin, unless m is rescaled.
• Embedding is the 1000-d FC output, not the penultimate feature. The paper keeps Xception's final FC layer as the embedding §4.1, so D = 1000 comes from that layer.
• Empty classes in a batch. M_nat and M_man are means over the real and fake samples in the batch eq. 2 eq. 3. If a batch has no fake or no real sample, the loss is undefined. The paper does not say how to handle this, and the batch size is not stated.
• Center update is under-specified. C is random at start and updated from mini-batches §3.3. The paper gives the gradient with respect to C eq. 5, but not the learning rate for C, whether it is a learnable parameter, or the exact role of the softmax loss in updating it beyond the sentence in §3.3.
• Gradient scaling. Eq. 4 and eq. 5 divide by s and t, the counts of real and fake samples in the batch §3.3. Implementations that use autograd on eq. 1 get this automatically; hand-written gradients must keep the 1/s and 1/t factors.
• Fusion details missing. The paper does not say how the two streams are joined before the 1x1 convolution, how their sizes are matched at the fusion point, or the stride of the AFIMB layers §3.2 §4.1. The DCT input is H/8 x W/8, so the RGB branch and the frequency branch must line up somewhere.
• Channel attention form not stated. The paper names a max pooling and two linear layers, but not the activations, the reduction ratio or how the attention is applied §3.2.
• Video-level metrics. Video AUC and pAUC are averages of per-frame scores for each video §4.1. Reproduce this averaging before comparing numbers with the table.
• pAUC subscript undefined. The text says "pAUC" and the tables say pAUC(0.1); the false-alarm cut-off is not defined in the text §4.1.
• Sweep caption and text disagree on m. The text says the λ sweep is robust for values 0.001 to 1 §4.2.1, and the caption of fig. 5 says m is fixed at 0.1 for that sweep. The final model uses m = 0.3 and λ = 0.5 §4.2.1. Values of the sweeps are only in the figure, not in the text.
• AUC gain stated differently. The text says the c40 AUC improvement is 6.4% §4.3. The table gives 0.924 - 0.861 = 0.063, which is 6.3 points. The pAUC gain of 15.8 points and accuracy gain of 2.86 points match the table §4.3 tab. 4.
• Softmax-only AUC differs between tables. tab. 1 gives 0.860 for softmax only, and tab. 3 and tab. 4 give 0.861 for the same baseline.
• Baseline differs from published Xception numbers. At c40 the paper's baseline reaches 86.14% accuracy, while the Xception row of tab. 4 reports 81.00% for the same dataset. The paper says it uses fewer training frames §4.1. Treat cross-paper comparisons with care.
• Costs and limits. The method needs both streams and a per-batch loss on the embedding. The paper acknowledges weak generalisation to unseen manipulation methods and that it ignores inter-frame information in video §5. Frequency features are claimed to be compatible with compression traces, but the paper does not test this §3.2.

Lineage

• Builds on: Xception [chollet2017xception] as the backbone; center loss [wen2016discriminative] and triplet loss [schroff2015facenet] as the metric losses compared against; FF++ [rossler2019faceforensics++] as the dataset.
• Compared against: Two-branch [masi2020two], the closest method on frequency and compact-loss ideas. The paper says its center is fixed and it constrains absolute distances, where SCL's center is updatable and it constrains the distance gap between real and fake samples §2.
• Superseded by: not known from content in this wiki.