Learning Self-Consistency for Deepfake Detection

In plain words

A face-swap video can be built by pasting a face from one source onto a face from another. The pasted part and the rest of the picture carry different traces of where they came from, even when the result looks real. This paper trains a network to notice when different parts of a face image disagree about their origin.

The network compares every small patch of the image with every other patch and predicts whether each pair looks like it came from the same source. A second head then says whether the whole frame is real or fake. Training needs to know which pixels were pasted, and that is rarely labelled. The authors avoid the labelling by pasting real faces onto real faces themselves, with random blobby masks and blending, so the masks are known for free. The authors report that this detector generalises better to forgeries it was not trained on.

TL;DR

• Binary deepfake detector for face images and video frames, based on a ResNet-34 backbone §4.1.
• Adds a pair-wise patch-consistency loss (PCL) that predicts a 4D similarity volume over all patch pairs §3.1 eq. 1 eq. 3.
• I2G generates forged training frames from real frames, with pixel masks, so no manual mask labels are needed §3.2 alg. 1.
• Author-reported AUC is higher than the prior state of the art in-dataset (FF++, CD2, DFDC-P) and in cross-dataset tests tab. 1 tab. 4 tab. 5.
• Key caveat: the paper's own code is not public (see Paper vs code), so all numbers are author-reported and not replicated.

Problem

Deepfake detectors that learn visible artifacts, such as blending boundaries or warping, tend to fail on new forgery methods that remove those artifacts §1. Pixel-level manipulation masks, which would allow supervised localisation, are rarely available in deepfake datasets, and labelling them by hand is laborious §1 §3.2. The paper assumes that stitched forgeries keep distinct source features in each region, so the features disagree within a forged image and agree across a pristine one §1.

Method

The method has two parts: a detector (PCL, §3.1) and a training-data generator (I2G, §3.2). The detector has a shared backbone with two branches.

Backbone and source features. A ResNet-34, initialised from ImageNet weights, is cut after its conv3 layer. For an input of size H×W×3 this yields a feature map F of size H′×W′×256 with H′ = H/16 and W′ = W/16. Each feature vector covers a 16×16 pixel patch §4.1.

Consistency branch (PCL). For each pair of patches i, j the consistency score is

s(f_i, f_j) = σ(θ(f_i) θ(f_j) / √C′) eq. 1

where θ is a 1×1 convolution embedding, C′ is the embedding dimension and σ is the sigmoid. Computing this for every pair gives the 4D consistency volume V̂ of size H′×W′×H′×W′ §3.1. The paper fuses V̂ into a 2D heatmap M̂ of size H′×W′, which is upsampled to H×W for visualisation. The fusion operation is not specified §3.1.

Ground-truth volume. The manipulation mask 𝓜 (H×W) is bilinearly downsampled to 𝓜 of size H′×W′. For each patch (h, w) the ground-truth consistency map is

M^(h,w)(h′,w′) = 1 − |𝓜(h,w) − 𝓜_(h′,w′)| eq. 2

A value near 1 means the two patches agree, near 0 means they disagree. Stacking these maps over all patches gives V. A pristine image has V = 1 everywhere §3.1.

Consistency loss. Binary cross-entropy between V and V̂, averaged over all N = (H′W′)² entries:

L_PCL = (1/N) Σ BCE(V, V̂) eq. 3

with h, h′ ranging over 1..H′ and w, w′ over 1..W′ §3.1. With H = W = 256 this gives H′ = W′ = 16 and N = 65,536 §4.1. The grid size follows from the stated input size and stride, and is not written in the paper.

Classification branch. The source feature map passes through one more convolution, then global average pooling and a fully connected layer give a real/fake probability. The loss is a two-class cross-entropy L_CLS §3.1. The kernel size of the extra convolution is not stated.

Joint loss.

L = λ L_PCL + L_CLS eq. 4

λ defaults to 10 §4.1. The ablation shows large λ values help generalisation §3.1 §4.5.

Training sample. Each sample is (X, V, y): frame, consistency volume and label (y = 0 real, y = 1 fake). Real frames have V = 1. Fake frames from existing deepfake datasets get a mask from the DSSIM between the fake frame and its real counterpart, blurred and thresholded, and then V from eq. 2 §4.2. The blur size and threshold are not stated.

I2G (inconsistency image generator). I2G creates a forged frame X_g from a real target frame X_t and its mask 𝓜, following the steps below alg. 1 §3.2:

1. Detect 68 facial landmarks on X_t alg. 1.
2. Pick a random source frame X_s from a different identity whose landmarks are close to those of X_t in L2 norm, below a threshold ε alg. 1. The value of ε is not stated.
3. Align X_s to X_t using the landmarks alg. 1.
4. Take the convex hull of the landmarks of X_t as the initial mask §3.2 alg. 1.
5. Deform the hull elastically: random displacements drawn from a Gaussian with standard deviation between 6 and 12 px, placed on a coarse 4×4 grid, interpolated per pixel with bicubic interpolation §3.2.
6. Blur the deformed mask with a Gaussian kernel of size 16 §3.2. This gives 𝓜.
7. Blend the masked face region of X_s into X_t using one blending method chosen at random from a set. The set is not enumerated §3.2.
8. Apply augmentation: JPEG compression, Gaussian noise or blur, brightness and contrast change, random erasing and colour jitter §3.2 §4.1. Their parameters are not stated.

During training, each real sample has a 50% chance of being replaced by an I2G fake, with V computed from its mask by eq. 2 §4.2. The cross-dataset setting trains only on real FF++ frames, with I2G fakes §4.2.

Inference. Frame scores are averaged per video. Unless noted, reported results are video-level §4.2.

# PCL training step, joint loss  [§3.1] [eq. 3] [eq. 4]
# Input:  frames (B, 3, H, W), H = W = 256; volumes (B, H', W', H', W')
#         labels (B,) with 0 = real, 1 = fake
# Output: scalar loss

def training_step(frames, volumes, labels):
    # Source features after ResNet-34 conv3  [§4.1]
    source = backbone_conv3(frames)               # (B, 256, H/16, W/16)
    # Embed each patch with a 1x1 convolution  [§3.1]
    embedded = theta(source)                      # (B, C', H', W')
    # Pairwise similarity of every patch pair  [§3.1] [eq. 1]
    volume_hat = sigmoid(pairwise_dot(embedded) / sqrt(C'))
    # Consistency loss, averaged over all entries  [eq. 3]
    loss_pcl = mean(binary_cross_entropy(volume_hat, volumes))
    # Classification head: conv, global average pool, FC  [§3.1]
    logits = fc(global_avg_pool(conv_cls(source)))
    # Two-class cross-entropy  [§3.1]
    loss_cls = cross_entropy(logits, labels)
    # Weighted sum, lambda = 10 by default  [eq. 4] [§4.1]
    return lambda_ * loss_pcl + loss_cls

# I2G: make one forged frame and its mask from a real frame  [alg. 1] [§3.2]
# Input:  target frame (H, W, 3)
# Output: forged frame (H, W, 3), mask (H, W)

def i2g(target):
    # 68 facial landmarks of the target  [alg. 1]
    target_points = landmarks(target)             # (68, 2)
    # Random frame of another identity with close landmarks (L2 < epsilon)
    # epsilon is not stated  [alg. 1]
    source = sample_other_identity(target_points)
    # Align the source face to the target using landmarks  [alg. 1]
    source = align(source, target_points)
    # Convex hull of the target landmarks  [§3.2] [alg. 1]
    mask = convex_hull(target_points)
    # Elastic deformation: Gaussian displacements (std 6-12 px) on a
    # 4x4 grid, bicubic per-pixel interpolation  [§3.2]
    mask = elastic_deform(mask)
    # Gaussian blur with kernel size 16  [§3.2]
    mask = gaussian_blur(mask, kernel_size=16)
    # Blend the source face into the target with one random method  [§3.2]
    forged = blend(target, source, mask, method=random_method())
    # Augmentations: JPEG, noise or blur, brightness/contrast, erasing,
    # colour jitter; parameters not stated  [§3.2] [§4.1]
    forged = augment(forged)
    return forged, mask

Hyperparameters

Results

Author-reported. Setup: in-dataset means train and test on the same dataset's train and test splits; cross-dataset means training on real FF++ frames with I2G fakes, then testing on unseen datasets §4.3 §4.4. Model: ResNet-34 + PCL + I2G. Baselines are cited from the respective papers, with their own backbones and protocols §4.3.

In-dataset, AUC (%) — train and test on the same dataset.

Average over FF++, CD2 and DFDC-P: prior best 96.45 → 98.05 §4.3 (the average is computed from the three per-dataset values).

Cross-dataset, FF++ training, AUC (%)

Five-dataset average over FF++, DFD, CD1, CD2 and DFDC-P: prior best 86.03 → 92.18 §4.4. DFR and DFDC are excluded because they have no published baselines §4.4.

Cross-dataset, AP (%) on CD1 and DFDC-P: Face X-ray 73.33 / 72.65; this paper 98.97 / 82.94 tab. 6.

Video-level AUC / AP / EER (%), cross-dataset app. Video-Level Comprehensive Results

Frame-level AUC on CD2, cross-dataset (FF++ training): this paper 81.8; Masi et al. 73.4; Xception-c23 65.3 app. Frame-Level Results on Celeb-DF-v2.

Ablation: loss weight λ, cross-dataset AUC (%) tab. 7

Ablation: I2G, AUC (%) tab. 8

Other ablations (text, not in a table).

• Patch size, cross-dataset model on FF++: 4×4 98.32, 8×8 98.35, 16×16 99.11, 32×32 98.74 §4.5.
• Backbone, average AUC over DFR, CD2, DFDC, DFDC-P: ResNet-18 75.74, ResNet-34 82.83, ResNet-50 83.90, ResNet-152 82.64 tab. 11.
• Pre-training only (DF real + I2G, then fine-tuned on DF): average 84.89 vs 79.19 without pre-training §4.5.

Parameters and compute. The PCL head adds 65.5 K parameters to the 21.3 M of the full model app. Computational Complexity.

Paper vs code

No official code. I searched the paper, the LaTeX source (no repository links, only dataset and tool links), and GitHub by method name, title words, and the author organisation (Amazon, org:amzn). The matches were jhaotingc/PCL-I2G, labelled by its owner as an unofficial implementation with no license, and Dreemurr-T/Deepfake-Detection-based-on-self-consistency, a personal repository under MIT. Neither is an author's repository, so neither was used. No code was read, and no code-based discrepancies are claimed.

Pitfalls

• Official code is absent. Every hyperparameter here comes from the paper alone. Re-implementations will have to choose the unstated values listed in the Hyperparameters table, and results are unlikely to transfer exactly.
• Heatmap fusion is unspecified. The 2D global map is "fused" from the 4D volume §3.1, but the operation is not written down. The appendix describes a different visualisation (grouping by cosine similarity to the top-left patch) app. Consistency Map Generation. Do not use that for scoring.
• Eq. 1 normalisation is ambiguous. The dot product is divided by √C′, where C′ is the embedding size, not the channel count C of F. Embedding layer details beyond "1×1 convolution" are not given.
• Loss normalisation is stated but ambiguous in one respect. Eq. 3 averages over all N = (H′W′)² entries, so the loss is a mean over 65,536 entries per image at 256×256 input. Per-sample or per-batch reduction is not stated. For real frames the target volume is all ones §4.2, so those samples carry only positive targets.
• Positive-class imbalance is inherent. Pristine frames give all-ones volumes, and I2G fakes use only the masked region. The effect of this imbalance on training is not discussed.
• I2G's landmark threshold ε is not stated. Without it, the rate at which a valid source frame is found, and so the difficulty of the pairs, is unknown.
• I2G's blending set is not enumerated. Only "various blending methods" §3.2 are cited, to Pérez et al. and Naruniec et al. Pick and document specific ones.
• DSSIM mask generation is under-specified. The blur size and threshold for the real-versus-fake mask are not stated §4.2.
• The LR schedule's middle phase is unstated. Only the start and end of the ramp are given §4.1.
• Cross-dataset numbers depend on the protocol. Training uses only real FF++ frames augmented by I2G, following Face X-ray §4.2. The in-dataset protocol trains with I2G on by default §4.2, so the two sets of numbers are not comparable.
• Frame-level versus video-level. Most results are video-level averages of frame scores §4.2. Frame-level CD2 (81.8) uses a different protocol and comparison set app. Frame-Level Results on Celeb-DF-v2.
• Numerical inconsistencies in the paper.
  • The caption of tab. 4 says the DFD gain is 5.67%, but 99.07 − 93.47 = 5.60.
  • The caption of tab. 5 and the text give DFR AUC as 99.51, but tab. 5 and the appendix list DFR AUC as 99.41. The value 99.51 is the DFR AP app. Video-Level Comprehensive Results.
  • Cross-dataset results on CD1 and CD2 are described as "about 18% and 13%" above the prior best; the table values give 17.72 and 13.38.
• Table and figure numbers follow source order. The LaTeX source has no compiled PDF. Numbers such as tab. 1–tab. 12 and fig. 1–fig. 6 are assigned by source order and float placement may differ in the printed paper. The appendix has no printed section numbers in the source, so it is cited by subsection title.
• Hidden cost. The paper reports 0.0234 s per image on one V100 with ResNet-34, of which PCL is 0.0009 s app. Computational Complexity. The quadratic term grows as (H/16)² × (W/16)², so larger inputs cost much more than the reported figures suggest.
• Forgery arms race. The authors note that whole-image synthesis, which the paper hypothesises produces self-consistent images, may defeat the method, and that low-quality or heavily compressed samples cause false negatives §4.7 §4.6.
• Benchmark caveat. DFDC and DFDC-P are very low quality, and every method in the tables performs poorly there §4.4.

Lineage

• Builds on: the dot-product pairwise similarity, cited to Wang et al. [wang-cvpr18] §3.1. The I2G elastic deformation is cited to Ronneberger et al. [ronneberger-icm15] §3.2. The cross-dataset protocol follows Face X-ray [li-cvpr20oral] §4.4.
• Compared against: Face X-ray [li-cvpr20oral], S-MIL-T [li-acm20], Masi et al. [masi-eccv2020], Tolosana et al. [tolosana-arxiv20], XN-avg [roessler-iccv19], Fakespotter [wang-arxiv19], MIL [wang2018revisiting], Dang et al. [dang-cvpr20] tab. 1 tab. 5.
• Superseded by: nothing in this wiki.