Face Forensics in the Wild

In plain words

Most face-swap detectors are tested on clips with one person in them. Real footage often has several people on screen, and usually only one of them has been swapped. The authors built a large dataset of such videos: each has about three faces per frame, and only some of those faces are fake. Building it by hand would be slow, so a separate model scores each swapped face and throws away the poor ones.

The detector is trained with one label per video (real or fake), with no marks on individual faces. It looks at each person's face track over time and learns to give more weight to the tracks that look tampered. Those weights also point to which face was likely swapped. On the new dataset this beats the older methods, including ones trained on per-face labels.

TL;DR

• Dataset FFIW-10K: 10,000 fake and 10,000 real videos, about three faces per frame, only some faces swapped §3.3.
• Fake faces are made by three automatic swap tools, and a quality network (Q-Net) keeps only faces scored above 0.6 §3.2, §4.
• Detector: ResNet-50 face features, temporal aggregation per face track, attention over tracks, and a sparsity penalty. It learns from video labels only §5.
• Author-reported test result: 69.4% ACC, 70.9% AUC, 30.8% mAP, the best of the compared methods tab. 2.
• The released repository holds only a README and dataset form. Model code is not published, so the architecture must be built from this card README.md.

Problem

Earlier face-forgery benchmarks mostly use trimmed clips with one identity each. Detectors trained on them fail on multi-person footage, and the paper argues that this setting is the realistic one §1, §2. When a video has both real and swapped faces, a per-video "fake" label is inexact: it does not say which face is fake §1, §5.1. The authors also report that methods trained on face-level labels can diverge on their data, because most faces in a fake video are real §6. Building per-face labels by hand is costly, so the dataset needs an automatic quality check §1, §4.

In this paper "forgery" means altering imagery by swapping faces §1, footnote 1.

Method

The paper has two parts: a dataset pipeline (§3 and §4), and a detector for multi-person videos (§5). Supplementary material is cited as supp. §n and is a separate PDF in the source.

Dataset pipeline (§3, §4)

1. Collect. Search YouTube with keyword queries written by ten people in several languages. Keep videos at 480p or higher, which gives 4,000 raw videos §3.1.
2. Cut. Divide each raw video into four equal segments and take one random 12-second sequence from each. Drop static or crowded sequences and those with few faces. This leaves about 12,000 pristine sequences §3.1.
3. Track. Detect faces in each frame and link them into tracklets with an off-the-shelf detector and tracker §3.2, §5.5. For a swap, pick a target video and a source video, and keep the longest, highest-resolution tracklet from each §3.2.
4. Swap. Each video uses one of three tools: two learning-based (DeepFaceLab, FSGAN) and one graphics-based (FaceSwap) §3.2.
5. Score and filter. Q-Net scores each swapped face. A tracklet is kept only if the mean score of its swapped faces is above 0.6 §4. Low-scoring swaps are discarded. The paper selects 10,000 fake videos, each from a different pristine video §3.2.

Q-Net is a VGG16-based network with two heads §4, supp. §1:

• a score head that regresses a quality score in [0, 1] with an l1 loss;
• a domain head that predicts which of three GANs (StyleGAN, StyleGAN2, PGGAN) produced the face. Its cross-entropy enters the loss with a negative sign §4, eq. 1. This is the domain-adversarial part.

Q-Net needs labels. The authors get them without human annotation. They sample faces from three GANs trained on FFHQ at many training iterations, and use the iteration as a pseudo-label: score = 0.9 × n / N, where n is the iteration and N = 5,000 is the maximum §4. That gives 300,000 training samples §4.

Detector (§5)

The detector takes one face tracklet at a time and then combines tracklets into one video decision. Each tracklet Γ has T face regions, and each region is a feature vector x_t of dimension D = 2048 from a ResNet-50 backbone §5.2, §5.5.

Instance aggregation (per tracklet, §5.2). The tracklet matrix X (D × T) goes through three stages:

• Short-term: three dilated convolution layers with rates 1, 2 and 4, where each layer sees the outputs of all earlier layers through dense connections. Each layer is a 1×1, then 3×3 dilated, then 1×1 convolution, with batch normalization and ReLU §5.2, eq. 7. The output S is D × T eq. 4.
• Long-term: self-attention over the T time steps of S. The affinity A is a softmax of the dot products SᵀS, and the output is S·A + S §5.2, eq. 8–9.
• Global: max-pooling over time gives one vector Y_k per tracklet §5.2, eq. 6.

Bag aggregation (per video, §5.3). The video is a bag of K tracklets, each a vector Y_k. Attention weights a_k are a softmax over the K tracklets of wᵀ·tanh(Wᵀ·Y_k), where W and w are learned §5.3, eq. 11. The video vector is the weighted sum O = Σ a_k·Y_k §5.3, eq. 10. Tracklet k is called fake for localization when a_k > 0.75 §5.3.

Loss (§5.4). A fully connected layer on O gives a video logit, trained with binary cross-entropy against the video label. A sparsity term, the l1 norm of the attention vector, is added with weight β §5.4, eq. 12. The paper says the sparsity term encourages attention to pick the few faces that are fake §5.4, §6.4.

Pseudocode

The first block is the Q-Net training loss, the second the quality filter, and the third the detector. The paper gives no algorithm box, so these are written from §4 and §5 and the equations cited.

# Q-Net training loss for one batch  [§4] [eq. 1]
def q_net_loss(images, pseudo_scores, domains, alpha):
    # Predicted quality score and domain logits per image  [§4] [supp. §1]
    pred_score, pred_domain = q_net(images)
    # l1 regression to the pseudo score 0.9 * n / N  [§4]
    regression = l1(pred_score, pseudo_scores)
    # Cross-entropy over the three source GANs  [§4]
    domain_ce = cross_entropy(pred_domain, domains)
    # Domain term enters with a minus sign and weight alpha > 0  [eq. 1]
    return regression - alpha * domain_ce

# Quality filter for one manipulated tracklet  [§4]
def keep_tracklet(swapped_faces):
    # Q-Net score for every swapped face in the tracklet  [§4]
    scores = [q_net(face) for face in swapped_faces]
    # Keep only if the mean score is above 0.6  [§4] [supp. tab. 1]
    return mean(scores) > 0.6

# Forward pass for one video, K tracklets  [§5.2] [§5.3]
# Input: tracklets, list of K tracks, each T faces of shape (3, 224, 224)
# Output: video logit (scalar), attention a (K,)
def forward(tracklets):
    # 1. Encode each face with ResNet-50, average-pooled, D = 2048  [§5.5]
    features = [
        stack([backbone(face) for face in track])
        for track in tracklets
    ]
    # features[k] has shape (T, D)
    # 2. Short-term dilated dense block, rates 1, 2, 4  [§5.2] [eq. 4, 7]
    shorts = [short_term_block(f.T) for f in features]  # each (D, T)
    # 3. Long-term self-attention, softmax per column  [§5.2] [eq. 8-9]
    longs = [s + s @ softmax(s.T @ s) for s in shorts]  # each (D, T)
    # 4. Max-pool over time, one descriptor per tracklet  [§5.2] [eq. 6]
    Y = stack([max_over_time(l) for l in longs])  # (K, D)
    # 5. Attention pooling over the K tracklets  [§5.3] [eq. 10-11]
    a = softmax(tanh(Y @ W) @ w)  # (K,)
    O = a @ Y  # (D,)
    # 6. One fully connected layer gives the video logit  [§5.4]
    return fc(O), a

# Training loss for one video  [§5.4] [eq. 12]
def video_loss(logit, label, a, beta):
    # Binary cross-entropy against the video label  [§5.4]
    bce = binary_cross_entropy(logit, label)
    # l1 norm of the attention vector, weight beta  [§5.4] [§5.5]
    sparsity = sum(abs(a))
    return bce + beta * sparsity

Hyperparameters

Results

Author-reported. Setup: FFIW-10K test split (3,500 videos) §3.3, §6. Each video is cut into face tracklets. Frame-level baselines use the mean face score per tracklet §6. The baselines are trained on FFIW-10K with face-level labels, and S-MIL with video labels §6. ACC is video-level accuracy, AUC is the tracklet-level area under the ROC curve, and mAP is the localization metric §6.

Cross-dataset AUC (%), all trained on FF++ unless the row says FFIW-10K tab. 3:

Ablations on the test split tab. 4:

Other reported numbers:

• Validation split, supplement: this paper reaches 71.3% ACC, 73.5% AUC and 31.1% mAP [supp. tab. 2].
• Q-Net agrees with the three-observer consensus on 85.63% of 1,357 pairs [supp. §1].
• Q-Net score bins in human ratings, share rated high quality: 43.9% for [0.4, 0.5), 67.6% for [0.5, 0.6), 89.5% for [0.6, 0.7), 91.3% for [0.7, 0.8) [supp. tab. 1].
• In a 50-person rating study, more FFIW-10K videos are rated real than in Celeb-DF or DeeperForensics-1.0. No numbers are given in the main text §3.3, fig. 4.
• Visual localization examples are in fig. 7. Dataset comparison is in tab. 1.

Paper vs code

The author line links https://github.com/tfzhou/FFIW, the first hit in the search order. The repository belongs to the first author. At commit f7d7275691233d5338883b7d3f362b83701fe7fc it contains only a README, two images, two GIFs and a dataset-access form. It has no model, training or preprocessing code, no configs and no LICENSE file README.md.

• The paper says "our codes are released" §5.5. At this commit the repository does not contain them, so all method details come from the paper alone.
• The README cites arXiv 2103.04570 as the paper, not this card's id 2103.16076 README.md. Treat that link as stale.
• The README describes a dataset folder layout with source, target and target_mask splits README.md. The repository does not contain the dataset itself: it is released on request after a Terms-to-Use form.

Pitfalls

• The sparsity term has no effect as written. Attention a_k is a softmax over the K tracklets, so the weights sum to 1 §5.3, eq. 11. Then ‖a‖₁ = 1 for every video, and its gradient is zero. Yet the paper reports 28.5% → 30.8% mAP from the term tab. 4, §6.4. The paper does not explain this. Check against the released code, which is absent, or try a non-softmax normalization, and report which one you used.
• The 0.75 localization rule can flag at most one track. With weights that sum to 1, two values can't both exceed 0.75. The rule is also not tied to a reported metric: mAP uses the ranking of a_k §5.3, §6.
• The domain-adversarial sign is not implemented in the text. Eq. 1 writes L_Q = Σ l1 − α·CE for the domain head §4, eq. 1. Minimizing −α·CE jointly with the rest is unbounded: the domain head can lower the loss by being confidently wrong. The text does not say how it is done. Gradient reversal, as in Ganin et al. (2016), is one standard way to get the adversarial effect.
• Self-attention is parameter-free. Eq. 8 uses SᵀS directly, with no learned query, key or value projections and no scaling by √D §5.2. The text calls the dot product a "normalized correlation", which it isn't. Reproduce it exactly as written, or add projections and say so.
• Output of the short-term stage is ambiguous. Eq. 4 types S as D × T. That suggests S is the last layer's output S_L, not the concatenation of all layers (which would be (L+1)·D tall). The text does not say which §5.2.
• Q-Net output size differs between main text and supplement. The supplement gives the score head as FC(1024)→FC(1024)→FC(2) [supp. §1], but the main text uses a scalar score in l1 §4. Take the first output as the score, or check the supplement's architecture before building it.
• α schedule uses epochs, pseudo scores use iterations. The supplement's α schedule takes n and N as epochs [supp. §1]. The pseudo score 0.9 · n/N uses iterations with N = 5,000 §4.
• Missing training details. Not stated: T and K, the attention width C, whether the ResNet-50 is ImageNet-pretrained, epochs, LR schedule, weight decay, batch unit, augmentation magnitudes, detector and tracker settings. A reimplementation must choose these and report them.
• The dataset totals count fake videos only. About 12 s × 10,000 videos gives 33 hours. The 7.2M frames §3.3 implies about 60 frames per second. The real videos' total duration is not stated.
• Quality filter acceptance rate is unknown. The paper does not state how many swaps were made before filtering, so the rejection rate is unknown §3.2.
• Dataset rights are separate. The paper is CC BY 4.0, but the dataset comes under the request-based Terms-to-Use form README.md. This card does not use or reproduce dataset contents.
• Baseline numbers may be re-runs or copied. Baselines were trained by the authors with face labels, and the paper says some diverged §6. The "not reported" cells in tab. 3 suggest some rows come from other papers, but the text does not say so.
• Validation and test differ. Numbers on val (supp.) and test (main) are not interchangeable. The 0.75 threshold was tuned on val §5.3, supp. tab. 2.
• Localization is tracklet-level. A face is labelled through its track, not per frame or per pixel §6. Results do not transfer to frame-level localization.
• Compute is not reported. Training used four V100s §5.5. Run time and memory are not stated.

Lineage

• Builds on: multiple instance learning §5.1; self-attention and non-local blocks §5.2; domain-adversarial training (Ganin et al., 2016) §4; a semi-supervised quality-scoring approach for GAN images, which the authors say inspired Q-Net §4.
• Compared against: Xception (trained as in FaceForensics++), MesoNet, PatchForensics, FWA, TSN, C3D, I3D (face-level labels); S-MIL (video-level labels) §6. Cross-dataset rows also include Capsule, Face X-ray, F³Net and TRN tab. 3.
• Superseded by: not known from content already in this wiki.