Image manipulation detection

Overview

The task is to find regions of an image that were edited. The card's source names three edit types: splicing (pasting content from another image), copy-move (duplicating content within the same image) and removal (erasing an object and filling the hole) 1805.04953 §4.3. Tampered regions need not be objects, and the detector has to rely on low-level traces rather than on what the content shows 1805.04953 §1 §3.1. A generic object detector is therefore a starting point, not a solution.

Post-processing such as smoothing can hide visible seams, so an RGB-only view is not enough by itself 1805.04953 §1 §3.2. Public forgery datasets hold only a few hundred images, which is too few for deep detectors 1805.04953 §1 §4.1.

Output format matters when you compare numbers. The RGB-N method returns boxes, not masks. Its pixel-level F1 and AUC come from giving every pixel inside a detected box that box's score, and the F1 threshold is picked per image 1805.04953 §4.2. Mask-producing methods are scored on the same benchmarks, so the figures are not like-for-like. The PCL method returns a consistency heatmap, and its paper reports image- and video-level AUC rather than pixel F1 2012.09311 §4.2. The FDFL method outputs a real-or-fake score for a face, with no location at all 2103.09096 §3.1 §4.1.

Benchmarks also differ between variants. RGB-N is evaluated on NIST16, Columbia, COVER and CASIA; the two face-forgery methods mainly on face-video sets such as FF++, Celeb-DF v2 and DFDC 1805.04953 tab. 3 2012.09311 tab. 1 tab. 5 2103.09096 tab. 4. Do not compare their numbers directly.

A separate, neighbouring task is detecting whole images that a generator produced. It asks "is this picture generated?", not "where was it edited?" 2303.14126 §1. The last variant below covers it so the two are not confused.

Variants

Two streams: RGB plus fixed noise, fused per region (RGB-N)

An RGB backbone (ResNet-101) proposes regions and regresses their boxes. A second backbone of the same shape runs on a noise map made by three fixed 5×5 SRM high-pass kernels, picked from 30 base filters 1805.04953 §3.2. Features from both streams are pooled over the same regions and fused by compact bilinear pooling into a 16384-dimensional vector for the tamper classifier 1805.04953 §3.3 §3.4. Card: Learning Rich Features for Image Manipulation Detection.

Reported results, author-reported:

• Pixel F1 on NIST16, Columbia, COVER and CASIA: 0.722, 0.697, 0.437, 0.408 1805.04953 tab. 3.
• Pixel AUC on the same four datasets: 0.937, 0.858, 0.817, 0.795 1805.04953 tab. 4.
• Pre-training on a synthetic set of 42K tampered and authentic pairs built from COCO objects, then fine-tuning per benchmark 1805.04953 §4.1. On the synthetic test split the AP is 0.627, against 0.445 for the RGB stream alone 1805.04953 tab. 1.

The paper picks 3 of the 30 SRM base filters and reports that using all 30 gave no significant gain 1805.04953 §3.2. The card's Paper vs code section lists differences between the paper and the released code, such as the loss weighting and the training schedule; check it before reproducing the numbers.

Single-stream and late-fusion baselines

The same study compares RGB-only, noise-only and late fusion of the two streams with the bilinear design 1805.04953 §4.2 tab. 3 tab. 4. The paper states that late fusion is always worse than bilinear fusion 1805.04953 §4.2. Noise-only is the strongest single stream on Columbia and the weakest on CASIA 1805.04953 tab. 3. The paper attributes the Columbia result to uncompressed splices, where the noise cue alone is enough 1805.04953 §4.2. Older hand-crafted baselines (ELA, NOI1, CFA1) appear in the same tables, with numbers the paper takes from an earlier study 1805.04953 §4.2 tab. 3.

Patch consistency for face forgeries (PCL with I2G)

Face images and video frames only. A ResNet-34, cut after its conv3 layer, gives one feature vector per 16×16 patch of a 256×256 face crop 2012.09311 §4.1. A 1×1-convolution embedding scores every pair of patches with a sigmoid of their scaled dot product, which forms a 4D consistency volume 2012.09311 §3.1 eq. 1. The training target is 1 where two patches agree and 0 where they do not, computed from a known manipulation mask 2012.09311 §3.1 eq. 2. A binary cross-entropy loss on that volume is added, with weight 10, to a real/fake classification loss 2012.09311 eq. 3 eq. 4 §4.1.

The masks are not labelled by hand. The I2G generator pastes a face from another identity into a real frame, inside an elastically deformed convex hull of the face landmarks, then adds augmentations 2012.09311 §3.2 alg. 1. Several of its values are not stated, including the landmark threshold and the blending methods used 2012.09311 alg. 1 §3.2. Card: Learning Self-Consistency for Deepfake Detection.

Reported results, author-reported and video-level unless noted:

• In-dataset AUC on FF++, Celeb-DF v2 and DFDC-P: 99.79, 99.98 and 94.38. The average over these three rises from 96.45 to 98.05 2012.09311 tab. 1 tab. 2 tab. 3 §4.3.
• Cross-dataset AUC (training on real FF++ frames with I2G fakes): the five-dataset average rises from 86.03 to 92.18 2012.09311 §4.4. The gains are not uniform. On DFDC-P the prior method, Face X-ray, scores 80.92 against 74.37, and on NeuralTextures it is 98.13 against 97.63 2012.09311 tab. 4 tab. 5.
• The I2G generator matters in the paper's ablation. Without it, the average AUC is 79.19 when training on DeepFake frames and 74.78 on DFDC-P; with it, 86.37 and 86.35 2012.09311 tab. 8.
• Loss weight λ: 10 gives the best average in the paper's sweep, 82.83 against 82.09 at 100 2012.09311 tab. 7.

The method outputs a heatmap whose fusion step is not specified 2012.09311 §3.1. It is therefore not a ready-made localiser. Its paper reports no pixel-level F1. Its evaluation sets are face-video datasets, not the NIST16 or CASIA benchmarks of the first variant 2012.09311 §4.3 §4.4.

Frequency branch and single-center loss for face videos (FDFL)

Face frames and videos only. An ImageNet-pretrained Xception handles the RGB image. A second branch takes the image in YCbCr, applies an 8 × 8 block DCT, and groups the coefficients of each frequency into one channel, giving 192 channels 2103.09096 §3.2. A small learned block mines that tensor and is fused with the RGB features by a 1 × 1 convolution, between Xception's entry and middle flows 2103.09096 §3.2 §4.1 §4.2.2. In the paper's fusion ablation this beats concatenation and sum 2103.09096 tab. 2.

The training change is the loss. Softmax loss is kept, and a single-center loss is added with weight λ: real embeddings are pulled toward one learnable centre, and fake embeddings must sit at least a margin m·√D farther from it 2103.09096 §3.3 eq. 1 eq. 6. The paper uses m = 0.3, λ = 0.5 and D = 1000 in the final model §4.2.1. Card: Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection.

Reported results, author-reported, FF++ c40, video-level 2103.09096 tab. 4:

• AUC 0.924 and pAUC(0.1) 0.810, against 0.861 and 0.652 for the authors' Xception baseline.
• Each part alone: the single-center loss gives AUC 0.916 and the frequency branch 0.906 2103.09096 tab. 3.
• Loss comparison with softmax in each case: center loss 0.868, triplet loss 0.863, single-center loss 0.916 2103.09096 tab. 1.

The paper gives no optimiser, learning rate, batch size or epoch count in the text; these are said to be in supplementary material 2103.09096 §4.1. Plan to choose and report them yourself. The paper also states weaker generalisation to forgery methods it was not trained on, and the supporting evidence is in supplementary material that is not in the source 2103.09096 §5. Use this variant for in-distribution face-video detection, not as evidence for unseen methods.

Whole-image detection of generated pictures (CIFAKE)

Not localisation. A small CNN (two convolution blocks of 32 filters, one dense layer of 64 units, one sigmoid output) decides whether a 32 × 32 picture is a real CIFAR-10 image or one of 60,000 images made with Stable Diffusion v1.4 2303.14126 §3.1 §3.2 §3.3 fig. 5. The final model reports 93.55% accuracy and an F1 of 0.936 for the REAL class 2303.14126 tab. 7 tab. 11. The Grad-CAM explanations look at a few random samples and are judged by eye 2303.14126 §4.2. The paper uses one generator and one source dataset, and names wider coverage as future work 2303.14126 §5. Card: CIFAKE.

Use this variant for a whole-image real-or-generated question on images of similar scale. Do not use it to locate an edit in a photograph.

When to use which

Implementation sketch

The concept-level shape of the two-stream design, with the paper's step numbers. Layer sizes and the fusion method are in the card for bilinear pooling.

# Two-stream manipulation detector, concept level  [arxiv:1805.04953 §3]
# Input: image; output: boxes with a tamper label and a score

def detect_tampering(image):
    # Fixed high-pass filters give a residual noise map, no learned weights  [§3.2]
    noise_map = srm_filter(image, kernels=3)
    # Two backbones of the same shape, one per stream  [§3.1] [§3.2]
    rgb_feat = backbone_rgb(image)
    noise_feat = backbone_noise(noise_map)
    # Proposals come from the RGB stream only  [§3.2] [tab. 1]
    rois = propose_regions(rgb_feat)
    # Pool both streams over the same regions  [§3.3]
    rgb_roi = pool(rgb_feat, rois)
    noise_roi = pool(noise_feat, rois)
    # Fuse per region before classifying  [§3.3]
    label_scores = classify(bilinear_fuse(rgb_roi, noise_roi))
    # Box regression uses RGB features only  [§3.3]
    boxes = regress_boxes(rgb_roi, rois)
    # Non-maximum suppression at IoU 0.2 at test time  [§3.4]
    return nms(boxes, label_scores, iou=0.2)

The patch-consistency variant replaces the box head with a pairwise target. Its loss is shown here; the backbone and the classification head are the same as in the card.

# Patch-consistency loss, concept level  [arxiv:2012.09311 §3.1] [eq. 1-4]
# Input: face frame, manipulation mask (or none for a real frame)

def consistency_loss(frame, mask):
    # Feature map: one vector per 16 x 16 patch  [§4.1]
    feat = backbone_conv3(frame)                  # (H', W', C)
    # Embed patches, then score every pair  [eq. 1]
    emb = embed_1x1(feat)
    volume_hat = sigmoid(pairwise_dot(emb) / sqrt(C_emb))
    # Target: 1 where two patches share a source, 0 otherwise  [eq. 2]
    volume = agreement_from_mask(mask)            # all ones if real
    # Binary cross-entropy averaged over all patch pairs  [eq. 3]
    return mean(binary_cross_entropy(volume_hat, volume))

Contradictions

• Fusion versus a single stream, pixel F1 on Columbia. Noise-only scores 0.705 and RGB-N 0.697 1805.04953 tab. 3. The paper's explanation is the uncompressed-splice case 1805.04953 §4.2.
• Pixel F1 on CASIA. MFCN reports 0.541 and RGB-N 0.408 1805.04953 tab. 3. The MFCN figures come from that paper, and MFCN outputs masks where RGB-N outputs boxes 1805.04953 §4.2. Read the claim "state of the art" as limited to NIST16 and COVER in this table.
• Headline versus final model in CIFAKE. The abstract's 92.98% is a feature-extractor-only topology 2303.14126 tab. 2. The model used for the result and the Grad-CAM figures reports 93.55% 2303.14126 tab. 7. The paper's text and tables also disagree on how the best topology was chosen 2303.14126 §3.3 §4.2.
• Validation versus test in CIFAKE. The result tables say "validation", while §3.3 defines a test set. If the topology was chosen on the reported images, the figures are optimistic 2303.14126 §3.3.
• Video-level versus frame-level on Celeb-DF v2. The PCL paper reports video-level AUC 90.03 on CD2 in cross-dataset tests, and frame-level 81.8 in an appendix that uses a different comparison set 2012.09311 tab. 5 app. Frame-Level Results on Celeb-DF-v2. Compare like with like.
• Numbers inside the PCL paper. The DFR AUC appears as 99.41 in the table and as 99.51 in a caption. The 99.51 figure is the AP 2012.09311 app. Video-Level Comprehensive Results. The text's "about 18% and 13%" gains on CD1 and CD2 correspond to 17.72 and 13.38 in the tables 2012.09311 §4.4.
• Softmax-only baseline in the FDFL paper. Its loss table gives AUC 0.860 for softmax only, while its ablation and main tables give 0.861 for the same baseline 2103.09096 tab. 1 tab. 3 tab. 4.
• Gain in the text versus the table. The text gives the c40 AUC gain as 6.4%. The table gives 0.924 minus 0.861, which is 6.3 points 2103.09096 §4.3 tab. 4. The pAUC and accuracy gains match the table.
• Own baseline versus published Xception. The FDFL paper's own baseline reaches 86.14% accuracy at c40, while its Xception row reports 81.00% for the same dataset. The paper attributes the gap to fewer training frames 2103.09096 tab. 4 §4.1. Cross-paper comparisons need this caveat.

Papers

• Learning Rich Features for Image Manipulation Detection: two-stream Faster R-CNN with fixed SRM noise features and bilinear fusion, for splicing, copy-move and removal.
• Learning Self-Consistency for Deepfake Detection: face-forgery detector with a pairwise patch-consistency loss, trained on I2G self-made forgeries. No official code.
• Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection: Xception with a learned DCT-frequency branch, trained with a single-center loss on top of softmax, for face videos.
• CIFAKE: binary classifier and dataset for real versus Stable Diffusion images. A neighbouring task, not localisation.