---
kind: concept
slug: image-manipulation-detection
title: Image manipulation detection
summary: >-
  Find edited regions in an image, or decide if a face or whole image is fake. Four cards: RGB-N, a two-stream box detector (NIST16 pixel F1 0.722); PCL, face-video patch consistency (no official code); FDFL, a face-video detector with a single-center loss; CIFAKE, a whole-image classifier. All results are author-reported.
tags: [image-forensics, fusion, two-stream, object-detection, deepfake-detection, synthetic-image-detection, metric-learning, frequency-domain]
papers: ["1805.04953", "2012.09311", "2103.09096", "2303.14126"]
related: [bilinear-pooling, object-detection, metric-learning, frequency-domain-learning, diffusion-models, data-augmentation, audio-deepfake-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

The task is to find regions of an image that were edited. The card's source
names three edit types: splicing (pasting content from another image),
copy-move (duplicating content within the same image) and removal (erasing an
object and filling the hole) [arxiv:1805.04953 §4.3]. Tampered regions need not
be objects, and the detector has to rely on low-level traces rather than on
what the content shows [arxiv:1805.04953 §1] [§3.1]. A generic object detector
is therefore a starting point, not a solution.

Post-processing such as smoothing can hide visible seams, so an RGB-only view
is not enough by itself [arxiv:1805.04953 §1] [§3.2]. Public forgery datasets
hold only a few hundred images, which is too few for deep detectors
[arxiv:1805.04953 §1] [§4.1].

Output format matters when you compare numbers. The RGB-N method returns boxes,
not masks. Its pixel-level F1 and AUC come from giving every pixel inside a
detected box that box's score, and the F1 threshold is picked per image
[arxiv:1805.04953 §4.2]. Mask-producing methods are scored on the same
benchmarks, so the figures are not like-for-like. The PCL method returns a
consistency heatmap, and its paper reports image- and video-level AUC rather
than pixel F1 [arxiv:2012.09311 §4.2]. The FDFL method outputs a real-or-fake
score for a face, with no location at all [arxiv:2103.09096 §3.1] [§4.1].

Benchmarks also differ between variants. RGB-N is evaluated on NIST16, Columbia,
COVER and CASIA; the two face-forgery methods mainly on face-video sets such as
FF++, Celeb-DF v2 and DFDC [arxiv:1805.04953 tab. 3] [arxiv:2012.09311 tab. 1]
[tab. 5] [arxiv:2103.09096 tab. 4]. Do not compare their numbers directly.

A separate, neighbouring task is detecting whole images that a generator
produced. It asks "is this picture generated?", not "where was it edited?"
[arxiv:2303.14126 §1]. The last variant below covers it so the two are not
confused.

## Variants

### Two streams: RGB plus fixed noise, fused per region (RGB-N)

An RGB backbone (ResNet-101) proposes regions and regresses their boxes. A
second backbone of the same shape runs on a noise map made by three fixed 5×5
SRM high-pass kernels, picked from 30 base filters [arxiv:1805.04953 §3.2].
Features from both streams are pooled over the same regions and fused by
compact bilinear pooling into a 16384-dimensional vector for the tamper
classifier [arxiv:1805.04953 §3.3] [§3.4]. Card:
[Learning Rich Features for Image Manipulation Detection](/papers/1805.04953).

Reported results, author-reported:

- Pixel F1 on NIST16, Columbia, COVER and CASIA: 0.722, 0.697, 0.437, 0.408
  [arxiv:1805.04953 tab. 3].
- Pixel AUC on the same four datasets: 0.937, 0.858, 0.817, 0.795
  [arxiv:1805.04953 tab. 4].
- Pre-training on a synthetic set of 42K tampered and authentic pairs built
  from COCO objects, then fine-tuning per benchmark [arxiv:1805.04953 §4.1].
  On the synthetic test split the AP is 0.627, against 0.445 for the RGB stream
  alone [arxiv:1805.04953 tab. 1].

The paper picks 3 of the 30 SRM base filters and reports that using all 30
gave no significant gain [arxiv:1805.04953 §3.2]. The card's Paper vs code section lists differences
between the paper and the released code, such as the loss weighting and the
training schedule; check it before reproducing the numbers.

### Single-stream and late-fusion baselines

The same study compares RGB-only, noise-only and late fusion of the two streams
with the bilinear design [arxiv:1805.04953 §4.2] [tab. 3] [tab. 4]. The paper
states that late fusion is always worse than bilinear fusion
[arxiv:1805.04953 §4.2]. Noise-only is the strongest single stream on Columbia
and the weakest on CASIA [arxiv:1805.04953 tab. 3]. The paper attributes the
Columbia result to uncompressed splices, where the noise cue alone is enough
[arxiv:1805.04953 §4.2]. Older hand-crafted baselines (ELA, NOI1, CFA1) appear in
the same tables, with numbers the paper takes from an earlier study
[arxiv:1805.04953 §4.2] [tab. 3].

### Patch consistency for face forgeries (PCL with I2G)

Face images and video frames only. A ResNet-34, cut after its conv3 layer,
gives one feature vector per 16×16 patch of a 256×256 face crop
[arxiv:2012.09311 §4.1]. A 1×1-convolution embedding scores every pair of
patches with a sigmoid of their scaled dot product, which forms a 4D
consistency volume [arxiv:2012.09311 §3.1] [eq. 1]. The training target is 1
where two patches agree and 0 where they do not, computed from a known
manipulation mask [arxiv:2012.09311 §3.1] [eq. 2]. A binary cross-entropy loss
on that volume is added, with weight 10, to a real/fake classification loss
[arxiv:2012.09311 eq. 3] [eq. 4] [§4.1].

The masks are not labelled by hand. The I2G generator pastes a face from another
identity into a real frame, inside an elastically deformed convex hull of the
face landmarks, then adds augmentations [arxiv:2012.09311 §3.2] [alg. 1]. Several
of its values are not stated, including the landmark threshold and the blending
methods used [arxiv:2012.09311 alg. 1] [§3.2]. Card:
[Learning Self-Consistency for Deepfake Detection](/papers/2012.09311).

Reported results, author-reported and video-level unless noted:

- In-dataset AUC on FF++, Celeb-DF v2 and DFDC-P: 99.79, 99.98 and 94.38. The
  average over these three rises from 96.45 to 98.05 [arxiv:2012.09311 tab. 1]
  [tab. 2] [tab. 3] [§4.3].
- Cross-dataset AUC (training on real FF++ frames with I2G fakes): the
  five-dataset average rises from 86.03 to 92.18 [arxiv:2012.09311 §4.4]. The
  gains are not uniform. On DFDC-P the prior method, Face X-ray, scores 80.92
  against 74.37, and on NeuralTextures it is 98.13 against 97.63
  [arxiv:2012.09311 tab. 4] [tab. 5].
- The I2G generator matters in the paper's ablation. Without it, the average
  AUC is 79.19 when training on DeepFake frames and 74.78 on DFDC-P; with it,
  86.37 and 86.35 [arxiv:2012.09311 tab. 8].
- Loss weight λ: 10 gives the best average in the paper's sweep, 82.83 against
  82.09 at 100 [arxiv:2012.09311 tab. 7].

The method outputs a heatmap whose fusion step is not specified
[arxiv:2012.09311 §3.1]. It is therefore not a ready-made localiser. Its
paper reports no pixel-level F1. Its evaluation sets are face-video datasets,
not the NIST16 or CASIA benchmarks of the first variant [arxiv:2012.09311 §4.3]
[§4.4].

### Frequency branch and single-center loss for face videos (FDFL)

Face frames and videos only. An ImageNet-pretrained Xception handles the RGB
image. A second branch takes the image in YCbCr, applies an 8 × 8 block DCT, and
groups the coefficients of each frequency into one channel, giving 192 channels
[arxiv:2103.09096 §3.2]. A small learned block mines that tensor and is fused
with the RGB features by a 1 × 1 convolution, between Xception's entry and
middle flows [arxiv:2103.09096 §3.2] [§4.1] [§4.2.2]. In the paper's fusion
ablation this beats concatenation and sum [arxiv:2103.09096 tab. 2].

The training change is the loss. Softmax loss is kept, and a single-center
loss is added with weight λ: real embeddings are pulled toward one learnable
centre, and fake embeddings must sit at least a margin m·√D farther from it
[arxiv:2103.09096 §3.3] [eq. 1] [eq. 6]. The paper uses m = 0.3, λ = 0.5 and
D = 1000 in the final model [§4.2.1]. Card:
[Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection](/papers/2103.09096).

Reported results, author-reported, FF++ c40, video-level [arxiv:2103.09096 tab. 4]:

- AUC 0.924 and pAUC(0.1) 0.810, against 0.861 and 0.652 for the authors'
  Xception baseline.
- Each part alone: the single-center loss gives AUC 0.916 and the frequency
  branch 0.906 [arxiv:2103.09096 tab. 3].
- Loss comparison with softmax in each case: center loss 0.868, triplet loss
  0.863, single-center loss 0.916 [arxiv:2103.09096 tab. 1].

The paper gives no optimiser, learning rate, batch size or epoch count in the
text; these are said to be in supplementary material [arxiv:2103.09096 §4.1].
Plan to choose and report them yourself. The paper also states weaker
generalisation to forgery methods it was not trained on, and the supporting
evidence is in supplementary material that is not in the source
[arxiv:2103.09096 §5]. Use this variant for in-distribution face-video
detection, not as evidence for unseen methods.

### Whole-image detection of generated pictures (CIFAKE)

Not localisation. A small CNN (two convolution blocks of 32 filters, one dense
layer of 64 units, one sigmoid output) decides whether a 32 × 32 picture is a
real CIFAR-10 image or one of 60,000 images made with Stable Diffusion v1.4
[arxiv:2303.14126 §3.1] [§3.2] [§3.3] [fig. 5]. The final model reports 93.55%
accuracy and an F1 of 0.936 for the REAL class [arxiv:2303.14126 tab. 7]
[tab. 11]. The Grad-CAM explanations look at a few random samples and are
judged by eye [arxiv:2303.14126 §4.2]. The paper uses one generator and one
source dataset, and names wider coverage as future work [arxiv:2303.14126 §5].
Card: [CIFAKE](/papers/2303.14126).

Use this variant for a whole-image real-or-generated question on images of
similar scale. Do not use it to locate an edit in a photograph.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Need tampered regions as boxes, one model for splicing, copy-move and removal, and the budget for two ResNet-101 trunks | Two-stream RGB-N design | Highest author-reported pixel F1 on NIST16 and COVER, and highest AUC on all four benchmarks [arxiv:1805.04953 tab. 3] [tab. 4] |
| Source images were heavily re-compressed or resized | Keep the RGB stream, and expect noise cues to fail | Under the strongest JPEG and resize settings, RGB-N's F1 on NIST16 stays at 0.677 to 0.681, while NOI1 falls to 0.140 to 0.155 [arxiv:1805.04953 tab. 6] |
| Uncompressed splices, similar to the Columbia benchmark | Check noise-only before committing to fusion | Noise-only F1 is 0.705 against 0.697 for RGB-N [arxiv:1805.04953 tab. 3] |
| Little labelled tampering data | Pre-train on synthetic splices, then fine-tune | The paper's answer to the data shortage [arxiv:1805.04953 §4.1]; the synthetic gain is measured only on its synthetic test split [arxiv:1805.04953 tab. 1] |
| Face images or video frames, forgery methods unseen at training time, no mask labels | Patch-consistency (PCL) with I2G | Average cross-dataset AUC 92.18 against 86.03 for the prior method [arxiv:2012.09311 §4.4]. Expect weak results on DFDC (AUC 67.52) [arxiv:2012.09311 tab. 5] |
| Face forgery, in-dataset benchmark with the same protocol | PCL, if you can reimplement it | Average in-dataset AUC 98.05 against 96.45 for the prior best [arxiv:2012.09311 §4.3]. No official code exists [arxiv:2012.09311 Paper vs code] |
| Face video, a real-or-fake score is enough, and you can train a second, frequency branch with a metric loss | FDFL (frequency branch plus single-center loss) | c40 AUC 0.924 against 0.861 for the authors' baseline, and pAUC(0.1) 0.810 against 0.652 [arxiv:2103.09096 tab. 4]. The training recipe is missing, so expect extra tuning [arxiv:2103.09096 §4.1] |
| Face video, forgery methods unseen at training time are a concern | PCL with its cross-dataset protocol, and a test of your own | PCL reports a cross-dataset gain on its own protocol [arxiv:2012.09311 §4.4]. FDFL's paper states weaker generalisation to unseen methods, with the evidence in supplementary material not in the source [arxiv:2103.09096 §5] |
| Need the edit type as well as its location | Multi-class head on the same design | AP 0.960 for splicing, 0.939 for removal, 0.903 for copy-move on NIST16 [arxiv:1805.04953 tab. 7]; copy-move is the hardest type [arxiv:1805.04953 §4.3] |
| Need pixel-level masks | Plan an extra step | The RGB-N pixel scores come from boxes [arxiv:1805.04953 §4.2]. The PCL heatmap fusion is not specified [arxiv:2012.09311 §3.1]. Neither paper describes a mask model |
| Decide whether a whole image is generated | Small CNN baseline, as in CIFAKE | Only one generator and one 32 × 32 dataset are tested [arxiv:2303.14126 §5]. Results are author-reported |
| Runtime or memory budget | Compare with care | RGB-N gives no runtime figures [arxiv:1805.04953 §4.2]. PCL reports 0.0234 s per 256 × 256 image on one V100 for the full model, of which the PCL head adds 0.0009 s [arxiv:2012.09311 app. Computational Complexity] |

## Implementation sketch

The concept-level shape of the two-stream design, with the paper's step
numbers. Layer sizes and the fusion method are in the card for
[bilinear pooling](/concepts/bilinear-pooling).

```pseudocode
# Two-stream manipulation detector, concept level  [arxiv:1805.04953 §3]
# Input: image; output: boxes with a tamper label and a score

def detect_tampering(image):
    # Fixed high-pass filters give a residual noise map, no learned weights  [§3.2]
    noise_map = srm_filter(image, kernels=3)
    # Two backbones of the same shape, one per stream  [§3.1] [§3.2]
    rgb_feat = backbone_rgb(image)
    noise_feat = backbone_noise(noise_map)
    # Proposals come from the RGB stream only  [§3.2] [tab. 1]
    rois = propose_regions(rgb_feat)
    # Pool both streams over the same regions  [§3.3]
    rgb_roi = pool(rgb_feat, rois)
    noise_roi = pool(noise_feat, rois)
    # Fuse per region before classifying  [§3.3]
    label_scores = classify(bilinear_fuse(rgb_roi, noise_roi))
    # Box regression uses RGB features only  [§3.3]
    boxes = regress_boxes(rgb_roi, rois)
    # Non-maximum suppression at IoU 0.2 at test time  [§3.4]
    return nms(boxes, label_scores, iou=0.2)
```

The patch-consistency variant replaces the box head with a pairwise target.
Its loss is shown here; the backbone and the classification head are the same
as in the card.

```pseudocode
# Patch-consistency loss, concept level  [arxiv:2012.09311 §3.1] [eq. 1-4]
# Input: face frame, manipulation mask (or none for a real frame)

def consistency_loss(frame, mask):
    # Feature map: one vector per 16 x 16 patch  [§4.1]
    feat = backbone_conv3(frame)                  # (H', W', C)
    # Embed patches, then score every pair  [eq. 1]
    emb = embed_1x1(feat)
    volume_hat = sigmoid(pairwise_dot(emb) / sqrt(C_emb))
    # Target: 1 where two patches share a source, 0 otherwise  [eq. 2]
    volume = agreement_from_mask(mask)            # all ones if real
    # Binary cross-entropy averaged over all patch pairs  [eq. 3]
    return mean(binary_cross_entropy(volume_hat, volume))
```

## Contradictions

- **Fusion versus a single stream, pixel F1 on Columbia.** Noise-only scores
  0.705 and RGB-N 0.697 [arxiv:1805.04953 tab. 3]. The paper's explanation is
  the uncompressed-splice case [arxiv:1805.04953 §4.2].
- **Pixel F1 on CASIA.** MFCN reports 0.541 and RGB-N 0.408
  [arxiv:1805.04953 tab. 3]. The MFCN figures come from that paper, and
  MFCN outputs masks where RGB-N outputs boxes [arxiv:1805.04953 §4.2]. Read
  the claim "state of the art" as limited to NIST16 and COVER in this table.
- **Headline versus final model in CIFAKE.** The abstract's 92.98% is a
  feature-extractor-only topology [arxiv:2303.14126 tab. 2]. The model used for
  the result and the Grad-CAM figures reports 93.55% [arxiv:2303.14126 tab. 7].
  The paper's text and tables also disagree on how the best topology was
  chosen [arxiv:2303.14126 §3.3] [§4.2].
- **Validation versus test in CIFAKE.** The result tables say "validation",
  while §3.3 defines a test set. If the topology was chosen on the reported
  images, the figures are optimistic [arxiv:2303.14126 §3.3].
- **Video-level versus frame-level on Celeb-DF v2.** The PCL paper reports
  video-level AUC 90.03 on CD2 in cross-dataset tests, and frame-level 81.8 in
  an appendix that uses a different comparison set [arxiv:2012.09311 tab. 5]
  [app. Frame-Level Results on Celeb-DF-v2]. Compare like with like.
- **Numbers inside the PCL paper.** The DFR AUC appears as 99.41 in the
  table and as 99.51 in a caption. The 99.51 figure is the AP
  [arxiv:2012.09311 app. Video-Level Comprehensive Results]. The text's
  "about 18% and 13%" gains on CD1 and CD2 correspond to 17.72 and 13.38 in the
  tables [arxiv:2012.09311 §4.4].
- **Softmax-only baseline in the FDFL paper.** Its loss table gives AUC 0.860
  for softmax only, while its ablation and main tables give 0.861 for the same
  baseline [arxiv:2103.09096 tab. 1] [tab. 3] [tab. 4].
- **Gain in the text versus the table.** The text gives the c40 AUC gain as
  6.4%. The table gives 0.924 minus 0.861, which is 6.3 points
  [arxiv:2103.09096 §4.3] [tab. 4]. The pAUC and accuracy gains match the table.
- **Own baseline versus published Xception.** The FDFL paper's own baseline
  reaches 86.14% accuracy at c40, while its Xception row reports 81.00% for the
  same dataset. The paper attributes the gap to fewer training frames
  [arxiv:2103.09096 tab. 4] [§4.1]. Cross-paper comparisons need this caveat.

## Papers

- [Learning Rich Features for Image Manipulation Detection](/papers/1805.04953):
  two-stream Faster R-CNN with fixed SRM noise features and bilinear fusion,
  for splicing, copy-move and removal.
- [Learning Self-Consistency for Deepfake Detection](/papers/2012.09311):
  face-forgery detector with a pairwise patch-consistency loss, trained on
  I2G self-made forgeries. No official code.
- [Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection](/papers/2103.09096):
  Xception with a learned DCT-frequency branch, trained with a single-center
  loss on top of softmax, for face videos.
- [CIFAKE](/papers/2303.14126): binary classifier and dataset for real versus
  Stable Diffusion images. A neighbouring task, not localisation.