Learning Rich Features for Image Manipulation Detection

In plain words

Photo editing makes it easy to cut an object out of one picture and paste it into another (splicing), duplicate something within the same picture (copy-move), or erase an object and fill the hole (removal). This paper trains a detector that draws boxes around regions that were tampered with, and can also say which of the three edits was used.

The detector looks at the image in two ways at once. The first view is the normal color image, where edits can leave visual clues such as sharp, unnatural edges or odd contrast. The second view keeps only the fine-grained "noise" of the image, obtained with three small fixed filters borrowed from steganalysis (the field that hunts for hidden messages in images). A pasted region usually comes from a different camera or photo, so its noise pattern does not match its surroundings, even when it looks seamless to the eye.

Both views are fed to the same kind of object detector. The color view decides where candidate regions are; for each candidate, features from both views are combined so that the classifier can pick up on how the two kinds of evidence occur together. Because public forgery datasets are small, the authors first train on tens of thousands of fake images they make automatically by pasting COCO objects into other COCO images, then fine-tune on each benchmark.

TL;DR

RGB-N is Faster R-CNN with two ResNet-101 backbones: one on the RGB image, one on a 3-channel noise map made by fixed 5×5 SRM filters. The RPN uses only RGB features; box regression uses only RGB RoI features; the tampered/authentic (or splicing/copy-move/removal) classifier uses compact bilinear pooling of the RGB and noise RoI features. Pre-trained on a synthetic COCO splicing set (42K pairs), it reports state-of-the-art pixel-level F1 on NIST16 and COVER and AUC on all four tested datasets, and beats either stream alone and late fusion of the two. It outputs boxes, not masks; pixel metrics are computed by assigning each box's score to its pixels.

Problem

Generic object detectors look for semantic objects, while manipulation detection must find tampered regions, which need not be objects (e.g. the inpainted area after removal), and rely on low-level artifacts rather than content §1 §3.1. Prior work typically targets one tampering technique or one cue (JPEG double compression, CFA patterns, local noise, tampered edges) §2. Post-processing (e.g. Gaussian smoothing) can hide visual boundary artifacts, so RGB alone is insufficient §1 §3.2. Public forgery datasets contain only a few hundred images, too few for deep detectors §1 §4.1.

Method

Overview §3 fig. 2: a multi-task Faster R-CNN. The RGB stream (ResNet-101) supplies features to the RPN and to box regression. The noise stream applies a fixed SRM filter layer to the image and runs a second backbone of the same architecture on the result. Both streams share the RPN's proposals and the RoI pooling layer. RoI features of both streams are combined by bilinear pooling before a fully connected layer and softmax that classify the RoI.

1. RGB stream §3.1. ResNet-101 pre-trained on ImageNet §4.1; the last conv layer features feed the RPN and box regression. The RPN is trained to propose regions likely to be manipulated, not object-like §3.1.
2. SRM noise layer §3.2 fig. 4. Noise is the residual between a pixel and its estimate from neighbouring pixels. Of the 30 SRM base filters, the authors keep 3, finding that all 30 give no significant gain. The layer has kernel size 5×5×3 and 3 output channels, so its output can feed a backbone pre-trained on 3-channel input. The three kernels fig. 4:

K_1=\frac{1}{4}\begin{bmatrix}0&0&0&0&0\\0&-1&2&-1&0\\0&2&-4&2&0\\0&-1&2&-1&0\\0&0&0&0&0\end{bmatrix},\; K_2=\frac{1}{12}\begin{bmatrix}-1&2&-2&2&-1\\2&-6&8&-6&2\\-2&8&-12&8&-2\\2&-6&8&-6&2\\-1&2&-2&2&-1\end{bmatrix},\; K_3=\frac{1}{2}\begin{bmatrix}0&0&0&0&0\\0&0&0&0&0\\0&1&-2&1&0\\0&0&0&0&0\\0&0&0&0&0\end{bmatrix}

In the code, each output channel applies the same kernel to all three input color channels and sums them, with zero (SAME) padding and stride 1; the weights are a constant, not a trainable variable resnet_fusion.py. 3. Noise backbone §3.2. Same architecture as the RGB backbone; shares the RoI pooling layer with the RGB stream. The noise stream is not used by the RPN, because RGB-only proposals performed better §3.2 tab. 1. 4. Bilinear fusion §3.3. For RoI features f_{RGB} and f_N, the bilinear output is x = f_{RGB}^{T} f_N, sum-pooled over spatial positions, then signed square root x \leftarrow \mathrm{sign}(x)\sqrt{|x|} and L_2 normalisation before the fully connected layer. Compact bilinear pooling (Gao et al.) replaces the full outer product to save memory and time §3.3. 5. Losses §3.1 §3.3. RPN loss:

L_{RPN}(g_i,f_i)=\frac{1}{N_{cls}}\sum_i L_{cls}(g_i,g_i^\star) +\lambda\frac{1}{N_{reg}}\sum_i g_i^\star L_{reg}(f_i,f_i^\star) \quad \text{[eq. 1]}

with g_i the predicted probability that anchor i is manipulated, g_i^\star its ground-truth label, f_i, f_i^\star the 4-d predicted and ground-truth boxes, L_{cls} cross entropy, L_{reg} smooth L_1, N_{cls} the RPN mini-batch size, N_{reg} the number of anchor locations, \lambda=10. Total loss:

L_{total}=L_{RPN}+L_{tamper}(f_{RGB},f_N)+L_{bbox}(f_{RGB}) \quad \text{[eq. 2]}

L_{tamper} is cross entropy on the bilinear feature; L_{bbox} is smooth L_1 on RGB RoI features only §3.3. 6. Training data §4.1. Ground-truth boxes are enlarged by 20 pixels during training so RoIs include some authentic context. For the benchmarks, boxes are extracted from the ground-truth masks §4.2. 7. Inference §3.4 §4.2. Standard NMS (threshold 0.2) over predicted boxes. For pixel-level F1 and AUC, each pixel inside a detected box takes that box's confidence score; F1 is computed at several thresholds and the best F1 per image is kept §4.2. 8. Multi-class variant §4.3. On NIST16 the classes become splicing, removal and copy-move instead of tampered/authentic.

# RGB-N forward and loss for one image (batch size 1)  [§3] [fig. 2]
# Input:  image (H, W, 3), resized so shorter side = 600; gt boxes and labels
# Output: total loss (training) or scored boxes (test)

def rgbn_forward(image, gt_boxes, gt_labels, training):
    # fixed SRM high-pass filters, 3 in -> 3 out channels  [§3.2] [fig. 4]
    noise_map = conv2d(image, srm_kernels_5x5x3x3, stride=1, padding="same")
    # ResNet-101 conv1..conv4 for each stream, separate weights  [§3.1] [§3.2]
    rgb_feat = rgb_backbone_c4(image)               # (h, w, 1024)
    noise_feat = noise_backbone_c4(noise_map)       # (h, w, 1024)
    # proposals from RGB features only  [§3.1] [tab. 1]
    proposals, rpn_loss = rpn(rgb_feat, gt_boxes)
    if training:
        # sample RoIs and assign targets as in Faster R-CNN  [§3.4]
        rois, roi_labels, box_targets = sample_rois(proposals, gt_boxes)
    else:
        rois = proposals
    # shared RoI pooling to 7x7 on both streams  [§3.4]
    rgb_roi = roi_pool(rgb_feat, rois, size=7)      # (R, 7, 7, 1024)
    noise_roi = roi_pool(noise_feat, rois, size=7)  # (R, 7, 7, 1024)
    # per-stream conv5 block on RoI crops (from code, not paper)
    # [code: lib/nets/resnet_fusion.py#L250-L261]
    rgb_roi = rgb_conv5(rgb_roi)                    # (R, 7, 7, 2048)
    noise_roi = noise_conv5(noise_roi)              # (R, 7, 7, 2048)
    # compact bilinear pooling, sum over 7x7 positions  [§3.3] [§3.4]
    fused = compact_bilinear(rgb_roi, noise_roi, out_dim=16384)
    # signed square root then L2 normalisation  [§3.3]
    fused = sign(fused) * sqrt(abs(fused))
    fused = l2_normalize(fused)
    # tamper classifier on fused features  [§3.3] [eq. 2]
    class_logits = fully_connected(fused, num_classes)
    # box regression on RGB RoI features only  [§3.3] [eq. 2]
    box_deltas = fully_connected(spatial_mean(rgb_roi), 4 * num_classes)
    if training:
        # total loss  [eq. 2]
        tamper_loss = cross_entropy(class_logits, roi_labels)
        box_loss = smooth_l1(box_deltas, box_targets)
        return rpn_loss + tamper_loss + box_loss
    # NMS with IoU threshold 0.2 at test time  [§3.4]
    return nms(apply_deltas(rois, box_deltas), softmax(class_logits), 0.2)

Pixel-level scoring used for F1/AUC §4.2:

# Turn box detections into a pixel score map  [§4.2]
def box_score_map(detections, height, width):
    score_map = zeros(height, width)
    for box, score in detections:
        # every pixel in a detected box gets the box's confidence  [§4.2]
        region = score_map[box.y1:box.y2, box.x1:box.x2]
        # overlap handling not stated in the paper
        score_map[box.y1:box.y2, box.x1:box.x2] = maximum(region, score)
    return score_map

The maximum over overlapping boxes in the second block is an implementation choice; the paper only states it for the Late Fusion baseline §4.2.

Hyperparameters

Benchmark splits (number of images) tab. 2:

Columbia is test-only, evaluated with the synthetic-trained model tab. 2. CASIA 2.0 is used for training and CASIA 1.0 for testing §4.2.

Results

All author-reported. Synthetic COCO pre-training set, AP with the COCO detection metric §4.1 tab. 1:

Pixel-level F1 on four benchmarks tab. 3 ("-" = not available in the literature; ELA, NOI1 and CFA1 numbers taken from Salloum et al., MFCN numbers from its paper §4.2):

Pixel-level AUC tab. 4 (ELA/NOI1/CFA1 AUC computed with the code of Zampoglou et al.; J-LSTM from its paper §4.2):

Data augmentation, F1/AUC tab. 5 (JPEG: quality 70; noise: Gaussian, variance 5):

Robustness on NIST16, F1 under JPEG quality / resize scale attacks tab. 6:

Manipulation-technique detection on NIST16, AP tab. 7:

Observations from the paper: RGB-N beats both single streams on every dataset except Columbia, where the uncompressed splices make noise features alone sufficient §4.2; late fusion is always worse than bilinear fusion §4.2; copy-move is hardest because source and target share a noise distribution and contrast §4.3. Qualitative examples are described in fig. 5 (COVER, Columbia, CASIA 1.0) and fig. 6 (multi-class NIST16).

Paper vs code

Checked against pengzhou1108/RGB-N at fdc612d (TensorFlow 0.12, built on tf-faster-rcnn).

• No λ=10 on the RPN box loss. Paper sets λ=10 §3.1 eq. 1; the code adds RPN cross entropy (plain mean over sampled anchors) and the RPN smooth L1 loss (σ=3) with weight 1, plus the two RCNN losses network_fusion.py network_fusion.py.
• Test-time proposals. Paper: 300 §3.4. Code default TEST.RPN_POST_NMS_TOP_N = 100 config.py, not overridden in cfgs/res101_fusion.yml.
• Schedule per dataset. Paper: 110K steps, LR drop at 40K §3.4. The script uses that for coco and casia, but NIST runs 60K steps with the drop at 30K and dist_fake 25K/10K train_faster_rcnn.sh. The README fine-tunes from the synthetic checkpoint at iteration 60000, not 110000 README.md.
• SRM before or after resizing. Paper says image and noise features are resized to shorter side 600 §3.4; the code resizes the image first and applies SRM in-graph to the resized, mean-subtracted image test.py blob.py.
• Where bilinear pooling happens. Paper describes pooling the 7×7×1024 RoI features §3.3 §3.4; the code first runs each stream's own ResNet conv5 block on its crops (2048 channels) and pools those; signed sqrt adds 1e-12 inside the root resnet_fusion.py.
• RoI pooling is TensorFlow crop-and-resize straight to 7×7 (no max-pool, RESNET.MAX_POOL = False) resnet_fusion.py config.py.
• Frozen layers (unstated in paper). RESNET.FIXED_BLOCKS = 1 freezes the RGB stem and block1 and runs them with frozen batch norm; the noise stem and all noise blocks are built trainable config.py resnet_fusion.py resnet_fusion.py.
• 20-pixel box enlargement §4.1 is not in the dataset loaders, which use the annotated boxes as given (clipped at 0) nist.py; it may have been applied when the annotation files were made, which are not in the repository.
• Synthetic data generation. Paper: random COCO objects, chosen via segmentation masks, copied into other images §4.1. The script picks source and target images from the same COCO category, resizes the masked source image to the target's size and pastes it at the origin, and applies a Gaussian blur (radius 1.5) to the whole composite for about two thirds of samples; the blur is not mentioned in the paper. The box stored in the file name is the source-image box, not rescaled demo.py.

Pitfalls

• Boxes, not masks. Pixel F1/AUC come from filling detected boxes with their scores §4.2; comparisons with mask-predicting methods (MFCN, J-LSTM) therefore mix output types. The paper does not say how overlapping boxes combine for RGB-N.
• F1 uses an oracle threshold. The best F1 over thresholds is chosen per image §4.2, which inflates numbers relative to a fixed threshold.
• SRM layer details. Each output channel applies one kernel to all three colour channels and sums them resnet_fusion.py; code channel order is K₂ (1/12), K₁ (1/4), K₃ (1/2), not the figure's order. All three kernels sum to zero, so the mean subtraction cancels in the interior, but zero padding at image borders produces strong edge responses.
• Noise backbone init. The noise stream reuses ImageNet RGB weights even though its input statistics are very different train_val.py; the paper only says a network pre-trained on 3-channel input §3.2.
• Hidden cost. Two full ResNet-101 trunks plus per-RoI conv5 blocks for both streams roughly double backbone compute versus Faster R-CNN; runtime is not reported.
• Compact bilinear pooling needs a custom FFT op in the original code (README build step); output dimension 16384 makes the classifier layer 16384 × num_classes.
• Schedules differ between paper and scripts (see above); reproduce with the per-dataset settings in train_faster_rcnn.sh if matching released results.
• Dataset-specific protocols. NIST16 and COVER splits follow J-LSTM, Columbia and CASIA follow MFCN §4.2; Columbia is evaluated without fine-tuning tab. 2.
• Noise stream is weak alone on CASIA (F1 0.283) tab. 3; fusion helps most where post-processing hides noise cues.

Lineage

• Builds on: Faster R-CNN (Ren et al.) for detection; steganalysis rich models (SRM, Fridrich and Kodovsky) for the noise filters; bilinear CNNs (Lin et al.) and compact bilinear pooling (Gao et al.) for fusion §3.
• Compared against: ELA, NOI1, CFA1, MFCN (arXiv:1709.02016), J-LSTM, plus single-stream and late-fusion ablations §4.2.