Image copy detection

Overview

The task is to decide whether a query image reuses an object, in the foreground or background, from a reference collection. Minor changes from augmentation do not count, and neither does style or general semantic similarity 2212.03860 §3. The paper uses a retrieval pipeline: embed the query and every reference image, score each query against all references, and flag the best match when its similarity passes a threshold 2212.03860 §4 2212.03860 §5. The method therefore assumes that a copy's source is its nearest neighbour in the reference set, so the encoder and similarity measure decide what the detector can find.

Variants

Only one card so far, so the variants below are the design choices it compares, not a survey of the field.

Split-product similarity on ViT tokens

Each image is kept as one feature vector per token. For a pair of images, the score is the best dot product over corresponding tokens, so one shared object is enough for a high score 2212.03860 §4. DINO ViT-B/16 with this measure ties Swin-B for the best average rank, 4.1 2212.03860 tab. 1. It reports 93.53 mAP on IN-Cutmix and 95.86 on IN-Dif-Outpaint, two synthetic sets built from the authors' own edits 2212.03860 tab. 1. Card: Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models.

Global inner product

The alternative takes one vector per image, usually the [CLS] token or a pooled vector, and uses a single dot product. The paper argues this measures global rather than local similarity, and can label unrelated images as similar 2212.03860 §4. The results table does not report numbers for this measure, so the card gives no measured comparison 2212.03860 tab. 1.

Copy-detection-trained extractor

SSCD, a ResNet-50 trained for copy detection, has the highest reported mAP on the Segmix-style synthetic sets, at 67.04 on MSCOCO Segmix and 65.82 on VOC-Segmix, against 46.42 and 45.29 for DINO split-product 2212.03860 tab. 1. It is weak on CUB-200, at 2.00 2212.03860 tab. 1.

Supervised and multimodal extractors

Swin-B trained on ImageNet-1k is one of the three extractors kept for the later experiments 2212.03860 §4.1. It is the highest on rOxford5k and rParis6k. A CLIP image encoder trained on LAION is the highest on GPR1200 and INSTRE 2212.03860 tab. 1. Neither is the top performer on every set.

Other self-supervised extractors

MoCo v3, MoCo with CutMix fine-tuning and VICRegL are also in the comparison. Their average ranks run from 5.0 to 6.5, and a ViT-B/16 trained on ImageNet-21k ranks last at 9.2 2212.03860 tab. 1. The paper keeps only three extractors for later experiments 2212.03860 §4.1.

When to use which

Results in this table are author-reported, on the paper's own benchmarks and synthetic edits, not on your data. The paper's headline replication result is that Stable Diffusion v1.4 gives top-1 matches above 0.5 for about 1.88% of 9,000 random generations, searching the 12M subset only 2212.03860 §7 2212.03860 §9.

Implementation sketch

The core is an embedding, a pairwise score, a top-1 search and a threshold. Embed both sets with the same frozen encoder. The paper's split-product variant keeps one vector per token, so the score is the maximum over corresponding tokens 2212.03860 §4.

# Replication check with split-product scores  [arxiv:2212.03860 §4] [§5]
# Input:  queries (N, T, d), refs (M, T, d), threshold (0.5)
# Output: best score, best reference index, flag per query

def find_copies(queries, refs, threshold):
    # Score every query against every reference, token pair by token pair
    sims = split_product(queries, refs)                  # (N, M)
    # Keep the best reference per query  [arxiv:2212.03860 §5]
    top_sim, top_idx = max_and_argmax(sims, axis=1)
    # Flag matches above the cut-off  [arxiv:2212.03860 §7]
    return top_sim, top_idx, top_sim > threshold

Implementation notes:

• The paper does not state how features are normalised before the dot product. The released code L2-normalises the whole flattened vector, not each token, and it includes the [CLS] token 2212.03860 §4. See the card's Paper vs code section for the details and line references.
• The paper's baseline is a train-to-train run: each reference is matched to the others with its own self-match removed. Generations are then compared with that baseline, not with a fixed number 2212.03860 §5.
• The paper's input size is not given in the text. The code resizes and centre-crops, see the card's Hyperparameters table 2212.03860 §4.

Contradictions

• Best extractor on real data. The text says Swin-B is best on the real datasets 2212.03860 §4.1. The table shows Swin-B best on rOxford5k, rParis6k and CUB-200, and CLIP best on GPR1200 and INSTRE 2212.03860 tab. 1.
• Best average. The text says DINO is best on average 2212.03860 §4.1. The table ranks DINO and Swin-B tied at 4.1 2212.03860 tab. 1.
• Best synthetic extractor. The text says SSCD does best on the synthetic sets 2212.03860 §4.1. The table shows SSCD best on four of five, with DINO best on IN-Cutmix (93.53 against 89.78) 2212.03860 tab. 1.
• CLIP size. The text says ViT-Base/32 2212.03860 §4. The table says ViT-B/16 2212.03860 tab. 1.
• Synthetic set name. The text says IN-Dif-Patch 2212.03860 §4. The table says IN-Dif-Outpaint 2212.03860 tab. 1.
• Threshold wording. The text says "above 0.5" 2212.03860 §7, and a later section says "≥ .5" 2212.03860 §9. This matters only for scores exactly at 0.5.
• Paper vs released code. The paper describes a plain inner product 2212.03860 §4. The code normalises the vectors, and it loads a different SSCD checkpoint and Stable Diffusion version from those the paper reports. The card's Paper vs code section lists each mismatch.

Papers

• Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models: a retrieval detector for object-level copies, used to measure replication in small diffusion models and Stable Diffusion v1.4. It compares ten extractors and similarity measures on ten datasets.