---
kind: concept
slug: image-copy-detection
title: Image copy detection
summary: >-
  Image copy detection checks whether a query image contains an object that also appears in a reference set, by retrieving the nearest reference images with a pretrained encoder. Variants differ in the encoder and in how patch or token features are compared. Only one card so far, on diffusion-model replication, with author-reported results.
tags: [memorization, copy-detection, image-retrieval, evaluation]
papers: ["2212.03860"]
related: [diffusion-models, self-supervised-learning, contrastive-learning, nearest-neighbor-search, dense-retrieval]
status: draft
updated_at: 2026-10-19
---

## Overview

The task is to decide whether a query image reuses an object, in the foreground
or background, from a reference collection. Minor changes from augmentation
do not count, and neither does style or general semantic similarity
[arxiv:2212.03860 §3]. The paper uses a retrieval pipeline: embed the query and
every reference image, score each query against all references, and flag the
best match when its similarity passes a threshold [arxiv:2212.03860 §4]
[arxiv:2212.03860 §5]. The method therefore assumes that a copy's source is
its nearest neighbour in the reference set, so the encoder and similarity
measure decide what the detector can find.

## Variants

Only one card so far, so the variants below are the design choices it
compares, not a survey of the field.

### Split-product similarity on ViT tokens

Each image is kept as one feature vector per token. For a pair of images, the
score is the best dot product over corresponding tokens, so one shared object
is enough for a high score [arxiv:2212.03860 §4]. DINO ViT-B/16 with this measure ties
Swin-B for the best average rank, 4.1 [arxiv:2212.03860 tab. 1]. It reports
93.53 mAP on IN-Cutmix and 95.86 on IN-Dif-Outpaint, two synthetic sets built
from the authors' own edits [arxiv:2212.03860 tab. 1]. Card:
[Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](/papers/2212.03860).

### Global inner product

The alternative takes one vector per image, usually the `[CLS]` token or a
pooled vector, and uses a single dot product. The paper argues this measures
global rather than local similarity, and can label unrelated images as similar
[arxiv:2212.03860 §4]. The results table does not report numbers for this
measure, so the card gives no measured comparison [arxiv:2212.03860 tab. 1].

### Copy-detection-trained extractor

SSCD, a ResNet-50 trained for copy detection, has the highest reported mAP on
the Segmix-style synthetic sets, at 67.04 on MSCOCO Segmix and 65.82 on
VOC-Segmix, against 46.42 and 45.29 for DINO split-product
[arxiv:2212.03860 tab. 1]. It is weak on CUB-200, at 2.00 [arxiv:2212.03860 tab. 1].

### Supervised and multimodal extractors

Swin-B trained on ImageNet-1k is one of the three extractors kept for the later
experiments [arxiv:2212.03860 §4.1]. It is the highest on rOxford5k and
rParis6k. A CLIP image encoder trained on LAION is the highest on GPR1200 and
INSTRE [arxiv:2212.03860 tab. 1]. Neither is the top performer on every set.

### Other self-supervised extractors

MoCo v3, MoCo with CutMix fine-tuning and VICRegL are also in the comparison.
Their average ranks run from 5.0 to 6.5, and a ViT-B/16 trained on ImageNet-21k
ranks last at 9.2 [arxiv:2212.03860 tab. 1]. The paper keeps only three
extractors for later experiments [arxiv:2212.03860 §4.1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Copies are objects pasted or outpainted into a larger image | DINO ViT-B/16 with split-product | Per-token matching lets one shared part drive the score. Best on IN-Cutmix; close to SSCD on the outpaint and diagonal sets [arxiv:2212.03860 §4] [arxiv:2212.03860 tab. 1]. |
| Heavy object pastes, a copy-detection checkpoint is available | SSCD ResNet-50 | Highest on the two Segmix sets in the paper's table [arxiv:2212.03860 tab. 1]. Weak on the fine-grained CUB-200 set [arxiv:2212.03860 tab. 1]. |
| Real photographs, object-level retrieval | Swin-B or CLIP, not DINO split-product | On the real sets, DINO split-product scores below these two on rOxford5k, rParis6k, GPR1200 and INSTRE [arxiv:2212.03860 tab. 1]. |
| Smallest index, one vector per image | Global inner product | One vector per image is cheaper to store than one per token. The paper says the measure mixes up unrelated images [arxiv:2212.03860 §4]. Check precision on your own data. |
| Large reference set, such as web-scale training data | Sample a subset and state the scope | The paper's search covered a 12M-image subset, about 0.6% of its fine-tuning data, and it calls the resulting rate an underestimate [arxiv:2212.03860 §7] [arxiv:2212.03860 §9]. |
| Choosing the flag threshold | Calibrate on your own pairs | The 0.5 cut-off is not justified beyond the copies it flagged [arxiv:2212.03860 §7]. The paper also checks generations against a train-to-train baseline [arxiv:2212.03860 §5]. |

Results in this table are author-reported, on the paper's own benchmarks and
synthetic edits, not on your data. The paper's headline replication result is that
Stable Diffusion v1.4 gives top-1 matches above 0.5 for about 1.88% of 9,000
random generations, searching the 12M subset only [arxiv:2212.03860 §7]
[arxiv:2212.03860 §9].

## Implementation sketch

The core is an embedding, a pairwise score, a top-1 search and a threshold.
Embed both sets with the same frozen encoder. The paper's split-product
variant keeps one vector per token, so the score is the maximum over
corresponding tokens [arxiv:2212.03860 §4].

```pseudocode
# Replication check with split-product scores  [arxiv:2212.03860 §4] [§5]
# Input:  queries (N, T, d), refs (M, T, d), threshold (0.5)
# Output: best score, best reference index, flag per query

def find_copies(queries, refs, threshold):
    # Score every query against every reference, token pair by token pair
    sims = split_product(queries, refs)                  # (N, M)
    # Keep the best reference per query  [arxiv:2212.03860 §5]
    top_sim, top_idx = max_and_argmax(sims, axis=1)
    # Flag matches above the cut-off  [arxiv:2212.03860 §7]
    return top_sim, top_idx, top_sim > threshold
```

Implementation notes:

- The paper does not state how features are normalised before the dot product.
  The released code L2-normalises the whole flattened vector, not each token,
  and it includes the `[CLS]` token [arxiv:2212.03860 §4]. See the card's
  Paper vs code section for the details and line references.
- The paper's baseline is a train-to-train run: each reference is matched to
  the others with its own self-match removed. Generations are then compared
  with that baseline, not with a fixed number [arxiv:2212.03860 §5].
- The paper's input size is not given in the text. The code resizes and
  centre-crops, see the card's Hyperparameters table [arxiv:2212.03860 §4].

## Contradictions

- **Best extractor on real data.** The text says Swin-B is best on the real
  datasets [arxiv:2212.03860 §4.1]. The table shows Swin-B best on rOxford5k,
  rParis6k and CUB-200, and CLIP best on GPR1200 and INSTRE [arxiv:2212.03860 tab. 1].
- **Best average.** The text says DINO is best on average [arxiv:2212.03860 §4.1].
  The table ranks DINO and Swin-B tied at 4.1 [arxiv:2212.03860 tab. 1].
- **Best synthetic extractor.** The text says SSCD does best on the synthetic
  sets [arxiv:2212.03860 §4.1]. The table shows SSCD best on four of five, with
  DINO best on IN-Cutmix (93.53 against 89.78) [arxiv:2212.03860 tab. 1].
- **CLIP size.** The text says ViT-Base/32 [arxiv:2212.03860 §4]. The table
  says ViT-B/16 [arxiv:2212.03860 tab. 1].
- **Synthetic set name.** The text says IN-Dif-Patch [arxiv:2212.03860 §4]. The
  table says IN-Dif-Outpaint [arxiv:2212.03860 tab. 1].
- **Threshold wording.** The text says "above 0.5" [arxiv:2212.03860 §7], and
  a later section says "≥ .5" [arxiv:2212.03860 §9]. This matters only for
  scores exactly at 0.5.
- **Paper vs released code.** The paper describes a plain inner product
  [arxiv:2212.03860 §4]. The code normalises the vectors, and it loads a
  different SSCD checkpoint and Stable Diffusion version from those the paper
  reports. The card's Paper vs code section lists each mismatch.

## Papers

- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](/papers/2212.03860): a retrieval detector for object-level copies, used to measure replication in small diffusion models and Stable Diffusion v1.4. It compares ten extractors and similarity measures on ten datasets.