Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

In plain words

Image generators such as Stable Diffusion learn from very large collections of pictures. People who use them ask whether the output is new work or a copy of something the model saw during training. This paper builds a tool for that question. It takes a generated picture and searches the training set for a picture that contains the same object, using image-similarity models that were built for finding copies and matching images.

The authors train small image generators on small datasets and find that they often reproduce training pictures. A model trained on a large, varied dataset (ImageNet) shows no copies that the tool could find. Stable Diffusion does produce copies of training images. This happens more often when a prompt contains phrases that also appear in its captions, and the model sometimes copies the style of an artist instead of the content. The search covered only part of Stable Diffusion's training data, so the rate the authors report is likely an underestimate.

TL;DR

• The paper asks whether diffusion models reproduce their training images, and builds a retrieval-based detector for object-level copies §1 §3 §4.
• Ten feature extractors were ranked on ten datasets. DINO ViT-B/16 with a token-wise "split-product" similarity is among the best on average, tied with Swin-B, and it is one of three extractors kept for the later studies §4.1 tab. 1.
• Diffusion models trained on 300 or 3000 Celeb-A images, or on 100 or 1083 Oxford Flowers images, produce blatant copies. The full-data Celeb-A model and an ImageNet latent model show none that the tool found §5 §6.
• Stable Diffusion v1.4 has top-1 matches above 0.5 for about 1.88% of random generations, searching only the 12M-image LAION Aesthetics v2 6+ subset §7 §9.
• Code is Apache-2.0 at github.com/somepago/DCR. It has the fine-tuning, inference and retrieval scripts, but no script for the DDPM experiments of Section 5 (see Paper vs code).

Problem

Diffusion models are used for commercial art and graphic design. Their web-scale training sets come with unknown origins and intellectual-property status, and large models can memorise training data. The open question is whether diffusion models reproduce training content, and how often §1.

No labelled dataset captures the paper's notion of replication, so the authors build one. They count an image as replicated when one of its objects, in the foreground or background, is the same as an object in a training image. Minor differences that come from data augmentation are ignored. Style and semantic similarity are excluded §3. Standard copy detection asks for high semantic similarity, while retrieval asks for less §2 §4.

Method

The study has three parts.

1. Choose a detector §4. Ten feature extractors are compared on five real and five synthetic datasets. The three best ones (SSCD, Swin-B, DINO with split-product) are used later §4.1.
2. Match generations to training data §5 §7. Each generated image is scored against every training image. The best match (top-1) is kept. Matches are compared with a baseline in which each training image is matched against the rest of the training set.
3. Study the models §5 §6 §7. Diffusion models are trained on small datasets (DDPM on Celeb-A and Oxford Flowers), a class-conditional latent diffusion model on ImageNet is analysed, and Stable Diffusion v1.4 is searched against LAION Aesthetics. Section 8 discusses possible causes.

Step 1: candidate extractors and datasets §4

• Candidates (ten prototypes): Multigrain (ResNet-50, retrieval training); SSCD (ResNet-50, copy-detection training); four supervised models (ViT-S/16 on ImageNet-1k, ViT-B/16 on ImageNet-21k, a CLIP image encoder trained on LAION, Swin-B on ImageNet-1k); and four self-supervised models (MoCo v3 ViT-B/16, the same MoCo model fine-tuned with CutMix for 50 epochs, VICRegL ResNet-50, DINO ViT-B/16) §4 tab. 1.
• Real datasets (five): rOxford5k and rParis6k, both with the cleaned labels of Radenović et al. §4; INSTRE; GPR1200; CUB-200 §4.
• Synthetic datasets (five), built from ImageNet, MS COCO and PASCAL VOC §4 fig. 2:
  • IN-Cutmix: a random square patch is pasted into another image at a random place, with a random patch size.
  • IN-Dif-Patch: about 80% of the image is masked except a random square, and the rest is outpainted with RePaint.
  • IN-Dif-Diagonal: a random triangular half is masked and outpainted, so half of the content is shared with the original.
  • MSCOCO Segmix and VOC Segmix: a random object or the background is cut out with its segmentation mask, given augmentations (flip, blur, autocontrast, solarize, colour jitter), and pasted into another image. Foreground objects are also resized and moved at random.
• Metrics: mAP as the main metric. Recall@k and MRR are reported in the supplement §4.

Step 2: similarity measures §4

The authors compare two measures on feature vectors.

• Global inner product: one dot product per image pair, using the [CLS] token or a pooled representation. The paper argues that this measures global, not local, similarity, and that it can label unrelated images as similar §4.
• Split-product: each feature vector is split into chunks. For vision transformers, each token is one chunk. Dot products are taken between corresponding chunks, and the maximum over chunks is kept §4. A high score then needs only one matching part, so two images that each share a different object with a generated image are not forced to look alike §4.

# Split-product similarity  [§4]
# Input:  queries (N, T, d), train (M, T, d)  # T tokens, d dims
# Output: sims (N, M)

def split_product_similarity(queries, train):
    # Flatten each image to one vector of T * d values  [code: utils_ret.py#L737-L738]
    queries = flatten(queries)                  # (N, T * d)
    train = flatten(train)                      # (M, T * d)
    # L2-normalise the whole vector  [code: diff_retrieval.py#L388-L389]
    queries = normalise(queries)
    train = normalise(train)
    # Split each vector back into T chunks  [code: diff_retrieval.py#L397-L398]
    queries = reshape(queries, (N, T, d))
    train = reshape(train, (M, T, d))
    # One inner product per token pair  [§4]
    chunk_sims = einsum("ntd,mtd->nmt", queries, train)     # (N, M, T)
    # Keep the best token pair for each image pair  [§4]
    return max_over_last_axis(chunk_sims)                   # (N, M)

The paper does not say how the vectors are normalised. The code's normalisation is covered under Paper vs code.

Step 3: matching and baseline §5 §7

• Each generated image is compared with all training images. The top-1 similarity and its index are recorded §5.
• For a baseline, each training image is compared with the other training images, and its own self-match is removed. Distributions of generation-to-train and train-to-train similarity are then compared. If generations sit to the right of the baseline, they are closer to the training data than training images are to each other §5 fig. 5.
• For Stable Diffusion, the cut-off is 0.5 §5 §7. Duplicates in LAION are images with SSCD similarity above 0.95 §7.

# Replication check for generated images  [§5] [§7]
# Input:  generations, train_images, threshold (0.5)
# Output: top-1 similarity, match index, and flag per generation

def find_replicas(generations, train_images, threshold):
    # Embed both sets with the chosen extractor  [§4.1]
    gen_feats = extract(generations)                       # (N, T, d)
    train_feats = extract(train_images)                    # (M, T, d)
    # Score every generation against every training image  [§4]
    sims = split_product_similarity(gen_feats, train_feats)  # (N, M)
    # Keep the best training match per generation  [§5]
    top_sim, top_idx = max_and_argmax(sims, axis=1)
    # Flag generations above the threshold  [§7]
    flagged = top_sim > threshold
    return top_sim, top_idx, flagged

Experiments

• DDPM on small datasets §5. Denoising diffusion models trained with the HuggingFace implementation on Celeb-A (300 and 3000 images, plus the full dataset from an official checkpoint) and Oxford Flowers (100 images, 1083 images from the top five classes, and all 8189 images). Each model trains for at least 300k steps and until FID is below 50 §5 app. B.1. Analysis uses 10,000 generations for Celeb-A and 5,000 for Flowers §5.
• ImageNet latent diffusion §6. The official class-conditional checkpoint. The authors pick 100 random classes and generate 1,000 samples per class.
• Stable Diffusion v1.4 §7 app. C. The model was trained on about 2B LAION images, then fine-tuned on 600M LAION Aesthetics v2 5+ images §2 §7. The authors sample 9,000 captions with replacement from LAION Aesthetics v6+, generate one image per caption, and search the 12M-image Aesthetics v2 6+ subset §7 app. C.
• Causes §8. The authors propose, without testing them, that replication comes from text conditioning, a skewed number of repeats per training image, and enough gradient updates to overfit a subset of the data.

Hyperparameters

Appendix prompt lists are in tab. 2, tab. 3 and tab. 4.

Results

Author-reported. Setup: ten retrieval models on ten datasets, mAP §4 tab. 1. Bold marks the best average rank as printed in the paper, and the rank is averaged over datasets (lower is better).

Source: tab. 1. The first five columns are real datasets and the next five are synthetic. Average ranks: Swin and DINO 4.1, SSCD 4.2 tab. 1.

Replication findings:

Paper vs code

Repository: github.com/somepago/DCR, at commit bac8b5fbf739c75be6a187f97e2b81e0fd51115c, licence Apache-2.0 LICENSE. The owner is the first author's GitHub account, and the README says the repo covers this CVPR 2023 paper and a NeurIPS 2023 follow-up README.md.

• Similarity is normalised. The paper describes an inner product §4. The code L2-normalises both feature sets before the dot product, so the baseline is cosine similarity diff_retrieval.py diff_retrieval.py.
• Split-product normalisation. The code normalises the whole flattened vector of all tokens, not each token. Chunk products therefore carry the global norm of the image utils_ret.py diff_retrieval.py diff_retrieval.py. The paper does not state a normalisation.
• [CLS] token included. The DINO model returns every token, including [CLS], after the final norm dino_vits.py. At 224 px that is 197 chunks diff_retrieval.py. The paper says "each token" §4 without saying whether [CLS] is included.
• SSCD checkpoint. The paper uses the official ResNet-50 checkpoint trained on ImageNet §4. In the code, resnet50 loads sscd_disc_mixup diff_retrieval.py. The ImageNet-trained checkpoint is resnet50_im diff_retrieval.py. The README example uses resnet50_disc, which loads sscd_disc_large diff_retrieval.py README.md. Which checkpoint produced the reported numbers is not stated.
• Stable Diffusion version. The paper uses v1.4 §2 §7 app. C. utils/stable_diff_gen.py loads stabilityai/stable-diffusion-2-1 stable_diff_gen.py, while sd_mitigation.py loads CompVis/stable-diffusion-v1-4 sd_mitigation.py. Which model made the figures is not stated.
• Sampling settings. The appendix gives guidance 7.5, strength 0.5 and 50 steps app. C. utils/stable_diff_gen.py defines guidance 7.5 and strength 0.5 as arguments stable_diff_gen.py, but the pipeline call passes only num_inference_steps=50 and a generator stable_diff_gen.py. So guidance uses the pipeline default, and strength is not used. The script also switches to a DPM-Solver multistep scheduler stable_diff_gen.py. The paper does not name a scheduler.
• DDPM experiments are not in the repo. Section 5 trains DDPMs with the HuggingFace implementation §5, and the repo has no script for that. diff_train.py instead fine-tunes a latent diffusion model: it encodes images with a VAE, scales the latents by 0.18215, and trains a noise-prediction MSE loss diff_train.py diff_train.py. Its defaults are learning rate 5e-6, batch size 4, constant schedule and 500 warm-up steps diff_train.py diff_train.py.
• Augmentation. The paper says "random horizontal flip and random crop" §5. The code has --random_flip and --center_crop, a centre crop and not a random one diff_train.py.
• Extra analysis in code. diff_retrieval.py also computes entropy, compression and total-variation correlations, and a plot of duplicate weights diff_retrieval.py diff_retrieval.py. The paper does not describe them.
• Mitigation options. --trainspecial, --rand_noise_lam and sd_mitigation.py are mitigation features diff_train.py README.md. The paper does not describe mitigation.
• Synthetic datasets. No script in the repo builds the five synthetic datasets. Helpers for CutMix and Segmix are in utils_ret.py utils_ret.py. Their parameters were not matched against the paper's description.

Pitfalls

• Text and table disagree. The text says Swin wins the real datasets §4.1. Table 1 shows Swin best on rOxford5k, rParis6k and CUB-200, but CLIP ViT-B/16 best on GPR1200 (62.13) and INSTRE (73.19) tab. 1. The text says DINO is best on average, but Table 1 gives it a tie with Swin at 4.1 §4.1 tab. 1. The text says SSCD does best on synthetic datasets. Table 1 shows SSCD best on four of five; DINO wins IN-Cutmix (93.53 vs 89.78) tab. 1.
• CLIP size. The text says the CLIP model is ViT-Base/32 §4. The table says ViT-B/16 tab. 1.
• Dataset name. The text calls one synthetic set IN-Dif-Patch §4. Table 1 calls it IN-Dif-Outpaint tab. 1.
• Threshold wording. Section 7 uses "above 0.5" §7, and Section 9 uses "≥ .5" §9. The difference matters only for scores exactly at 0.5.
• The 12M search is large. A DINO split-product vector is T × d = 197 × 768 = 151,296 values, about 0.6 MB in fp32 per image. Over 12M images that is about 7 TB. The paper says the subset was chosen "to keep storage costs manageable" §2, but gives no index or storage details for its search §7. The code computes one full similarity matrix over all training features diff_retrieval.py diff_retrieval.py. Budget storage or chunk the search before running it at this scale.
• Scope of the copy rate. The search covered the 12M-image LAION Aesthetics v2 6+ subset, which is under 0.6% of the fine-tuning data. Copies from outside that subset exist §9 fig. 12. The authors say the reported rate is an underestimate §9.
• Caption sampling. Captions from LAION can contain key phrases that bring back memorised images. The authors say this may raise the copy rate compared with other caption samplers, and that no unbiased "typical caption" sampler exists to check it §7.
• Figures are selected. The authors state the figure examples come from the highest-similarity set and are not random samples §5 §7.
• The full-data Celeb-A model is external. It is the official checkpoint, not one the authors trained app. B.1. The "no copies" result for that model depends on that checkpoint.
• Self-similarity baseline. Some training images repeat or nearly repeat, with self-similarity above 0.9 §5. That raises the baseline and can hide copies.
• Object-level definition. The definition is subjective. The authors acknowledge grey areas and leave the judgement to readers §1 §3.
• Cost of the experiments. Each DDPM trains for at least 300k steps §5. Hardware and optimiser details are not stated.

Lineage

• Builds on: DINO §4, SSCD §4, Multigrain §4, MoCo v3 §4, VICRegL §4, Swin Transformer §4, CLIP §4, DDPM §5, Stable Diffusion and latent diffusion §2 §6. These are named in the paper and have no cards here.
• Compared against: the ten retrieval extractors in tab. 1 §4. Replication studies of GANs and language models are discussed in §2.
• Follow-up: the code README links the NeurIPS 2023 paper "Understanding and Mitigating Copying in Diffusion Models" (arXiv:2305.20086), which this paper does not describe README.md.
• Superseded by: not known from this wiki.