Synthetic Data from Diffusion Models Improves ImageNet Classification

In plain words

Image generators can now make very convincing pictures from a text prompt. This paper asks whether those pictures are good enough to train another model that recognises objects. It takes a large general-purpose image generator, lightly retrains it on the ImageNet photo collection, and then asks it to draw pictures for each ImageNet category, using only the category name as the prompt.

The drawn pictures are then used as extra training examples for ordinary image classifiers. The authors report that the drawn pictures are judged better by standard generator scores, and that adding them to the real photos makes classifiers more accurate. The gain is bigger for larger drawings, and it stops helping once the synthetic set grows very large.

TL;DR

• Imagen (a text-to-image diffusion cascade) is fine-tuned on ImageNet-1K training images, with fixed class-name prompts §4.1 §4.
• Sampling settings (guidance, log-variance, steps, noise augmentation) are picked by sweeps on FID, IS and classification accuracy §4.2.
• Author-reported: FID 1.76 and IS 239.18 at 256x256 tab. 1; CAS top-1 64.96 at 256 and 69.24 at 1024 tab. 2.
• Adding synthetic images to real training data improves ResNet and ViT top-1 accuracy, with gains up to +1.78 points for ResNet-50 at 1.2M added images tab. 3. Gains shrink or turn negative for very large synthetic sets at 256x256 and 1024x1024 tab. 4.

Problem

Synthetic data is widely used for dense vision tasks, but for ImageNet classification it had not helped. Models trained only on generated samples underperformed models trained on real data §3, and earlier work that used an off-the-shelf pretrained Stable Diffusion model did not improve clean ImageNet accuracy §2. The standard generator metrics (FID, Inception Score) do not reliably predict downstream classification accuracy, so the authors use Classification Accuracy Score (CAS): train a classifier only on generated data and test it on real ImageNet validation images §3 §5.2.

ImageNet classification is a hard benchmark for augmentation because its architectures and training recipes are heavily tuned §1 §4.

Method

Backbone. The model is Imagen, a cascade: a frozen T5-XXL text encoder, a 2B-parameter 64x64 base diffusion model, a 600M super-resolution (SR) model from 64 to 256, and a 400M SR model from 256 to 1024 §4.1. All three diffusion stages use text cross-attention. The base model also receives a pooled text embedding added to its timestep embedding §4.1.

Fine-tuning. Only the 64x64 base and the 64-to-256 SR model are fine-tuned on the ImageNet-1K training split. The text encoder and the 1024 SR stage stay unchanged §4.1. The base model is trained with Adafactor and the SR model with Adam §4.1. Checkpoints are selected by FID on 10K samples from the default Imagen sampler against the ImageNet validation set §4.1.

Prompts. Each class gets its fixed one- or two-word class name, taken from the CLIP class-name list. The authors do not optimise prompts. They argue that naive short prompts give poor data, and handle this by tuning the generator and its sampling parameters instead §4.

Sampling parameters. Four knobs are swept §4.2 app. A.1:

• classifier-free guidance weight;
• the log-variance mixing coefficient, a convex combination of the log upper and lower bounds on the predictive variance, following Eq. 15 of Nichol and Dhariwal, which the paper cites rather than restates app. A.1;
• the number of diffusion steps;
• noise-conditioning augmentation for the SR stages (added to the SR input during training; set to zero at sampling time) app. A.1.

Selection criteria are FID, IS and CAS, measured on the ImageNet validation set §4.2. The chosen values are in the Hyperparameters table.

Generation protocol. For each class, the number of generated images matches the real count for that class, so the class balance of ImageNet is kept. Synthetic sets range from 1.2M images (1x the real training set) to 12M (10x) §4.3.

Evaluation.

• Sample quality: FID and IS, against the training and validation sets as reported in the table columns §5.1 tab. 1.
• CAS: ResNet-50 trained on generated images only, with the CAS recipe (single crop, 90 epochs), tested on real ImageNet validation images §5.2 app. A.4.
• Classifier study: ResNet, ResNet-RS and ViT/DeiT models trained on real, generated, or real plus generated data using each model's published ImageNet recipe §5.3 tab. 3 app. A.4.
• Scaling study: a ResNet-50 trained on real data mixed with 1x to 10x synthetic data, at 64, 256 and 1024 resolution §5.4 fig. 6 tab. 4.

Pseudocode. The paper has no algorithm environment. The sketch below is our own summary of the pipeline in §4 and §5, not the authors' code.

# Synthetic-data augmentation pipeline  (sketch from §4 and §5)
# Input:  real_train (ImageNet-1K train), class_names (1-2 words each)
# Output: classifier trained on real + generated images

# Fine-tune the 64x64 base on real images  [§4.1]
# Batch 2048, 210K steps, Adafactor  [§4.1]
base = finetune_base(imagen_base, real_train, steps=210000, batch=2048)

# Fine-tune the 64 -> 256 SR model  [§4.1]
# 490K steps, Adam  [§4.1]
sr_256 = finetune_sr(imagen_sr_256, real_train, steps=490000)

# Sample per class with the real per-class counts  [§4.3]
# The scale factor k runs from 1 to 10  [§4.3]
generated = []
for each class c in class_names:
    # Prompt is the bare class name; template not stated  [§4]
    prompt = class_names[c]
    for i in range(k * real_count[c]):
        # Base: DDPM, 1000 steps, guidance 1.25, log-variance 0.0  [§4.2]
        low_res = sample_base(base, prompt, guidance=1.25, logvar=0.0)
        # SR 256: DDPM, 1000 steps, guidance 1.0, log-var 0.1  [§4.2]
        # No noise-conditioning augmentation at sampling  [§4.2]
        image = sample_sr(sr_256, low_res, guidance=1.0, logvar=0.1)
        generated.append(image)

# Train the classifier on real plus generated data  [§5.3] [§5.4]
# Mixing ratio per batch is not stated  [§5.3]
pool = real_train + generated
for each epoch in range(130):
    # Standard ResNet-50 recipe with random crops  [§5.4] [app. A.4]
    for batch in shuffled(pool):
        loss = cross_entropy(model(batch.images), batch.labels)
        update(model, loss)

Figures. Figures are not reproduced here. Figure 1 shows a CAS and augmentation teaser fig. 1; Figure 2 compares fine-tuned and vanilla Imagen samples fig. 2; Figures 3 and 4 show sweeps over guidance and log-variance fig. 3 fig. 4; Figure 5 gives per-class accuracy fig. 5; Figure 6 shows the 64x64 scaling curve fig. 6. Appendix figures are cited by section app. A.1 app. A.2.

Hyperparameters

Model-specific recipes for ConvNets and transformers (learning rates, weight decay, mixup and CutMix) are in tab. A.1 and tab. A.2.

Results

Author-reported. Setup: ImageNet-1K class-conditional generation at 64, 256 and 1024 resolution, scored by FID and IS; CAS on ImageNet validation §5.1 §5.2. Baseline numbers for other models are taken from the cited papers, not re-run by the authors §5.1 §5.2.

Sample quality tab. 1

The paper says these are state of the art "without any design changes" to the generator §5.1.

Classification Accuracy Score tab. 2 (ResNet-50 trained on generated images only, top-1 and top-5 on real ImageNet validation, %)

Classifiers with 1.2M real, generated, or combined images tab. 3 (top-1, %; Δ = real + generated minus real only)

Source: tab. 3. Models trained only on generated data do worse than real data in every row.

Scaling synthetic data for ResNet-50 tab. 4 (top-1, mean ± s.d.; change over the real-only baseline in parentheses; the "Train Set" column is read as the total size, see Pitfalls)

Source: tab. 4. At 64x64 the text reports gains that continue up to a total of 12M images §5.4 fig. 6.

Paper vs code

No official code.

The paper names no code repository, project page or code link. Its source (arXiv v1) has no URL or repository reference. The paper uses Imagen, whose code and weights are not linked from this paper. Searches: a GitHub repository search for "imagen" within the google-research organisation returned only unrelated repositories (for example reassessed-imagenet, dp-imagenet). A repository search for "synthetic imagenet diffusion" returned third-party work on knowledge distillation and brain MRI, not from the authors. A broad search for "imagen imagenet" returned unrelated ImageNet repositories. None of these belong to the authors, so none is recorded here.

Pitfalls

• Which FID is quoted. The abstract's FID 1.76 is the "FID train" column in tab. 1. The caption does not define the reference set for that column, and the validation FID for the same model is 2.81 tab. 1. Compare only with the same column.
• Selection on the evaluation set. Guidance, log-variance, steps and noise augmentation are chosen on FID, IS and CAS measured on the ImageNet validation set §4.2. CAS is then reported on that same validation set §5.2. The headline numbers are therefore not from a held-out split. The paper states the selection but does not discuss its effect on the reported numbers.
• Two different real-data baselines. The CAS real-data row (top-1 73.09, tab. 2) follows the CAS recipe (90 epochs, single crop). The real-only ResNet-50 in the classifier tables (76.39, tab. 3 and tab. 4) uses the 130-epoch random-crop recipe §5.4 tab. A.1. Do not mix the two numbers.
• Undefined mixing rule. How real and generated images are combined (ratio per batch, or fixed epoch counts) is not stated for tab. 3 and tab. 4. The 130-epoch run is described only as the conventional recipe §5.4.
• Ambiguous size axis in tab. 4. The "Train Set (M)" column is read as the total size, since the 1.2 row equals the real-only baseline. The paper does not define the column. The conclusion places the drop at "larger than 1M" synthetic images §6. The table shows gains until 4.8M total (3.6M synthetic) at 256x256 and a fall below baseline from 6.0M total. Use the table, not the conclusion's threshold.
• Rounding in tab. 3. For DeiT-B at 224, 82.84 minus 81.79 is 1.05, but the Δ column prints +1.04. Other rows agree with their differences.
• Resolution path for CAS. Images at 1024 are downsampled to 256 and centre-cropped to 224 before the ResNet-50 is trained §5.2 §6. The gain at 1024 is not from a larger classifier input. The paper calls it an open question §6.
• Unstated fine-tuning details. Learning rate, warmup, EMA and the exact prompt template are not stated §4.1 §4. An implementer must choose them and say so.
• Static threshold value. The static clipping threshold used with small guidance is not given. Dynamic thresholding is not used app. A.1.
• Log-variance formula. The mixing coefficient is defined by reference to Nichol and Dhariwal rather than written out here app. A.1. Implement that formula from the cited paper.
• Costs the abstract hides. Fine-tuning uses 256 TPU-v4 chips for 210K and 490K steps at batch 2048 §4.1. Sampling uses 1000 DDPM steps at 256 and the 64x64 base; generation cost for the 1.2M-12M sets is not reported §4.2 §4.3. Generated data is per-class matched to ImageNet counts, so class imbalance is not studied.

Lineage

• Builds on: Imagen, the text-to-image cascade, arXiv:2205.11487 §4.1. The paper uses it unchanged except for fine-tuning and sampling choices.
• Builds on: classifier-free guidance, arXiv:2207.12598 §4.2. Guidance weights are swept per stage.
• Builds on: CAS, the classification accuracy score of Ravuri and Vinyals (2019) §3 §5.2. It is the main proxy for downstream value.
• Compared against: CDM, BigGAN-deep, VQ-VAE-2, LDM-4, DiT-XL/2 and Simple Diffusion at 64 and 256 resolution tab. 1 tab. 2.
• Superseded by: not known from the content of this wiki.