Autoregressive image generation

Overview

An autoregressive image generator predicts the image one piece at a time, each piece conditioned on the ones before it. Most variants first map the image to a grid of discrete tokens with a learned tokenizer, then generate those tokens with a transformer. The usual order is raster: one token at a time, row by row. The VAR paper argues this order fits images poorly, since an image has no natural reading order, and that it costs one decoding step per token, so an n-by-n grid takes O(n²) steps 2404.02905 §3.1 app. B.

The shared invariant: the transformer and the sampling loop are the core of every variant, and a variant changes mainly the order in which units are predicted, what a prediction unit is, or what the transformer is conditioned on. The tokenizer is the usual but not required part: the pixel-level models in the survey predict pixel values directly 2103.04922 §5. VAR keeps the VQGAN-style tokenizer architecture and changes the order and the unit 2404.02905 §3.2 §4. DALL-E keeps raster order and adds a caption as a prefix in the same token stream 2102.12092 §2.2.

Variants

Raster next-token prediction

The baseline predicts one token of the flattened grid at a time, with a transformer that sees only earlier tokens. In the paper's ImageNet 256 class-conditional table, raster-order rows range from FID 15.78 (VQGAN, 1.4B parameters, 256 decoding steps) to FID 3.04 (ViTVQ with rejection sampling, 1.7B parameters, 1024 steps) 2404.02905 tab. 1. The table reports these as the authors' comparisons, not as a fair benchmark of the raster method.

Pixel-level models without a tokenizer

The survey covers autoregressive models that predict pixel values directly, so no tokenizer is trained 2103.04922 §5.1. Its table compiles CIFAR-10 numbers from the cited works. They are not re-measured, and the survey gives no training budget for them 2103.04922 tab. 1. Negative log-likelihood in bits per dimension is 3.00 for PixelRNN, 3.03 for Gated PixelCNN and 2.66 for a Sparse Transformer with data augmentation 2103.04922 tab. 1. The Gated PixelCNN row also has FID 65.93 and the Sparse Transformer row FID 14.74 2103.04922 tab. 1. The survey marks all three as exact-density models and rates their sampling as "autoregressive", the lowest class on its scale 2103.04922 tab. 1 tab. 2.

The survey lists the output order (raster or zig-zag) as one of the output modelling choices 2103.04922 §5.2. It also notes that multi-scale designs reduce the cost of high-resolution images 2103.04922 §5.1.5. That idea is close to the coarse-to-fine order of VAR, though VAR works on tokens, not pixels 2404.02905 §3.2.

Text-conditioned single-stream transformer (DALL-E)

The DALL-E paper is the text-to-image case. A discrete VAE maps each 256x256 image to a 32x32 grid of tokens from an 8192-entry codebook. A decoder-only transformer then reads the lowercased, BPE-encoded caption (at most 256 tokens) followed by the image tokens, as one stream 2102.12092 §2.1 §2.2. Image tokens are still produced in raster order, but the attention is sparse: most layers use row or column masks, and the last layer uses a convolutional mask 2102.12092 §2.2 app. B.1 fig. 11. The reported model has 12 billion parameters and 64 layers 2102.12092 §2.2 tab. 2. Card: Zero-Shot Text-to-Image Generation.

At sampling time the model draws 512 candidate grids at temperature 1 and keeps the best ones by a separate contrastive model 2102.12092 §2.6. The authors report that human raters preferred its MS-COCO images to DF-GAN's 90% of the time for realism and 93% for caption match. These are author-reported votes on 1,000 caption comparisons 2102.12092 §3.1 fig. 7 app. F. On the CUB test captions the paper reports FID about 40 points worse than the best prior method 2102.12092 §3.1 fig. 9b.

Next-scale prediction (VAR)

VAR predicts a whole token map at each scale, from the coarsest to the finest. Each map is predicted in one pass, conditioned on all earlier maps through a block-causal attention mask 2404.02905 §3.2 eq. 6. The tokenizer is multi-scale: it quantizes the residual at each scale and removes the quantized part before the next scale. All scales share one codebook of 4096 entries 2404.02905 §3.2 §4 alg. 1. The default schedule has 10 steps, with map sizes from 1x1 to 16x16 2404.02905 tab. 1. The transformer is decoder-only, GPT-2-like, with adaptive layer norm for the class condition 2404.02905 §4 eq. 7. Card: Visual Autoregressive Modeling.

Reported ImageNet 256 class-conditional FID: 1.92 for the 2.0B model without rejection sampling and 1.73 with it. The diffusion baseline DiT-XL/2 reports 2.27 and uses 250 steps 2404.02905 tab. 1. At 512×512 the authors report FID 2.63 for VAR-d36 with a single shared AdaLN layer, against 3.04 for DiT-XL/2 2404.02905 tab. 2. The shared layer is used because of compute limits tab. 2 caption.

When to use which

The VAR results are class-conditional on ImageNet. The DALL-E results are zero-shot on MS-COCO captions and on CUB. The survey's pixel-level numbers are on CIFAR-10. The papers measure different things, so their numbers should not be compared directly.

For the scaling question, the paper fits power laws of test loss and error against model size and against training compute, over 12 model sizes from 18.5M to 2.0B parameters 2404.02905 §5.2 eq. 11–eq. 16. Use these fits to judge trends, not as exact predictions: the paper gives no confidence intervals for the exponents 2404.02905 §5.2.

Reranking is a cost trade-off. DALL-E's main results draw 512 candidates per caption, and the paper reports gains that flatten after 32 in its reranking plot 2102.12092 §3.1 fig. 9c. The paper does not say what k was used for the main results 2102.12092 §2.6.

Implementation sketch

The paper describes the method in words and equations. The points below follow the paper's Algorithms 1 and 2 alg. 1 and the training setup in §3.2. The sampling schedule for guidance is not in the paper; it is in the repository var.py and described in the VAR card.

# Multi-scale encoding  [alg. 1]
def encode_scales(image):
    # Continuous feature map from the tokenizer encoder  [§3.2]
    residual = encoder(image)
    token_maps = []
    for size in scale_sizes:
        # Pool the residual to this scale, then take nearest codes  [§3.2]
        pooled = area_resize(residual, size)
        tokens = nearest_code(pooled, codebook)
        token_maps.append(tokens)
        # Remove what these codes explain before the next scale  [alg. 1]
        residual = residual - upsample(lookup(tokens), 16)
    return token_maps

# One training step  [§3.2] [eq. 6]
def training_step(images, labels):
    token_maps = encode_scales(images)
    # Each scale is predicted from the summed features of the coarser ones
    prefix = teacher_forcing_input(token_maps)
    # Mean cross-entropy over all tokens  [§3.2]
    logits = transformer(labels, prefix)
    return cross_entropy(logits, flatten(token_maps))

The single-stream variant changes the input rather than the sampler. Its prefix is the caption tokens, padded to a fixed length, followed by the flat image tokens, and the loss is a weighted sum over the two parts 2102.12092 §2.2. The full DALL-E training step is in its card.

Sampling for VAR runs the transformer once per scale and predicts all tokens of that scale together. Classifier-free guidance works on the logits of each scale. The paper uses a guidance ratio of 2.0 without describing a schedule 2404.02905 tab. 3; the repository ramps the guidance up with the scale index var.py. See the classifier-free guidance page for the general form.

Contradictions

No direct disagreement: the papers evaluate different tasks and datasets, so their numbers do not contradict each other. Inside and across the papers:

• Table 3's last row has FID 1.73, which matches the rejection-sampling row of Table 1. The paper does not say whether rejection sampling was used there 2404.02905 tab. 3 tab. 1.
• The paper states a base learning rate of 1e-4 per 256 batch for all sizes 2404.02905 §4. The repository's training commands use 8e-5 for d24, d30 and d36 README.md.
• The repository's README gives FID 1.97 for d30 where Table 1 gives 1.92. The cause is not stated README.md 2404.02905 tab. 1.
• The survey rates autoregressive sampling as the slowest class [arxiv:2103.04922 tab. 2], while VAR's time column gives raster VQGAN 24 and VAR-d30 1, relative to VAR-d30 2404.02905 tab. 1. These come from different models, datasets and hardware, and the survey's ratings are coarse author judgments, so they are not a contradiction. Do not use the survey's rating to predict VAR's speed.

Papers

• Zero-Shot Text-to-Image Generation: a 12B single-stream transformer over caption and image tokens, with a discrete VAE and reranking; author-reported MS-COCO human preference.
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models: a survey whose autoregressive section and CIFAR-10 table cover pixel-level models, with numbers compiled from cited works.
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction: next-scale prediction with a multi-scale tokenizer, a scaling study, and zero-shot editing examples.