---
kind: concept
slug: autoregressive-image-generation
title: Autoregressive image generation
summary: >-
  Autoregressive image generators turn an image into discrete tokens, or predict pixels directly, and generate with a transformer. Covers raster-order token models, pixel-level models, a text-conditioned single-stream model (DALL-E) and next-scale prediction (VAR), with reported results, a scaling study and the constraints behind each choice.
tags: [image-generation, autoregressive-models, sampling]
papers: ["2102.12092", "2103.04922", "2404.02905"]
related: [vector-quantization, variational-autoencoder, text-to-image-generation, kv-cache-management, classifier-free-guidance, masked-image-modeling, diffusion-models, decoding-strategies]
status: draft
updated_at: 2026-10-31
---

## Overview

An autoregressive image generator predicts the image one piece at a time, each
piece conditioned on the ones before it. Most variants first map the image to a
grid of discrete tokens with a learned tokenizer, then generate those tokens
with a transformer. The usual order is raster: one token at a time, row by row.
The VAR paper argues this order fits images poorly, since an image has no
natural reading order, and that it costs one decoding step per token, so an
n-by-n grid takes O(n²) steps [arxiv:2404.02905 §3.1] [app. B].

The shared invariant: the transformer and the sampling loop are the core of
every variant, and a variant changes mainly the order in which units are
predicted, what a prediction unit is, or what the transformer is conditioned
on. The tokenizer is the usual but not required part: the pixel-level models
in the survey predict pixel values directly [arxiv:2103.04922 §5]. VAR keeps
the VQGAN-style tokenizer architecture and changes the order and the unit
[arxiv:2404.02905 §3.2] [§4]. DALL-E keeps raster order and adds a caption as a
prefix in the same token stream [arxiv:2102.12092 §2.2].

## Variants

### Raster next-token prediction

The baseline predicts one token of the flattened grid at a time, with a
transformer that sees only earlier tokens. In the paper's ImageNet 256
class-conditional table, raster-order rows range from FID 15.78 (VQGAN, 1.4B
parameters, 256 decoding steps) to FID 3.04 (ViTVQ with rejection sampling,
1.7B parameters, 1024 steps) [arxiv:2404.02905 tab. 1]. The table reports these
as the authors' comparisons, not as a fair benchmark of the raster method.

### Pixel-level models without a tokenizer

The survey covers autoregressive models that predict pixel values directly, so
no tokenizer is trained [arxiv:2103.04922 §5.1]. Its table compiles CIFAR-10
numbers from the cited works. They are not re-measured, and the survey gives no
training budget for them [arxiv:2103.04922 tab. 1]. Negative log-likelihood in
bits per dimension is 3.00 for PixelRNN, 3.03 for Gated PixelCNN and 2.66 for
a Sparse Transformer with data augmentation [arxiv:2103.04922 tab. 1]. The
Gated PixelCNN row also has FID 65.93 and the Sparse Transformer row FID 14.74
[arxiv:2103.04922 tab. 1]. The survey marks all three as exact-density models
and rates their sampling as "autoregressive", the lowest class on its scale
[arxiv:2103.04922 tab. 1] [tab. 2].

The survey lists the output order (raster or zig-zag) as one of the output
modelling choices [arxiv:2103.04922 §5.2]. It also notes that multi-scale
designs reduce the cost of high-resolution images [arxiv:2103.04922 §5.1.5].
That idea is close to the coarse-to-fine order of VAR, though VAR works on
tokens, not pixels [arxiv:2404.02905 §3.2].

### Text-conditioned single-stream transformer (DALL-E)

The DALL-E paper is the text-to-image case. A discrete VAE maps each 256x256
image to a 32x32 grid of tokens from an 8192-entry codebook. A decoder-only
transformer then reads the lowercased, BPE-encoded caption (at most 256 tokens)
followed by the image tokens, as one stream [arxiv:2102.12092 §2.1] [§2.2].
Image tokens are still produced in raster order, but the attention is sparse:
most layers use row or column masks, and the last layer uses a convolutional
mask [arxiv:2102.12092 §2.2] [app. B.1] [fig. 11]. The reported model has 12
billion parameters and 64 layers [arxiv:2102.12092 §2.2] [tab. 2]. Card:
[Zero-Shot Text-to-Image Generation](/papers/2102.12092).

At sampling time the model draws 512 candidate grids at temperature 1 and keeps
the best ones by a separate contrastive model [arxiv:2102.12092 §2.6]. The
authors report that human raters preferred its MS-COCO images to DF-GAN's 90%
of the time for realism and 93% for caption match. These are author-reported
votes on 1,000 caption comparisons [arxiv:2102.12092 §3.1] [fig. 7] [app. F]. On
the CUB test captions the paper reports FID about 40 points worse than the best
prior method [arxiv:2102.12092 §3.1] [fig. 9b].

### Next-scale prediction (VAR)

VAR predicts a whole token map at each scale, from the coarsest to the finest.
Each map is predicted in one pass, conditioned on all earlier maps through a
block-causal attention mask [arxiv:2404.02905 §3.2] [eq. 6]. The tokenizer is
multi-scale: it quantizes the residual at each scale and removes the quantized
part before the next scale. All scales share one codebook of 4096 entries
[arxiv:2404.02905 §3.2] [§4] [alg. 1]. The default schedule has 10 steps, with
map sizes from 1x1 to 16x16 [arxiv:2404.02905 tab. 1]. The transformer is
decoder-only, GPT-2-like, with adaptive layer norm for the class condition
[arxiv:2404.02905 §4] [eq. 7]. Card: [Visual Autoregressive Modeling](/papers/2404.02905).

Reported ImageNet 256 class-conditional FID: 1.92 for the 2.0B model without
rejection sampling and 1.73 with it. The diffusion baseline DiT-XL/2 reports
2.27 and uses 250 steps [arxiv:2404.02905 tab. 1]. At 512×512 the authors
report FID 2.63 for VAR-d36 with a single shared AdaLN layer, against 3.04 for
DiT-XL/2 [arxiv:2404.02905 tab. 2]. The shared layer is used because of compute
limits [tab. 2 caption].

## When to use which

The VAR results are class-conditional on ImageNet. The DALL-E results are
zero-shot on MS-COCO captions and on CUB. The survey's pixel-level numbers are
on CIFAR-10. The papers measure different things, so their numbers should not
be compared directly.

| Situation | Use | Why |
|---|---|---|
| Class-conditional generation with few sequential decoding steps | VAR-d30 | FID 1.92 (no rejection sampling) with 10 steps; the diffusion rows use 250 [arxiv:2404.02905 tab. 1] |
| Smaller compute budget | VAR-d16 or VAR-d20 | FID 3.30 at 310M and 2.57 at 600M; the time column is relative to d30 and the hardware is not stated [arxiv:2404.02905 tab. 1] |
| Exact likelihood on pixels, with no tokenizer to train | Pixel-level models from the survey | Exact-density rows with NLL 2.66 to 3.03 bpd on CIFAR-10, author-compiled [arxiv:2103.04922 tab. 1]; sampling is rated the slowest class [tab. 2] |
| Text-conditioned generation with a caption prefix in one stream | Single-stream design of DALL-E | Human preference over DF-GAN on MS-COCO, author-reported [arxiv:2102.12092 §3.1]; trained on about 250 million pairs [§2.3] |
| Text-conditioned generation, to run the model | Not possible from this evidence | The 12B transformer is not released; the code repository has only the dVAE [arxiv:2102.12092 §2.2] [code: README.md#L5] |
| Text-conditioned generation with a published sampler | Not covered by these cards | VAR reports only class-conditional results [arxiv:2404.02905 tab. 1] [tab. 2] |
| Inpainting or extending an image without finetuning | VAR, as a qualitative demo | Shown with examples only; the paper reports no metric for these [arxiv:2404.02905 §6] [fig. 8] |
| Reproducing the headline numbers | Budget extra work | The tokenizer training code and the rejection-sampling code are not in the repository [code: train.py#L93-L99] |
| Reproducing the DALL-E sample quality | Budget extra work | Transformer, training code, reranker and data are not released; the paper's dVAE is the only code [code: README.md#L5] |

For the scaling question, the paper fits power laws of test loss and error
against model size and against training compute, over 12 model sizes from 18.5M
to 2.0B parameters [arxiv:2404.02905 §5.2] [eq. 11]–[eq. 16]. Use these fits to
judge trends, not as exact predictions: the paper gives no confidence intervals
for the exponents [arxiv:2404.02905 §5.2].

Reranking is a cost trade-off. DALL-E's main results draw 512 candidates per
caption, and the paper reports gains that flatten after 32 in its reranking
plot [arxiv:2102.12092 §3.1] [fig. 9c]. The paper does not say what k was used
for the main results [arxiv:2102.12092 §2.6].

## Implementation sketch

The paper describes the method in words and equations. The points below follow
the paper's Algorithms 1 and 2 [alg. 1] and the training setup in §3.2. The
sampling schedule for guidance is not in the paper; it is in the repository
[code: models/var.py#L161-L173] and described in the
[VAR card](/papers/2404.02905).

```pseudocode
# Multi-scale encoding  [alg. 1]
def encode_scales(image):
    # Continuous feature map from the tokenizer encoder  [§3.2]
    residual = encoder(image)
    token_maps = []
    for size in scale_sizes:
        # Pool the residual to this scale, then take nearest codes  [§3.2]
        pooled = area_resize(residual, size)
        tokens = nearest_code(pooled, codebook)
        token_maps.append(tokens)
        # Remove what these codes explain before the next scale  [alg. 1]
        residual = residual - upsample(lookup(tokens), 16)
    return token_maps

# One training step  [§3.2] [eq. 6]
def training_step(images, labels):
    token_maps = encode_scales(images)
    # Each scale is predicted from the summed features of the coarser ones
    prefix = teacher_forcing_input(token_maps)
    # Mean cross-entropy over all tokens  [§3.2]
    logits = transformer(labels, prefix)
    return cross_entropy(logits, flatten(token_maps))
```

The single-stream variant changes the input rather than the sampler. Its
prefix is the caption tokens, padded to a fixed length, followed by the flat
image tokens, and the loss is a weighted sum over the two parts
[arxiv:2102.12092 §2.2]. The full DALL-E training step is in its
[card](/papers/2102.12092).

Sampling for VAR runs the transformer once per scale and predicts all tokens of
that scale together. Classifier-free guidance works on the logits of each scale.
The paper uses a guidance ratio of 2.0 without describing a schedule
[arxiv:2404.02905 tab. 3]; the repository ramps the guidance up with the scale
index [code: models/var.py#L161-L173]. See the
[classifier-free guidance](/concepts/classifier-free-guidance) page for the
general form.

## Contradictions

No direct disagreement: the papers evaluate different tasks and datasets, so
their numbers do not contradict each other. Inside and across the papers:

- Table 3's last row has FID 1.73, which matches the rejection-sampling row of
  Table 1. The paper does not say whether rejection sampling was used there
  [arxiv:2404.02905 tab. 3] [tab. 1].
- The paper states a base learning rate of 1e-4 per 256 batch for all sizes
  [arxiv:2404.02905 §4]. The repository's training commands use 8e-5 for d24,
  d30 and d36 [code: README.md].
- The repository's README gives FID 1.97 for d30 where Table 1 gives 1.92. The
  cause is not stated [code: README.md#L91-L95] [arxiv:2404.02905 tab. 1].
- The survey rates autoregressive sampling as the slowest class [arxiv:2103.04922
  tab. 2], while VAR's time column gives raster VQGAN 24 and VAR-d30 1, relative
  to VAR-d30 [arxiv:2404.02905 tab. 1]. These come from different models,
  datasets and hardware, and the survey's ratings are coarse author judgments,
  so they are not a contradiction. Do not use the survey's rating to predict
  VAR's speed.

## Papers

- [Zero-Shot Text-to-Image Generation](/papers/2102.12092): a 12B single-stream transformer over caption and image tokens, with a discrete VAE and reranking; author-reported MS-COCO human preference.
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](/papers/2103.04922): a survey whose autoregressive section and CIFAR-10 table cover pixel-level models, with numbers compiled from cited works.
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](/papers/2404.02905): next-scale prediction with a multi-scale tokenizer, a scaling study, and zero-shot editing examples.