Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

In plain words

Image generators built on transformers usually make a picture the way a language model writes a sentence: one small token at a time, in reading order, from top left to bottom right. That is slow, and it fits images badly, because an image has no natural reading order.

VAR works more like a sketch artist. It first makes a tiny, very coarse version of the picture, then a slightly finer one, and keeps going until it reaches full resolution. Each step fills in a whole level at once, using all the coarser levels before it. The predictor is a standard GPT-style transformer.

The paper reports that this is faster and gives better images than the earlier token-by-token models on ImageNet. It also reports that bigger versions follow the same kind of predictable scaling trend that language models show, and that the trained model can fill in or extend parts of an image without extra training.

TL;DR

• VAR predicts a whole token map at each scale, from coarse to fine, instead of predicting single tokens in raster order §3.1 §3.2.
• The tokenizer is a VQGAN-style autoencoder with a multi-scale quantizer and a shared codebook of 4096 codes. The default scales are 1, 2, 3, 4, 5, 6, 8, 10, 13 and 16 (10 steps) §3.2 §4 var.py.
• The transformer is decoder-only and GPT-2-like, with adaptive layer norm (AdaLN) for conditioning. Width is 64 times the depth, and the number of heads equals the depth §4 eq. 7.
• Author-reported on ImageNet 256 class-conditional generation: FID 1.73 for the 2.0B model with rejection sampling and 1.92 without it. DiT-XL/2 reaches FID 2.27 tab. 1.
• Author-reported: test loss and token error follow power laws in model size and training compute §5.2 eq. 11–eq. 16. Zero-shot in-painting, out-painting and class-conditional editing work without finetuning, shown qualitatively §6 fig. 8.

Problem

Standard autoregressive image models first quantize the image into a grid of discrete tokens, then flatten the grid into a sequence and predict it token by token §3.1 eq. 1 eq. 2 eq. 3. The paper names four problems with this §3.1:

1. Premise violation. The quantized tokens keep correlations in every direction, but next-token models assume each token depends only on its prefix. The paper shows this with the attention map of the VQGAN encoder, where every token depends on every other token fig. 9 app. A.
2. Limited zero-shot tasks. Because the model only goes forward in the token order, it cannot, for example, predict the top of an image from the bottom §3.1.
3. Structure loss. Flattening breaks the spatial neighborhood of each token §3.1.
4. Cost. Generating an n-by-n token grid takes O(n²) decoding steps and O(n⁶) total computation §3.1 app. B.

Method

The method has two separate training stages fig. 4:

• Stage 1: an autoencoder with a multi-scale quantizer turns an image into K coarse-to-fine token maps. Its training uses the compound loss eq. 5.
• Stage 2: a VAR transformer is trained to predict the maps in order, one scale at a time, with cross-entropy loss eq. 6 fig. 4.

The autoencoder is the VQGAN architecture with a modified multi-scale quantization layer §3.2. The paper says the tokenizer follows the VQGAN baseline's training setup §4. The code in this repository does not include the autoencoder's training loop; train.py loads a pretrained checkpoint from Hugging Face train.py.

\mathcal{L} = \|im - \hat{im}\|_2 + \|f - \hat{f}\|_2 + \lambda_{\text{P}} \mathcal{L}_{\text{P}}(\hat{im}) + \lambda_{\text{G}} \mathcal{L}_{\text{G}}(\hat{im}) eq. 5

Here L_P is a perceptual loss and L_G a discriminator loss. The values of λ_P and λ_G are not stated §3.1.

Next-scale likelihood. VAR factorizes the probability over token maps instead of tokens §3.2:

p(r_1, r_2, \dots, r_K) = \prod_{k=1}^{K} p(r_k \mid r_1, \dots, r_{k-1}) eq. 6

All tokens of r_k are predicted in parallel, conditioned on the earlier maps and on a learned position embedding for that scale §3.2. Training uses a block-wise causal mask: tokens in scale k may attend to scales up to and including k §3.2 var.py. Inference uses KV caching and needs no mask §3.2 var.py var.py.

Multi-scale encoding (Algorithm 1). The residual is pooled to each scale, quantized, and then its quantized reconstruction is removed before the next scale alg. 1 §3.2:

# Multi-scale VQ encoding  [alg. 1]
def encode_multiscale(image):
    # Continuous feature map from the VQGAN-style encoder  [§3.2]
    residual = encoder(image)                    # (B, C_vae, 16, 16)
    token_maps = []
    for k, size in enumerate(scale_sizes):
        # Area-pool the residual to this scale, find nearest codes  [§3.2]
        pooled = area_resize(residual, size)     # (B, C_vae, size, size)
        tokens = nearest_code(pooled, codebook)  # (B, size, size)
        token_maps.append(tokens)
        # Look up codes, bring them to 16x16, apply the shared 3x3 conv
        # (phi) first  [code: models/quant.py#L150-L163]
        restored = bicubic_resize(lookup(tokens), 16)
        residual = residual - phi(k)(restored)
    return token_maps

Two details matter: the codes are looked up in full resolution (16x16), so each scale's contribution is summed, not replaced; and the scales are the paper's K steps in order §3.2.

Decoding (Algorithm 2). The inverse sums the convolved, upsampled code vectors of all scales into one feature map, then runs the VQGAN decoder alg. 2 §3.2. The shared codebook has V = 4096 entries across all scales §3.2 §4.

Transformer training. The input is a start token, the class embedding, followed by the earlier scales. Each scale's input is the summed, downsampled feature map of the scales before it, not the embeddings of discrete tokens quant.py trainer.py. The label is replaced by a null class with probability 0.1 so that classifier-free guidance can be used var.py. The loss is the mean cross-entropy over all L tokens, with equal weight per token trainer.py trainer.py.

# One training step of the VAR transformer  [§3.2] [eq. 6]
def training_step(images, labels):
    # Encode the image into K discrete token maps  [alg. 1]
    token_maps = encode_multiscale(images)
    # Teacher-forcing input: summed features of the earlier scales
    # [code: models/quant.py#L169-L184]
    prefix = teacher_forcing_input(token_maps)   # (B, L - 1, C_vae)
    # Replace the label by a null class 10% of the time
    # [code: models/var.py#L201]
    labels = drop_labels(labels, p=0.1)
    # One forward pass with the block-causal mask  [§3.2]
    logits = var_transformer(labels, prefix)     # (B, L, V)
    # Mean cross-entropy, equal weight per token
    # [code: trainer.py#L112-L120]
    targets = flatten(token_maps)                # (B, L)
    return cross_entropy(logits, targets)

Sampling with classifier-free guidance. Sampling runs one scale at a time. Guidance is applied to the logits and grows with the scale index var.py. The paper's text does not describe this schedule §3.2:

# Sample one image with CFG and KV caching  [code: models/var.py#L127-L190]
def sample_image(label, cfg, top_k, top_p):
    # Class embedding as the start token, plus a null-class copy  [§3.2]
    start = class_embedding([label, NULL_CLASS])
    # Running feature map starts at zero  [code: models/var.py#L157]
    feature = zeros(C_vae, 16, 16)
    next_input = start_embedding(start)
    for k, size in enumerate(scale_sizes):
        # One pass predicts all size x size tokens of this scale  [§3.2]
        logits = transformer_step(next_input)    # (2, size*size, V)
        # Guidance grows linearly with the scale index
        # [code: models/var.py#L172]
        t = cfg * k / (K - 1)
        logits = (1 + t) * logits[cond] - t * logits[uncond]
        # Sample the tokens of this scale  [code: models/helpers.py#L6-L19]
        tokens = sample_top_k_top_p(logits, top_k, top_p)
        # Add the scale's residual, build the next input
        # [code: models/quant.py#L187-L196]
        feature, next_input = add_scale(feature, tokens, k)
    # Decode the summed features to pixels  [code: models/var.py#L190]
    return vqvae_decode(feature)

Attention. The paper normalizes queries and keys to unit length to stabilize training §4. The code also multiplies the queries by a learned per-head scale, which starts at 4 and is capped at 100 basic_var.py basic_var.py. See the comparison in Paper vs code.

Architecture. The transformer has depth d, width 64d, and d heads. Each block has self-attention, a feed-forward network, and AdaLN conditioning §4 eq. 7 [code: models/init.py#L19-L20]. The paper does not use RoPE, SwiGLU or RMSNorm §4.

w = 64d, \quad h = d, \quad dr = 0.1 \cdot d / 24 eq. 7

N(d) = 18 \, d w^2 = 73728 \, d^3 eq. 8

Here N(d) counts the main parameters. The paper's footnote gives about 49152 d³ when AdaLN is shared across blocks (used for 512×512) §4 eq. 8.

Hyperparameters

Values in the Where column are the paper's unless marked code. "not stated" means the paper and the code's defaults do not give it, or the paper's text does not.

Results

Author-reported. Setup: class-conditional ImageNet generation at 256×256 and 512×512, scored with FID, IS, precision and recall. The 256 table uses 50,000 generated samples. The time column is wall-clock inference time relative to VAR-d30 (set to 1), so the absolute hardware is not given in the text. Baseline numbers marked † are quoted from MaskGIT tab. 1.

ImageNet 256×256, class-conditional tab. 1

"-re" marks rows that use rejection sampling tab. 1 caption. The paper's text does not describe rejection sampling beyond the caption.

ImageNet 512×512, class-conditional tab. 2

"-s" means one shared AdaLN layer, used because of compute limits tab. 2 caption.

Ablation on ImageNet 256×256 tab. 3

The paper labels two rows "5"; they are kept as printed. Row 6's FID of 1.73 matches the "-re" row of Table 1, not the 1.92 row, but the table does not say whether rejection sampling was used there.

Power-law scaling §5.2. Models range from d = 6 to d = 30, which is 18.5M to 2.0B parameters, 12 sizes in total. The test set is the ImageNet validation set of 50,000 images §5.2. Fitted laws against parameters N eq. 11 eq. 12:

L_{\text{last}} = (2.0 \cdot N)^{-0.23} \quad \text{and} \quad L_{\text{avg}} = (2.5 \cdot N)^{-0.20} eq. 11

\text{Err}_{\text{last}} = (4.9 \cdot 10^2 N)^{-0.016} \quad \text{and} \quad \text{Err}_{\text{avg}} = (6.5 \cdot 10^2 N)^{-0.010} eq. 12

Fitted laws against optimal training compute C_min in PFLOPs §5.2 eq. 13 eq. 14 eq. 15 eq. 16:

L_{\text{last}} = (2.2 \cdot 10^{-5} C_{\min})^{-0.13} eq. 13

L_{\text{avg}} = (1.5 \cdot 10^{-5} C_{\min})^{-0.16} eq. 14

\text{Err}_{\text{last}} = (8.1 \cdot 10^{-2} C_{\min})^{-0.0067} eq. 15

\text{Err}_{\text{avg}} = (4.4 \cdot 10^{-2} C_{\min})^{-0.011} eq. 16

The paper reports Pearson correlations near −0.998 for log N against log L or log Err fig. 5, and near −0.99 for log C_min against log L or log Err fig. 6. The paper says the compute relation holds across six orders of magnitude of C_min §5.2. No confidence intervals are given.

Zero-shot tasks §6 fig. 8. VAR-d30 does in-painting and out-painting by forcing ground-truth tokens outside the mask and generating only inside it, with no class label. It also does class-conditional editing inside a bounding box. These results are qualitative only; no numbers are reported §6.

Paper vs code

Code: official repository FoundationVision/VAR, MIT license, commit 78b95394fc5896192e3a003e4b295f8ea743c48f. The repository's LICENSE file says "MIT License" LICENSE.

• Attention scale. The paper says queries and keys are normalized to unit vectors §4. The code also multiplies the queries by a learned per-head scale, which starts at 4 and is capped at 100, and uses an attention scale of 1 basic_var.py basic_var.py.
• Tokenizer convolutions. The paper describes K extra convolutions φ_k, one per scale §3.2 §4. The code uses 4 convolutions shared across the scales, selected by scale position train.py quant.py quant.py.
• CFG schedule. The paper uses CFG ratio 2.0 without describing a schedule tab. 3. The code scales it per scale: t = cfg × k/(K−1), so the first scale is unguided and the last is fully guided var.py var.py.
• Sampling settings. The ablation uses top-k 600 and CFG 2.0 tab. 3. The repository's FID command uses cfg 1.5, top-p 0.96 and top-k 900 README.md. The demo notebook uses cfg 4, top-k 900 and top-p 0.95 demo_sample.ipynb. The paper does not say which settings produced Table 1.
• Reported FIDs. The repository's model table gives FID 3.55 (d16), 2.95 (d20), 2.33 (d24), 1.97 (d30) and 1.80 (d30 with "-re") on 256×256 README.md. Table 1 gives 3.30, 2.57, 2.09, 1.92 and 1.73 for the same rows tab. 1. The 512×512 figure of 2.63 for d36-s matches tab. 2. The cause of the 256×256 difference is not stated.
• Training learning rate. The paper gives a base rate of 1e-4 per 256 batch for all sizes §4. The repository's training commands use 8e-5 for d24, d30 and d36 README.md Training Scripts.
• Rejection sampling. The "-re" rows use rejection sampling tab. 1. The repository contains no rejection-sampling code (a search for "rejection" finds nothing), so those numbers cannot be reproduced from it.
• Tokenizer training. The paper describes training the tokenizer with the compound loss on OpenImages §3.1 §4. The repository only trains the VAR transformer and loads a pretrained tokenizer checkpoint train.py. The tokenizer's training code and loss weights are not in the repository.
• Drop rate. The paper's "dr" is applied in the code as stochastic depth (drop-path), not as dropout. Dropout is 0 [code: models/init.py#L21] [code: models/init.py#L31].
• Label dropout. The paper does not mention label dropout for CFG training. The code drops 10% of labels to a null class var.py.
• Interpolation. The paper does not say how it resizes maps. The code uses area interpolation when downsampling and bicubic when upsampling quant.py quant.py quant.py.

No other discrepancies found between the sampling loop, the residual update and Algorithm 1 and 2.

Pitfalls

• Teacher-forcing input. The transformer input for each scale is the summed, area-downsampled continuous feature map of the earlier scales, not the embedding of the earlier discrete tokens. The figure caption says the transformer takes ([s], r_1, ..., r_{K-1}) as input fig. 4. Feeding embeddings of the discrete tokens will not match training quant.py trainer.py.
• Resizing modes. Area interpolation for downsampling and bicubic for upsampling are used both in training and in sampling. The paper does not name either. Changing them changes the outputs quant.py quant.py.
• CFG is not constant. Copying a fixed cfg of 2.0 into a sampler does not match the code. The guidance is 0 at scale 1 and cfg at the last scale var.py.
• Sampling settings for the headline numbers are unstated. Use the code's FID command (cfg 1.5, top-p 0.96, top-k 900) if you want to match the repository, and expect small differences from Table 1 README.md.
• Loss weighting. Every token gets weight 1/L in the loss. The largest scale carries most of the loss: with the default scales, L = 680 tokens and the 16×16 map gives 256 of them, about 38% of the loss weight. This is computed from the code's scale list, not stated in the paper trainer.py var.py.
• Optimizer details the paper omits. The code applies zero weight decay to biases, 1-D parameters, and the position, level, start and scale_mul parameters train.py lr_control.py. It also clips gradients at 2.0, uses no warmup by default, and uses the lin0 schedule arg_util.py arg_util.py arg_util.py.
• Initialization. The automated init uses a standard deviation of sqrt(1/(3C)) for linear and embedding weights, and the attention and feed-forward output projections are scaled by 1/sqrt(2·depth) var.py var.py var.py.
• Complexity claim. The O(n⁴) bound assumes the scale sizes grow geometrically with a fixed ratio app. B. The default scales (1, 2, 3, 4, 5, 6, 8, 10, 13, 16) do not, so the bound is a rough guide var.py.
• Speed claims. The time column is relative to VAR-d30 and the hardware is not stated. Do not compare these times across papers tab. 1.
• Zero-shot tasks have no numbers. The paper shows examples, not metrics §6 fig. 8. Do not treat them as measured results.
• Scaling-law fits. The fits use 12 model sizes §5.2. The paper states no confidence intervals for the fitted exponents, and the exponents differ between the last scale and the average over scales.
• Tokenizer is not in the repository. To retrain the full pipeline you need your own tokenizer training code. The paper's loss weights are not stated, and the repository has no tokenizer trainer train.py.
• Tokenizer is fixed. The authors kept the VQGAN tokenizer architecture and training unchanged to isolate the effect of the new ordering, and name tokenizer improvements as future work §8 §3.2.

Lineage

• Builds on: VQGAN for the tokenizer architecture and the autoregressive baseline §3.1 tab. 1; the residual quantization idea of RQ-Transformer §3.2; the GPT-2 decoder-only transformer §4; AdaLN from the StyleGAN family and DiT §4.
• Compared against: DiT and L-DiT (diffusion transformers), MaskGIT, ADM, BigGAN, GigaGAN, StyleGAN-XL, LDM, RCG, ViT-VQGAN, VQVAE-2 and RQ-Transformer tab. 1 tab. 2.
• Superseded by: none recorded in this wiki.