Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis

In plain words

Infinity is a text-to-image model that draws in stages. It first sketches the picture at a very coarse size, then adds finer detail at each stage, a bit like a blurry photo sharpening step by step.

Each detail is stored as a row of on/off bits rather than as one pick from a very long list. Using bits lets the model use a vocabulary of codes so large that a normal lookup table would not fit in memory. Each bit gets its own small yes/no decision. During training the model is also shown slightly wrong versions of its own earlier stages, so it learns to recover instead of letting an early mistake spread into the final image.

The authors report that this beats several diffusion models on prompt-following benchmarks and that it generates a 1024x1024 image faster than the diffusion baselines they compare against.

TL;DR

• Autoregressive text-to-image model: it predicts bit labels from a binary tokenizer with 2^32 codes, scale by scale, conditioned on Flan-T5 text embeddings §3.1 §4.5.
• A per-bit two-way head replaces a single softmax over the whole vocabulary (the Infinite-Vocabulary Classifier) §3.3.
• Bitwise self-correction flips some training bits and re-quantizes them, to close the gap between teacher-forced training and free generation §3.4 alg. 2.
• Author-reported: GenEval 0.73 (with prompt rewriting), DPG 83.46, ImageReward 0.962, HPSv2.1 32.25, and 0.8 s at 1024x1024 for a 2B model tab. 1 tab. 2.
• The released training script is a single debug configuration on toy data, not the paper's three-stage schedule. Use the paper's Hyperparameters and the Pitfalls below before reproducing numbers.

Problem

Autoregressive image models that predict one token at a time lose detail when they use a small discrete vocabulary, and their raster-order generation is slow at high resolution §1. Index-wise tokens (one integer per position) also give fuzzy training targets: a tiny change in a near-zero feature can flip the whole integer §3.3. Teacher forcing, which conditions each stage on the true earlier stages during training, causes a train/test mismatch, because at inference the model conditions on its own mistakes, and those mistakes propagate to later scales §3.4.

The method needs a binary tokenizer, trained with an entropy penalty, and a text encoder (Flan-T5) §3.1 §3.2.

Method

Infinity keeps the next-scale formulation of VAR and changes three parts: the tokenizer's codes, the output classifier, and the training inputs. Section numbers refer to the paper; code paths are relative to the repository root at code_commit.

1. Multi-scale residual bit tokenizer §3.2 eq. 1 eq. 4 alg. 1. The image is encoded to a feature map with stride s. The map is quantized at K scales, each scale quantizing the residual the previous scales did not explain. Quantization is binary (BSQ): each channel becomes a sign, and the code is normalized to unit length, so each entry is ±1/sqrt(d) eq. 4. The running reconstruction is the sum of upsampled residuals eq. 1. The vocabulary is 2^d. The paper's text-to-image model uses d = 32 §4.5; the tokenizer ablation goes up to d = 64 §4.4 tab. 4.
2. Next-scale autoregressive likelihood §3.1 eq. 2 eq. 3. The joint probability factorizes over scales, each scale conditioned on earlier scales and the text embedding Ψ(t). The first scale is driven by a start token projected from the text embedding. For scale k, the transformer input is the previous reconstruction downsampled to that scale eq. 3.
3. Transformer block §3.1. Each block has RoPE2d, self-attention, cross attention to the text embedding, and a feed-forward layer. Training uses a block-wise causal mask so each scale sees only earlier scales. Inference uses KV-caching and no mask.
4. Infinite-Vocabulary Classifier (IVC) §3.3 fig. 3. Instead of a V-way softmax, the head predicts each of the d bits as a binary decision. The paper motivates this by the cost of a V-way classifier (its example: V = 2^32 and width 2048 would need about 8.8 trillion weights) and by the noisy targets of index labels.
5. Bitwise self-correction (BSC) §3.4 alg. 2. During training, the bit labels of each scale are randomly flipped at a rate drawn between 0 and p. The flipped bits are re-quantized into the reconstruction, which then feeds the next scale. Targets remain the clean bits, so the model learns to correct flipped inputs.
6. Aspect ratios and positions §3.5 app. A. Each aspect ratio gets a predefined scale schedule. Scales keep roughly equal area across ratios so training sequences have similar lengths. Positions use RoPE2d on each scale, plus a learnable embedding per scale.
7. Decoding §4.7 tab. 9. Classifier-free guidance is applied to the bit logits with cfg = 4. The paper reports this as its best setting, with temperature 1.00. It reports that large cfg values are stable from the earliest scales once BSC is used.

Pseudocode for the self-correction encoder, following alg. 2:

# Encoding with bitwise self-correction  [alg. 2]  [§3.4]
# Input:  features F (B, d, H, W), flip rate p, scale sizes
# Output: bit targets per scale, transformer inputs per scale

def encode_with_self_correction(features, flip_rate, scale_sizes):
    # Running reconstruction from flipped bits, starts at zero  [alg. 2]
    recon = zeros_like(features)
    bit_targets = []
    transformer_inputs = []
    for k, size in enumerate(scale_sizes):
        # Residual relative to the flipped reconstruction  [alg. 2]
        residual = downsample(features - recon, size)
        # Clean bits R_k are the training targets  [§3.2] [eq. 4]
        bits = binary_quantize(residual)
        bit_targets.append(bits)
        # Flip each bit with a rate drawn uniformly from [0, p]  [§3.4]
        rate = uniform(0.0, flip_rate)
        flip_mask = random_like(bits) < rate
        flipped = where(flip_mask, 1 - bits, bits)
        # Re-quantize flipped bits, add to reconstruction  [alg. 2]
        recon = recon + upsample(bits_to_codes(flipped), features.size)
        # Next scale's input is the downsampled flipped reconstruction  [eq. 3]
        if k < len(scale_sizes) - 1:
            transformer_inputs.append(downsample(recon, scale_sizes[k + 1]))
    return bit_targets, transformer_inputs

Pseudocode for the bit-wise classifier loss, following §3.3:

# Infinite-Vocabulary Classifier: per-bit loss  [§3.3]
# Input:  hidden states (B, L, C); bit targets (B, L, d) with values in {0, 1}
# Output: scalar loss

def bit_loss(hidden, bit_targets):
    # One linear head outputs two logits for each of the d bits  [§3.3]
    logits = head(hidden)                   # (B, L, 2d)
    logits = logits.reshape(B, L, d, 2)     # (B, L, d, 2)
    # Two-way cross-entropy per bit  [code: infinity/models/infinity.py#L116]
    per_bit = cross_entropy(logits, bit_targets)   # (B, L, d)
    # Average over bits, the default  [code: infinity/utils/arg_util.py#L138]
    return mean(per_bit)

Pseudocode for one decoding step with classifier-free guidance on the logits, following §4.7 and infinity.py:

# One scale of sampling with CFG on logits  [§4.7]  [code: infinity.py#L569]
# Input:  cond_logits, uncond_logits (B, L, d, 2); cfg; tau
# Output: sampled bits (B, L, d)

def sample_scale(cond_logits, uncond_logits, cfg, tau):
    # Temperature divides the logits before guidance  [code: infinity.py#L571]
    cond_logits = cond_logits / tau
    uncond_logits = uncond_logits / tau
    # Guidance combines conditional and unconditional logits  [§4.7]
    logits = cfg * cond_logits + (1 - cfg) * uncond_logits
    # Sample each bit, two-way  [code: infinity.py#L578-L579]
    return sample_bits(logits)

Hyperparameters

Values marked "paper" come from the paper text or tables. Values marked "code" are released defaults or script settings, not stated in the paper. A value that the paper does not state is marked "not stated".

Results

Author-reported. Setup: 2B Infinity trained progressively from 256 to 1024 px, evaluated on GenEval and DPG with and without prompt rewriting, plus human preference, ImageReward, HPSv2.1 and latency §4.2 §4.3.

Paper vs code

Official code: the paper's author block links https://github.com/FoundationVision/Infinity, which is the source rule this card uses. Cloned at 9f9fcd1de7ead377ddf1106db1249a35f3a59143 (MIT). Code paths are relative to the repository root.

• Flip rate and layers. The paper says each scale is flipped at a rate drawn uniformly from [0, p] §3.4. The code draws one rate per scale for the whole batch, in steps of 0.01 up to p bitwise_self_correction.py. It applies this only to scales with index below noise_apply_layers, which the script sets to 13 train.sh. The paper's flip is applied to all scales. The number of scales in the 0.06M schedule used by the script is not stated in the paper and was not verified here.
• Head. The paper describes d binary classifiers that predict positive or negative §3.3. The code uses one linear head that outputs two logits per bit, trained with two-way cross-entropy infinity.py trainer.py. This is equivalent in form to a per-bit two-class softmax, not a single sigmoid output.
• Sampling temperature. The paper's best decoding uses tau = 1.00 tab. 9. The inference script uses tau = 0.5 infer.sh.
• Model width. The paper's 2.2B row has width 2080 and 20 heads tab. 6. The model type the inference script loads, infinity_2b, has width 2048 and 16 heads infinity.py infer.sh. Which one matches the released weights is not stated; the card reports both.
• Learning rate. The paper gives lr 6e-5 at batch 4096 §4.2. The code scales a base rate by batch/256 arg_util.py. The script's base rate is 6e-3 train.sh. For the paper's 6e-5 at batch 4096, the base rate would be 3.75e-6, which is 1600 times smaller than the script's value. The script does not set the GPU count, so the global batch is unknown.
• Training schedule. The paper trains in three stages with stated iteration counts §4.2. The script is one run, exp_name=debug, on data/infinity_toy_data with --ep=100 and --pn 0.06M train.sh [L56] [L70] [L90]. It is a debug configuration, not the paper's recipe.
• Scale embedding. The paper applies a learnable embedding to the features of each scale §3.5. The script sets add_lvl_embeding_only_first_block 1 train.sh, and the model adds the scale embedding only in the first block infinity.py.
• Flip path. multiscale_bsq.py has its own sign-flip function multiscale_bsq.py. It is off by default (random_flip = False) multiscale_bsq.py. Bitwise self-correction, as the paper describes it, is in bitwise_self_correction.py.
• Precision. The self-correction step runs with autocast disabled bitwise_self_correction.py, and so does the head infinity.py. Training uses bf16 (--fp16=2) train.sh. The paper does not discuss precision.
• Guidance per scale. The sampler takes a guidance value per scale infinity.py, so the pyramid schedule can be expressed. The script uses one value, 4, for all scales infer.sh.

Pitfalls

• The DPG claim does not match Table 1. The text says Infinity surpasses DALL-E 3 on DPG §4.3. Table 1 lists DALL-E 3 at 83.50 and Infinity at 83.46 tab. 1. It also lists SD3 (2B) at 84.08. Report the table values, not the text claim.
• GenEval is only reported with prompt rewriting. Every Infinity GenEval entry in tab. 1 carries the rewriter marker (†). No un-rewritten GenEval number is given for Infinity in the paper.
• The 66% win rate has no criterion. The abstract and intro cite a 66% win rate §1. The appendix gives 76.0%, 79.0%, and 66.0% against PixArt-Sigma, SDXL and SD3-Medium, without naming the criterion app. B. Treat the 66% figure as unspecified.
• Latency is not tied to hardware. The 0.8 s figure tab. 2 has no GPU, precision, batch size or software stated. The claim of 7x faster than SD3.5 at 8B is in the text §4.3, with no table row or hardware.
• IVC parameter counts do not reconcile. Table 5 lists 0.65M for the IVC head and 124M for the conventional one tab. 5. The text says IVC saves 99.95% tab. 5 §3.3. My check: for width 2048, d = 16 and two logits per bit, the head has 2048 × 16 × 2 ≈ 0.066M weights, which matches the 99.95% claim, not the 0.65M in the table. Treat the table's parameter column as inconsistent.
• Entropy cost. The paper says the LFQ entropy penalty costs O(2^d) and that the BSQ approximation makes it O(d) §3.2. The weight used in the code (0.1) is not given in the paper.
• "No extra computational cost" for self-correction. The paper says self-correction adds no computation §3.4. The code re-runs the quantizer on every scale for every batch bitwise_self_correction.py. The cost is likely small next to the transformer, but the paper does not measure it.
• The 2^64 tokenizer is not the text-to-image tokenizer. The intro and tokenizer ablation use up to 2^64 §1 tab. 4. The text-to-image model uses 2^32 §4.5. Do not assume the generator uses the 2^64 tokenizer.
• Top-k and top-p are not stated. The paper does not state any sampling truncation. The code's defaults differ by entry point (see Hyperparameters).
• Missing from the paper. Dataset sizes (pre-training and high-quality), number of GPUs, optimizer, weight decay, warmup, LR schedule details, human-study volunteer count, and the hardware for latency are all not stated.
• Teacher-forcing ablation is a single setting. Table 7 compares only one baseline against self-correction at 512 px with 5M data tab. 7. It does not test scaling of the effect.

Lineage

• Builds on: VAR (arXiv:2404.02905) for next-scale prediction, with Infinity replacing its index-wise classifier and tokenizer §1 §3.1 §3.3. BSQ (arXiv:2406.07548) and LFQ / MagViT-2 (arXiv:2310.05737) for the binary quantizer §3.2. RoPE2d for position encoding §3.5. Flan-T5 for text §3.1.
• Compared against: SD3 (2B and 8B), SDXL, DALL-E 3, PixArt-Sigma, Playground v2.5 (text only), HART, Show-o, Emu3, LlamaGen tab. 1 tab. 2 §4.3.
• Superseded by: not known from content in this wiki.