Paper · 2023-01-02 · cs.CV, cs.AI, cs.LG

Muse: Text-To-Image Generation via Masked Generative Transformers

Huiwen Chang, Han Zhang, Jarred Barber et al.

Muse generates images by predicting randomly masked discrete VQGAN tokens with a transformer, conditioned on frozen T5-XXL text embeddings. A base model and a super-resolution model decode in parallel over a few steps, with classifier-free guidance. Author-reported: FID 6.06 on CC3M (base plus super-res), and zero-shot COCO FID 7.88 with CLIP 0.32 for the 3B model.

Text-to-image generation Masked image modeling Vector quantization Classifier-free guidancetext-to-imagemasked-generative-transformerparallel-decoding

arXiv PDF Markdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-10-24, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Muse makes pictures from text. An image is first turned into a short grid of codes, like a mosaic of labelled tiles taken from a fixed set of 8192 tile types. A text encoder that was trained on language reads the prompt. A transformer then learns to fill in hidden tiles, given the visible tiles and the prompt. Training hides a random share of tiles each time and asks the model to guess them.

To make a picture, the model starts with every tile hidden. It fills in the tiles it is most sure about, then fills more on each pass, until the grid is complete. A second model turns the small grid into a larger one, so the final picture is sharper. The same fill-in-the-blanks idea lets the model repaint a masked region of a photo or change a picture a little, with no extra training.

The authors argue this is cheaper than diffusion models, which need many denoising steps, and cheaper than models that write one tile at a time, which need one step per tile.

TL;DR

Discrete-token, masked-prediction text-to-image model: a base transformer predicts masked VQGAN tokens; a super-resolution transformer maps 16×16 tokens to 64×64 tokens §2.2 §2.4.
Frozen T5-XXL embeddings condition both stages §2.1.
Sampling fills tokens in parallel over a few steps, not one at a time §2.8.
Author-reported: best CC3M FID 6.06 (base plus super-res, trained on CC3M); zero-shot COCO FID 7.88 and CLIP 0.32 for the 3B model tab. 1 tab. 2.
No code or weights were released §5. Reimplementation must rely on the paper.

Problem

Pixel-space diffusion models (for example Imagen and DALL-E 2) need many sampling steps and work on large images. Autoregressive token models (for example Parti) need one forward pass per token, which is 256 steps for a 16×16 grid and 4096 steps for a 64×64 grid §2.8. The paper asks whether a masked-token transformer, conditioned on a frozen language model, can match their image quality and text alignment while being much faster §1.

Method

Overview. The pipeline has three parts: a frozen text encoder, a base masked transformer on 16×16 tokens (256×256 pixels), and a super-resolution masked transformer on 64×64 tokens (512×512 pixels). Each stage has its own VQGAN tokenizer §2 (the pipeline figure is in §2).

Text encoding. A frozen T5-XXL encoder gives a sequence of 4096-dimensional embeddings. A linear projection maps them to the hidden size of each transformer §2.1.
Image tokenization. A convolutional VQGAN quantizes images into codes from a codebook of size 8192 §3 tab. 6. The base stage uses a VQGAN with downsampling factor f=16 on 256×256 images, giving 16×16 tokens. The super-resolution stage uses a VQGAN with f=8 on 512×512 images, giving 64×64 tokens §2.2. The encoder and decoder have no non-local blocks, so they are fully convolutional app. A.2.
Base model. A transformer with self-attention over image tokens and cross-attention to the text embeddings §3. Image tokens are linearly mapped to the hidden size and get learned 2D positional embeddings. Some tokens are replaced by a special [MASK] token. An MLP output head gives logits over the 8192 codes, and cross-entropy is taken at the masked positions against the true codes §2.3. The 3B model has 48 layers, hidden size 2048 and MLP size 8192 tab. 4.
Super-resolution model. The 16×16 tokens go through 16 self-attention layers. Their output is concatenated with the projected text embeddings. Cross-attention from that concatenation to the masked 64×64 tokens predicts the high-resolution codes. The model has 32 multi-axis transformer layers §2.4 §3 tab. 5. The high-resolution VQGAN decoder then produces the image §3.
Decoder fine-tuning (optional in the description). A larger decoder (four residual blocks per resolution, base width 256) is trained on top of frozen encoder, codebook and transformers, to sharpen fine detail §2.5 app. A.2. The paper shows a qualitative example only app. A.2. Whether the reported numbers use it is not stated.
Variable masking rate (training). For each example, a masking rate r is drawn from a truncated arccosine density on [0, 1], p(r) = (2/π)(1−r²)^(−1/2) §2.6. Its mean is about 0.64, so most training steps mask most tokens §2.6. The paper says it masks "a varying fraction" of tokens at random §2.3; the per-token sampling rule is not stated.
Classifier-free guidance. During training, the text condition is removed for 10% of samples §2.7. At sampling time, a conditional logit ℓ_c and an unconditional logit ℓ_u are combined as in eq. 1 §2.7. The guidance scale t grows linearly during sampling, so early tokens are free and later tokens follow the prompt more closely §2.7. A negative prompt can replace the unconditional branch §2.7.
Pseudocode
```
# Guided logits  [eq. 1] [§2.7]
# Input:  cond_logits (N, 8192), uncond_logits (N, 8192), scale t
# Output: guided_logits (N, 8192)

def guide(cond_logits, uncond_logits, scale):
    # Move away from the unconditional logits by the guidance scale  [eq. 1]
    return (1 + scale) * cond_logits - scale * uncond_logits
```

Iterative parallel decoding. Sampling starts with all image tokens masked. At each step the model predicts all masked tokens in one forward pass. It then fixes the most confident ones, so that the number still masked follows a cosine schedule §2.8. The paper cites MaskGIT for the schedule. Using this, 256 tokens need 24 steps in the base model and 4096 tokens need 8 steps in the super-resolution model §2.8. The paper gives no pseudocode. The sketch below is our reading of §2.8 and §2.7.

Pseudocode

# Iterative parallel decoding for one token grid  [§2.8] [§2.7]
# Input:  text_emb (T, 4096), neg_emb (T, 4096), num_tokens N, steps S
# Output: codes (N,) codebook indices in 0..8191

def sample(text_emb, neg_emb, num_tokens, num_steps, max_scale):
    # Start with every image token masked  [§2.8]
    codes = full(num_tokens, MASK_ID)
    for step in range(num_steps):
        # Guidance ramps up linearly over the run  [§2.7]
        # Ramp endpoints are not stated; see Pitfalls.
        scale = ramp(step, num_steps, max_scale)
        # Conditional and negative-prompt predictions  [§2.3] [§2.7]
        cond_logits = base_model(codes, text_emb)
        uncond_logits = base_model(codes, neg_emb)
        logits = guide(cond_logits, uncond_logits, scale)  # (N, 8192)
        # Draw a token and a confidence for each masked position  [§2.8]
        # Temperature is swept but its value is not stated  [§3.2]
        tokens, confidence = sample_with_confidence(
            logits, codes == MASK_ID
        )
        # Target number of tokens still masked after this step  [§2.8]
        # The cosine formula is not given in the paper; see Pitfalls.
        target_masked = cosine_masked_count(step + 1, num_steps, num_tokens)
        # Commit the most confident tokens until the target is reached  [§2.8]
        codes = commit_most_confident(
            codes, tokens, confidence, target_masked
        )
    return codes

Mask-free editing and inpainting. Inpainting masks the tokens of a region and resamples them under a text prompt §3.3.1. For high-resolution inpainting, a 512×512 input is downsampled to 256×256. Both token grids are masked in the same region, the low-resolution tokens are inpainted first, and the high-resolution tokens are then inpainted conditioned on them §3.3.1. Mask-free editing encodes a real image, then repeatedly masks and resamples a random 8% of tokens for 100 iterations. The run uses guidance scale 4 and top-k sampling with k=3 on the logits §3.3.2. It runs on the base model and then super-resolution is applied to the edited output §3.3.2.

Training recipe. Models are trained on the Imagen dataset of 460M text-image pairs, for 1M steps at batch size 512 on 512-core TPU-v4 chips, about one week of training §3. The optimizer is Adafactor, which lets the 3B model fit without model parallelism. No exponential moving average is kept during training; instead, checkpoints are saved every 5000 steps and averaged offline with decay 0.7 §3.

Hyperparameters

Name	Value	Where	Notes
Base model layers / hidden / MLP	48 / 2048 / 8192	tab. 4	3B model; hidden and MLP sizes are for this model
Super-res layers / hidden / MLP	16 low-res encoder + 32 / 1024 / 4096	tab. 5 §3	multi-axis transformer layers §3
Optimizer	Adafactor, β1=0.9, β2=0.96	tab. 4 tab. 5 §3
Base learning rate	1e-4	tab. 4 tab. 5	cosine decay, 5000 warmup steps (both models)
Weight decay	0.045	tab. 4 tab. 5
Batch size	512	tab. 4 tab. 5 §3
Training steps	1.5M (base table) or 1M (text)	tab. 4 §3	conflicting, see Pitfalls
Super-res training steps	1M	tab. 5
Text encoder	frozen T5-XXL, 4096-dim output	§2.1 §3	encoder has 4.6B parameters §3
Codebook size	8192	§3 tab. 6	per-tokenizer size not stated separately
Base tokenizer	VQGAN, f=16, 256×256 → 16×16	§2.2
Super-res tokenizer	VQGAN, f=8, 512×512 → 64×64	§2.2
VQGAN training	Adam, lr 1e-4 (G and D), wd 1e-4, β=(0.9, 0.99), batch 256, 1M steps, 10000 warmup	tab. 6
VQGAN losses	perceptual weight 0.05, adversarial weight 0.1	tab. 6
VQGAN architecture	2 residual blocks per resolution, base width 128, no non-local block	app. A.2
Decoder fine-tune architecture	4 residual blocks per resolution, base width 256	app. A.2	optimizer and lr for this stage: not stated
Mask rate distribution	p(r) = (2/π)(1−r²)^(−1/2) on [0,1]; mean ≈ 0.64	§2.6
Text dropout for CFG	10% of training samples	§2.7	representation of dropped text: not stated
Guidance scale	ramped linearly during sampling	§2.7	start and end values: not stated
Decoding steps	24 (256 base tokens), 8 (4096 super-res tokens)	§2.8
Editing: resample ratio	8% of tokens per iteration	§3.3.2
Editing: iterations	100	§3.3.2
Editing: guidance scale	4	§3.3.2
Editing: top-k	k=3	§3.3.2
EMA	not used during training; offline average, decay 0.7, checkpoints every 5000 steps	§3
Human eval prompts	PartiPrompts, 1650 prompts; 16 samples per prompt, best CLIP kept	§3.2.1
Human eval consensus	at least 3 of 5 raters	§3.2.1
Sampling temperature	swept; value used for headline results not stated	§3.2
Trained-on data	CC3M (CC3M results); Imagen dataset, 460M pairs (3B / COCO)	§3 tab. 1

Note on the VQGAN row for the perceptual loss weight: the appendix table gives 0.05 tab. 6. The duplicate fine-tuning table in the source uses 0.005; it is not included in the main text (see Pitfalls).

Results

Author-reported. Each row is the paper's own number, with the setup from the source table. We did not rerun any model.

CC3M (both Muse models trained on CC3M; FID is lower-is-better) tab. 1

Approach	Model type	Params	FID	CLIP
VQGAN tab. 1	Autoregressive	600M	28.86	0.20
ImageBART tab. 1	Diffusion + autoregressive	2.8B	22.61	0.23
LDM-4 tab. 1	Diffusion	645M	17.01	0.24
RQ-Transformer tab. 1	Autoregressive	654M	12.33	0.26
Draft-and-revise tab. 1	Non-autoregressive	654M	9.65	0.26
Muse base tab. 1	Non-autoregressive	632M	6.8	0.25
Muse base + super-res tab. 1	Non-autoregressive	632M + 268M	6.06	0.26

MS-COCO, zero-shot, 256×256 tab. 2. The FID column for this table is the zero-shot FID-30K column, except the first group, which lists FID-30K directly, as in the source.

Approach	Model type	Metric	Value
Muse-3B tab. 2	Non-autoregressive	Zero-shot FID	7.88
Parti-20B tab. 2	Autoregressive	Zero-shot FID / FID-30K	7.23 / 3.22
Parti-3B tab. 2	Autoregressive	Zero-shot FID	8.10
Imagen-3.4B tab. 2	Diffusion	Zero-shot FID	7.27
DALL-E 2 tab. 2	Diffusion	Zero-shot FID	10.39
GLIDE tab. 2	Diffusion	Zero-shot FID	12.24
LDM tab. 2	Diffusion	Zero-shot FID	12.63
Make-A-Scene tab. 2	Autoregressive	FID-30K	7.55

CLIP score for Muse-3B on COCO: 0.32. This value appears in the text and the caption of tab. 2, not in the table cells. The text compares it to 0.27 for Imagen; the Imagen value is read off its figure §3.2.

PartiPrompts human evaluation (Muse-3B vs Stable Diffusion v1.4) §3.2.1

Outcome	Share of prompts
Muse preferred (3+ rater consensus)	70.6%
Stable Diffusion preferred	25.4%
No consensus	4%

The paper reads this as roughly 2.7× better caption alignment §3.2.1. It does not report a realism rating; the authors say that question is not meaningful between models with very different alignment §3.2.1.

Inference time per batch, as reported tab. 3

Model	Resolution	Time	Hardware
Muse-3B	256×256	0.5 s	TPU-v4 (internal)
Muse-3B	512×512	1.3 s	TPU-v4 (internal)
Parti-3B	256×256	6.4 s	TPU-v4 (internal)
Imagen	256×256	9.1 s	TPU-v4 (internal)
Imagen	1024×1024	13.3 s	TPU-v4 (internal)
LDM, 50 steps	512×512	3.7 s	A100 (external benchmark)
LDM, 250 steps	512×512	18.5 s	A100 (scaled from 50 steps)

The batch size for these times is not stated tab. 3.

Qualitative results. The paper shows success cases for cardinality, composition, style and text rendering §3.1 and failure cases for long phrases, high cardinality and multiple cardinalities §3.1. It also shows editing examples in §3.3 (inpainting and mask-free editing).

Paper vs code

No official code. The authors say they chose not to release code or a public demo for now, citing misuse risks §5. Search steps: (1) the paper and abstract link only the project page muse-model.github.io; (2) that page was fetched and lists no repository, weights or demo; (3) a GitHub repository search for "muse" and "muse text-to-image masked transformer" found only unofficial reimplementations (lucidrains/muse-maskgit-pytorch, Sygil-Dev/muse-maskgit-pytorch, viv92/muse), none owned by the authors. Those are not used here.

Pitfalls

Sampler details are missing. The paper does not give the cosine formula for the number of masked tokens at each step, the confidence measure, the temperature used for headline results, or the guidance ramp's start and end values. It points to MaskGIT for the schedule §2.8. Implementers have to pick these, and the paper's Pareto sweeps (guidance, temperature, linear ramp on or off) show that the choice changes FID and CLIP §3.2.
Training steps conflict. Section 3 says 1M steps. The base-model table says 1.5M tab. 4. The super-resolution table says 1M tab. 5. The text does not say which applies to the headline 3B model.
Dataset conflict. The text says both Muse models for CC3M were trained on CC3M §3.2. The 3B model and the COCO results use the Imagen dataset of 460M pairs §3. Which data trained the 900M model used for editing is not stated.
VQGAN size counts disagree in form. The results section describes a CNN with 19 ResNet blocks §3. The appendix describes 2 residual blocks per resolution at width 128 app. A.2. Both may be true (different counts), but the paper does not reconcile them. Use the appendix for the tokenizer architecture.
Decoder fine-tuning is shown only qualitatively. The section describes it as a method and gives an architecture §2.5 app. A.2, but no metric reports it. The paper does not say whether it is used in the reported results. The source contains a second fine-tuning table (perceptual weight 0.005) that the main text does not include. Do not mix its values into the VQGAN settings.
Speed comparison crosses hardware. Muse, Imagen and Parti were timed internally on TPU-v4 tab. 3. The LDM numbers come from an external A100 benchmark. The "3× faster than Stable Diffusion" claim §1 therefore compares different hardware and an unstated batch size. The LDM 250-step time is 50-step time times five tab. 3.
Selection inflates the human and CLIP numbers. For each prompt, 16 images are generated and the one with the highest CLIP score is kept §3.2.1. The same selection rule is applied to Stable Diffusion. Reproducing the 70.6% result requires the same selection.
Costs the abstract hides. The model uses a frozen 4.6B-parameter T5-XXL encoder §3. The 3B model needed Adafactor to fit without model parallelism, and training skipped EMA to save memory §3. Editing runs 100 iterations of resampling §3.3.2, so editing is not one-step.
Known failures. Long rendered phrases are often wrong (duplicated or cut words). Accuracy in cardinality drops as the count rises, and multi-count prompts often get one count wrong §3.1. These are the paper's own limitations, not fixed in the paper.
Mask sampling rule. The paper says tokens are masked "randomly" at a rate drawn per example §2.3 §2.6. It does not say whether each token is masked independently. Use independent Bernoulli masking at rate r for an implementation, and note it as an assumption.
Dropped-text representation. The 10% text dropout does not say what replaces the text embedding (for example a null prompt or zero vector) §2.7.
Super-resolution resource use. The super-resolution model adds 268M parameters on top of 632M for the base model tab. 1 §1. Its inputs are the base model's sampled tokens, so errors in base sampling carry through.

Lineage

Builds on: MaskGIT, for masked-token prediction with variable masking rate, cosine-scheduled parallel decoding and the iterative sampler §2.3 §2.6 §2.8. VQGAN, for the image tokenizer §2.2. Imagen and T5-XXL, for frozen language-model text conditioning §2.1. Classifier-free guidance and negative prompting §2.7. Adafactor for memory-efficient training §3.
Compared against: Imagen, DALL-E 2, Parti, GLIDE, LDM and Stable Diffusion v1.4, DALL-E, Make-A-Scene and GAN text-to-image models tab. 1 tab. 2 §3.2.
Superseded by: not stated in the content of this card.

Source and license

Adapted from “Muse: Text-To-Image Generation via Masked Generative Transformers” by Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, Dilip Krishnan (arXiv:2301.00704), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.