Text-to-image generation

Overview

A text-to-image model takes a caption and returns an image that matches it. Captions for one picture vary in wording, so a model that treats one caption as the only truth can paint a different-looking object for each paraphrase 1904.01480 §1.

The cards on this page reach the same goal with different machinery: a stacked GAN 1904.01480 §3.1, discrete image tokens predicted by a transformer 2102.12092 §2.2, masked token prediction in parallel 2301.00704 §2.3, a diffusion decoder steered by CLIP image embeddings 2204.06125 §2, latent diffusion 2307.01952 §2, a frozen-model adapter for spatial control 2302.08453 §3.3, a one-step student of a diffusion teacher 2311.18828 §3.1, and one autoregressive model that writes both text and image embeddings 2312.13286 §2.1. Four cards change an existing diffusion model rather than offer a new generator: a safety term added to guidance 2211.05105 §3, a reward model whose score is back-propagated into Stable Diffusion 2304.05977 §3, a key fixed into the decoder so that every output carries a watermark 2303.15435 §4.2, and a training-free term that moves attention into user-given boxes during sampling 2304.03373 §3.2. Five more cards support the field without being generators: a web-scale image-text dataset [arxiv:2210.08402 §3.1], two learned scorers of human preferences 2305.01569 §3 2303.14420 §5, a benchmark that checks each prompt's objects, counts, colours and positions one by one 2310.11513 §3.2, and a reference-free faithfulness metric that asks questions about the image 2303.11897 §3.

For diffusion families, see Diffusion models.

Variants

Stacked GAN with a Siamese contrastive term (SD-GAN)

Three generator–discriminator stages, from a 64 × 64 initial image to higher resolutions. Each stage has its own discriminator, and the discriminators share no parameters 1904.01480 §3.1. Training uses two captions per step, one per Siamese branch. A contrastive loss on the discriminator features of the two branches, applied at every stage, pulls together pairs that describe the same image and pushes apart pairs that do not [arxiv:1904.01480 §3.1] eq. 2. Same-image pairs receive no pull once they are closer than a floor α = 0.1, and different-image pairs are pushed until a margin ε = 1.0 1904.01480 §3.1 eq. 2.

The generator's batch normalization layers are replaced by semantic-conditioned batch norm. Its scale and shift come from the sentence vector (sentence cue) or from word-level attention (word cue) 1904.01480 §3.2 eq. 5 eq. 6.

Author-reported results: Inception score 4.67 ± .09 on CUB against 4.36 ± .03 for AttnGAN, and 35.69 ± .50 on MS-COCO against 25.89 ± .47 [arxiv:1904.01480 tab. 1]. In the authors' ablation, removing SCBN (Siamese term kept) gives 4.51 ± .07 on CUB and 30.18 ± .47 on COCO; removing the Siamese term (SCBN kept) gives 4.49 ± .06 and 29.79 ± .61; removing both gives 4.11 ± .04 and 23.76 ± .40 1904.01480 tab. 4. Card: Semantics Disentangling for Text-to-Image Generation.

Discrete image tokens with an autoregressive transformer (DALL-E)

A discrete autoencoder maps each 256 × 256 image to a 32 × 32 grid of indices into a codebook of 8192 entries 2102.12092 §2.1. A 12-billion-parameter decoder-only transformer reads the caption tokens followed by those image tokens and predicts the next token 2102.12092 §2.2. Candidates are sampled at temperature 1, with no temperature reduction for the main results. The authors draw 512 candidates, decode each one, score it with a pretrained contrastive model, and keep the top k; k is not stated for the main results 2102.12092 §2.6.

Author-reported zero-shot results on MS-COCO: raters preferred the images over DF-GAN's by majority vote 90% of the time for realism and 93% for caption match 2102.12092 §3.1 fig. 7. FID is within about two points of the best prior method on MS-COCO, and about 40 points worse on CUB [arxiv:2102.12092 fig. 9]. Reranking gains flatten after 32 candidates 2102.12092 fig. 9c.

The transformer is not released, and the official repository holds only the autoencoder. A reproduction must write the transformer from the paper's text and figures: see the card's pitfalls. Card: Zero-Shot Text-to-Image Generation.

Masked tokens filled in parallel (Muse)

A VQGAN tokenizer turns each image into a grid of codes from a codebook of 8192 entries. The base model works on a 16 × 16 grid and the super-resolution model on 64 × 64 2301.00704 §2.2. A transformer with cross-attention to frozen T5-XXL text embeddings is trained to predict hidden codes 2301.00704 §2.1 §2.3. The share of hidden codes is drawn per example from a fixed density with mean about 0.64, so most training steps hide most of the grid 2301.00704 §2.6.

At sampling time every code starts hidden. The model fills in its most confident codes over a fixed number of passes: 24 for the base grid and 8 for the super-resolution grid 2301.00704 §2.8. A token-by-token autoregressive model needs one pass per code, which is 256 passes for the base grid 2301.00704 §2.8.

Author-reported: CC3M FID 6.06 for the base model with super-resolution, against 17.01 for LDM-4 on the same dataset 2301.00704 tab. 1. Zero-shot on MS-COCO at 256 × 256, the 3B model reaches FID 7.88 [arxiv:2301.00704 tab. 2], against 7.27 for Imagen and 10.39 for DALL-E 2. On PartiPrompts, raters preferred the 3B model on 70.6% of prompts, against 25.4% for Stable Diffusion v1.4 2301.00704 §3.2.1. Each model's output was the best of 16 by CLIP score, so the same selection rule is needed to reproduce this figure. Inference takes 0.5 s per batch at 256 × 256 on the authors' TPUs, against 3.7 s for LDM with 50 steps on an external A100 benchmark. The batch size is not stated, and the hardware differs 2301.00704 tab. 3.

No code or weights were released 2301.00704 §5. The cosine schedule, the sampling temperature and the guidance ramp are not stated, so a reimplementation must choose them 2301.00704 §2.8 §3.2. Card: Muse: Text-To-Image Generation via Masked Generative Transformers.

Caption to CLIP image embedding, then a diffusion decoder (unCLIP)

A prior maps a caption to a CLIP image embedding, and a diffusion decoder inverts the CLIP image encoder to render that embedding 2204.06125 §2. The authors try two priors: an autoregressive one over quantised principal components of the embedding, and a diffusion one that predicts the clean embedding directly 2204.06125 §2.2. At similar size, the authors find the diffusion prior gives better results 2204.06125 §5.1 §5.2. The decoder is a 3.5-billion-parameter model built on GLIDE's architecture 2204.06125 tab. 3.

Zero-shot FID on MS-COCO at 256 × 256 is 10.39 with the diffusion prior and 10.63 with the autoregressive prior 2204.06125 tab. 2. In human comparisons against GLIDE, the diffusion-prior model won on diversity 70.5% of the time, while GLIDE was preferred on photorealism (48.9% for unCLIP) and on caption similarity (45.3%) 2204.06125 tab. 1. The same CLIP space supports variations, interpolation between two images, and moving an image toward a text direction 2204.06125 §3.

The image encoder was trained by the authors on about 650 million images. The paper does not say that its weights are released, and it does not evaluate public CLIP weights in its place 2204.06125 app. C. Card: Hierarchical Text-Conditional Image Generation with CLIP Latents.

Latent diffusion with size and crop conditioning (SDXL)

A latent diffusion model whose UNet has about 2.6 billion parameters 2307.01952 §2.1. It reads the prompt with two text encoders, which together have 817 million parameters 2307.01952 §2.1.

Rather than discarding small or oddly shaped training images, the model is given each image's original size and crop offset as extra inputs. At inference the crop offset is set to zero 2307.01952 §2.2. Training then uses buckets of several aspect ratios near 1024² pixels 2307.01952 §2.3. An optional refiner, trained in the same latent space, re-denoises the base output, starting from noise level 200 of 1000 2307.01952 §2.5.

Author-reported user study on four-way picks: refiner output won 48.44%, the base model 36.93%, SD 1.5 7.91% and SD 2.1 6.71% 2307.01952 §2.5. The model weights carry a licence separate from the paper and the code; check it before any commercial use (see the card's pitfalls). Card: SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.

Training-free layout steering of Stable Diffusion (backward guidance)

The Stable Diffusion weights stay fixed. During the first denoising steps the latent itself is updated by gradients of an energy. For each prompt word, the energy measures how little of that word's cross-attention mass falls inside its user-given box 2304.03373 §3.2 eq. 2 eq. 3. The main setting uses guidance on the first 10 steps, up to 5 updates per step, and a strength of 30 2304.03373 §4.1. The paper's alternative, forward guidance, edits the attention maps directly; the authors find the latent update works better 2304.03373 tab. 1.

Author-reported, on a 1,000-prompt sample of the VISOR protocol: conditional accuracy rises from 59.8% for Stable Diffusion to 96.9%, and the full protocol gives 95.95% against 62.98% 2304.03373 tab. 1 tab. 2. The table caption for the full protocol gives no prompt count; the appendix describes 25,280 prompts 2304.03373 app. B. On COCO layouts scored with a detector, mAP rises from 19.2 to 35.7, while FID moves from 72.3 to 73.3 [arxiv:2304.03373 tab. 3]. Sampling takes about twice as long, roughly 8 s against 4 s per image in the authors' table, and the hardware is not stated 2304.03373 tab. 1 §4. A large strength hurts fidelity: FID is 88.4 at 100 and 129.7 at 500 2304.03373 app. C. For models trained on layouts, see Layout-to-image generation. Card: Training-Free Layout Control with Cross-Attention Guidance.

Sampling-time safety steering of Stable Diffusion (SLD)

Safe latent diffusion keeps Stable Diffusion frozen and changes only the guidance step. Each step makes three noise predictions: unconditional, one conditioned on the prompt, and one conditioned on a text description of unwanted content 2211.05105 §3 eq. 3. The prompt prediction is pushed away from the unwanted-content prediction, element by element. The push is applied only where the prompt prediction does not already exceed the unwanted-content prediction by a threshold 2211.05105 §3 eq. 4 eq. 5. The safety term starts only after a warm-up of some steps, and a momentum term smooths it over steps 2211.05105 §3 alg. 1. The default concept is a list of about twenty words, from violence and nudity to drug use 2211.05105 §4.

Author-reported on the I2P benchmark, the probability of an inappropriate image falls from 0.39 for Stable Diffusion v1.4 to 0.09 at the strongest preset, and to 0.18 with negative prompting 2211.05105 tab. 2. The cost shows in image quality: COCO FID rises from 14.43 to 18.28 at the strong preset 2211.05105 tab. 3. In a DrawBench user study, 62% to 64% of users rated quality as better or equal to the baseline 2211.05105 tab. 3.

The authors note that the same mechanism could be used for censorship, and ask that any deployment state which content is suppressed 2211.05105 §7. The released code differs from the paper's equations in a few places; see the card before reproducing numbers. Card: Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models.

Watermark fixed into the decoder (Stable Signature)

The method fine-tunes only the decoder of a latent diffusion model. The copy is trained for about 100 steps on fewer than 500 images, which the paper reports as about one minute on one GPU 2303.15435 §4.2. The result carries one fixed 48-bit key in every output. A separate extractor reads the key, and it is pre-trained with simulated edits and then whitened 2303.15435 §4.1 app. A.1. Generation itself is unchanged 2303.15435 §4.2.

Detection counts matching bits against a threshold. Identification compares the image with each user's key and raises the threshold as the number of keys grows 2303.15435 §3 §5.2.

Author-reported: 0.99 bit accuracy on unmodified text-to-image output, 0.92 after a combined crop, brightness and JPEG edit, and an FID change of −0.3 2303.15435 tab. 1. With 10^5 users, identification is 98% on unmodified images and 40% after the combined edit 2303.15435 §5.2. The headline text-to-image numbers come from an internal 2.2B-parameter model, not from the public code 2303.15435 §5. The released code is CC BY-NC 4.0, so it is not cleared for commercial reuse LICENSE. Card: The Stable Signature: Rooting Watermarks in Latent Diffusion Models.

One-step student of a diffusion teacher (DMD)

The student is a one-step generator, initialised from the teacher 2311.18828 §3.1. Its update combines the difference of two score estimates, one from the frozen teacher and one from a fake-score network trained alongside on the generator's own outputs 2311.18828 §3.2. A regression term to teacher outputs on paired noise, weighted 0.25 by default, is added to keep the sample modes covered 2311.18828 §3.3.

Class-conditional ImageNet 64 × 64: FID 2.62 in one forward pass, against 2.32 for the EDM teacher with 512 2311.18828 tab. 1. Text-to-image, distilled from Stable Diffusion v1.5 at guidance 3: 0.09 s against 2.59 s per image, with FID 11.49 against 8.78 for the teacher 2311.18828 tab. 3. At guidance 8 the teacher is better on FID and CLIP score (13.45 against 14.93; 0.322 against 0.320) 2311.18828 tab. 4. The authors report about 30 times the speed of the teacher 2311.18828 §4.3; the hardware for the latency figures is not stated.

Cost: the LAION-based text-to-image run took about 36 hours on 72 A100 GPUs 2311.18828 §4.3. The method trains two networks at once, so memory use is high 2311.18828 §5. No official code exists; see the card. Card: One-step Diffusion with Distribution Matching Distillation.

Multimodal language model that emits image embeddings (Emu2-Gen)

Emu2 is one autoregressive model over interleaved text and images. Each image is written as 64 embeddings, and the model predicts either the next text token or the next image embedding 2312.13286 §2.1 §2.2.2. A visual decoder built on SDXL-base turns those 64 embeddings into a 1024 × 1024 image 2312.13286 §2.2.3. Generation is a separate instruction-tuned variant, Emu2-Gen, trained on text phrases, bounding boxes and object images for controllable output 2312.13286 §2.3.2. Emu2-Chat, for dialogue and grounding, is a different variant 2312.13286 §2.3.

Author-reported zero-shot results on MS-COCO, with CLIP-I comparing against reference images and CLIP-T comparing against the prompt: Emu2-Gen scores 0.686 and 0.297, SDXL 0.674 and 0.310, and SDv1.5 0.667 and 0.302 [arxiv:2312.13286 tab. 4]. On the authors' CLIP measures, Emu2-Gen is ahead on image similarity and behind SDXL on prompt similarity. The sampler uses 50 steps and a guidance scale of 3.0 2312.13286 app. E.3.

The code is Apache-2.0, but its README lists the model weights as non-commercial. Treat the weights as not cleared for commercial use; see the card's pitfalls. The released code also runs the full language model once per image slot, with no key/value cache, so generation is slow. Card: Generative Multimodal Models are In-Context Learners.

Data, scoring and evaluation

These cards mostly do not generate images on their own. They supply training data, a way to judge outputs, or a benchmark. ImageReward's ReFL is the exception: it tunes a generator with a learned score.

Web-scale image-text data (LAION-5B)

LAION-5B is a dataset release, not a model. It keeps about 5.85 billion image-text pairs drawn from web alt-text. A CLIP ViT-B/32 similarity filter keeps English pairs at 0.28 or above and other pairs at 0.26 or above 2210.08402 §3.1. Potentially offensive pairs are tagged rather than removed, and about 3% of images are flagged NSFW 2210.08402 §3.2 §4. Many captions are spam or keyword lists 2210.08402 §6.

Stable Diffusion checkpoints and GLIDE fine-tunes were trained on LAION subsets 2210.08402 app. F. The paper's CC BY 4.0 licence covers the metadata; the authors say they do not own the images or the text, so check the rights for each source before reuse 2210.08402 app. A. Card: LAION-5B: An open large-scale dataset for training next generation image-text models.

Learned preference scorer (Pick-a-Pic, PickScore)

PickScore is a CLIP ViT-H/14 model fine-tuned on user choices between two images for one prompt, with a two-way softmax loss and tie targets 2305.01569 §3. It predicts held-out picks with 70.5% accuracy, against 68.0% for expert annotators and 60.8% for CLIP-H 2305.01569 tab. 1.

The authors report a Pearson correlation of 0.790 with real-user Elo ranks across 45 models, against 0.670 for HPS, 0.492 for ImageReward and 0.313 for CLIP-H 2305.01569 fig. 7. When 100 candidates per prompt are ranked by the scorer, human raters preferred PickScore's pick over CLIP-H's pick 71.3% of the time 2305.01569 tab. 2.

The human baseline is a group of the authors' friends and colleagues, and the number of annotators is not stated 2305.01569 §4. Licences for the dataset and model are not stated in the paper or repository. Card: Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.

Human-choice classifier on CLIP (HPS)

HPS fine-tunes CLIP ViT-L/14 so that a prompt's text embedding sits close to the embedding of the image people picked, out of several images made from the same prompt. The score is 100 times the cosine similarity between the two embeddings 2303.14420 §5. The paper gives the loss in words only, so the exact form is not stated 2303.14420 §5.

The training picks come from a public Discord channel of Stable Diffusion images: 25,205 prompts and 2,659 users 2303.14420 §3. The authors say the data are distributed under a non-commercial licence [arxiv:2303.14420 app. datasheet], and the dataset reflects a small, active community 2303.14420 §8.

Author-reported: HPS picks the human choice 43.5% of the time on 5,000 user choices, against 42.0% for human participants, 32.9% for CLIP ViT-L/14 and 26.1% for random 2303.14420 tab. 2. On 398 Stable Diffusion against DALL·E pairs, HPS agrees with human picks 61.5% of the time, against 56.8% for CLIP and 63.5% between humans 2303.14420 tab. 3. These choices are not the Pick-a-Pic test set, so the numbers do not compare with the PickScore row above.

The same score selects training images for a LoRA adapter on Stable Diffusion v1.4. The paper uses rank 32 and 10,000 steps 2303.14420 §6. Its user study shows 74% of the adapted model's images with more than ten positive votes, against 22% for the original, from 20 participants on 100 prompts; no variance is reported 2303.14420 §7.2 fig. 8. The paper's released code differs from it in LoRA rank and score scale, so see the HPS card before reproducing these results. Card: Human Preference Score: Better Aligning Text-to-Image Models with Human Preference.

Learned preference scorer and reward tuning (ImageReward)

ImageReward is a BLIP-based scorer trained on 136,892 pairwise comparisons, made by expert annotators on prompts from DiffusionDB 2304.05977 §2.1 §2.2. Training uses only the order of images for each prompt, and the score is a single number for a prompt and image pair 2304.05977 §2.2 eq. 1. The same score can also tune Stable Diffusion v1.4. ReFL runs most denoising steps without gradients, keeps gradients for one randomly chosen late step, and back-propagates the score of the resulting image 2304.05977 §3.

On its own held-out comparisons, ImageReward reaches 65.14% preference accuracy. The same table gives 62.78% for PickScore, 60.79% for HPS, 57.76% for BLIP score and 54.82% for CLIP score 2304.05977 tab. 3 tab. 5. These numbers use the authors' test set, not Pick-a-Pic's, so they do not compare directly with the PickScore row above. On six models and 100 prompts, the authors' ranking matches ImageReward with Spearman 1.00, against 0.60 for CLIP and 0.09 for FID 2304.05977 tab. 1. In human comparisons on real user prompts, ReFL beats Stable Diffusion v1.4 in 58.79% of comparisons 2304.05977 tab. 4. The number of raters for these studies is not stated.

The released training script differs from the paper in its loss, its regularising term and its training length. See the card's pitfalls before reproducing ReFL. Card: ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.

Object-level benchmark (GenEval)

GenEval turns each prompt into a checklist of object classes, counts, colours and relative positions 2310.11513 §3.1. An object detector trained on the 80 COCO classes finds the objects, a CLIP ViT-L/14 classifier names their colours, and the relative positions are read from box centres. Each image gets a yes or no, and the six task scores are averaged without weights 2310.11513 §3.2.

Author-reported: IF-XL scores 0.61 overall, followed by SD-XL at 0.55; the best position score is 0.15 (SD-XL) and the best attribute-binding score is 0.35 (IF-XL) 2310.11513 tab. 2. Overall scores move by 0.01 to 0.02 across random seeds, so smaller gaps are not meaningful [arxiv:2310.11513 tab. 2].

Against annotators, GenEval agrees on 83% of images, against 88% between annotators and 80% for thresholded CLIPScore 2310.11513 §4. Two of its thresholds (the counting threshold and the position dead-zone factor) were chosen on the same annotations; K-fold checks give 0.823 and 0.822 [arxiv:2310.11513 app. A.2]. The detector cannot verify fine details such as finger counts, and it generalises poorly to artistic renderings 2310.11513 §6. The released scorer differs from the paper's text in a few places, such as a per-class object cap and the position rule; see the card before reproducing numbers. Card: GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.

Question-based faithfulness score (TIFA)

TIFA does not generate images and needs no reference image. A language model writes multiple-choice and yes/no questions, each about one element of the prompt, with known answers 2303.11897 §3.1. A second model, UnifiedQA, keeps only the questions whose answers it reproduces 2303.11897 §3.2. A visual question answering (VQA) model then answers each question on the generated image, and the score is the share answered correctly [arxiv:2303.11897 §3] eq. 1.

Author-reported Spearman correlation with human faithfulness ratings on 800 images: 59.7 for TIFA with mPLUG, against 33.2 for CLIPScore and 18.3 to 34.0 for the caption-based metrics 2303.11897 tab. 2. Of the five VQA models the authors tested, mPLUG had the highest VQA accuracy against human answers and the highest correlation 2303.11897 tab. 3. Models do poorly on shape, counting and spatial relations 2303.11897 §5.3.

The authors release TIFA v1.0: 4,081 text inputs and 25,829 questions, with the questions generated once and then filtered 2303.11897 tab. 1 §4. Users then need only a VQA model to score a new set of images. Questions written by a different language model change the wording and counts, so scores are not directly comparable 2303.11897 §4.3. Card: TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering.

When to use which

Implementation sketch

The first two blocks are SD-GAN's and cover its losses. The third is the shape shared by the diffusion and autoregressive cards. Pseudocode in our own words, following the papers' equations.

# Pair loss for one discriminator stage  [arxiv:1904.01480 §3.1] [eq. 2]
# Input:  d (B,): feature distance between the two branches
#         same (B,): 1 if both captions describe one image, else 0
# Output: scalar

def pair_loss(d, same, alpha=0.1, eps=1.0):
    # Same-image pairs: pull together, no gradient below floor alpha  [eq. 2]
    pull = same * max(d, alpha) ** 2
    # Different-image pairs: push apart until the margin eps  [eq. 2]
    push = (1 - same) * max(eps - d, 0) ** 2
    # The 1/(2N) scaling is not settled by the paper: the sum bound is
    # ambiguous (see the card's pitfalls)  [arxiv:1904.01480 §3.1]
    return mean(pull + push) / 2

# Semantic-conditioned batch norm, sentence cue  [arxiv:1904.01480 §3.2] [eq. 4-5]
# Input:  x (B, C, H, W) feature maps; s (B, D) sentence vector
# Output: modulated features (B, C, H, W)

def scbn_sentence(x, s):
    # Plain batch norm per channel  [eq. 3]
    x_hat = normalize_per_channel(x)
    # Caption-derived scale and shift, broadcast over H and W  [eq. 5]
    gamma_c = mlp_gamma(s)
    beta_c = mlp_beta(s)
    # Learned scale and shift plus the caption-derived terms  [eq. 4]
    return (gamma + gamma_c) * x_hat + (beta + beta_c)

# Guided sampling with optional reranking
# Shape shared by the diffusion cards; see the notes below the block.
# Input:  caption; guidance strength w; number of candidates n
# Output: one image

def generate(caption, w, n, scorer=None):
    # Encode the caption; the null embedding serves the unconditional branch
    # [arxiv:2307.01952 app. Diffusion Models]
    cond = text_encoder(caption)
    uncond = null_embedding()
    candidates = []
    for _ in range(n):
        # Guidance: (1 + w) * cond - w * uncond; the step rule is not shown
        # [arxiv:2307.01952 app. Diffusion Models]
        latent = sample_guided(cond, uncond, w)
        candidates.append(decode(latent))
    if scorer is None:
        return candidates[0]
    # Keep the highest-scoring candidate, as in the reranking cards
    # [arxiv:2102.12092 §2.6] [arxiv:2305.01569 §6]
    return max(candidates, key=lambda img: scorer(caption, img))

Where the variants differ: DALL-E's sampler draws its candidates at temperature 1 and reranks them as a separate step; the paper's description has no guidance term 2102.12092 §2.6. T2I-Adapter adds its control features once per sample instead of inside the loop 2302.08453 §1. DMD takes one step, so the loop is a single pass 2311.18828 §3.1. SLD adds a third noise prediction and subtracts a scaled concept term inside the loop 2211.05105 §3 eq. 3. Stable Signature keeps the loop as it is and swaps in the fine-tuned decoder before decoding 2303.15435 §4.2. Muse replaces the noise loop with parallel token filling over a few passes 2301.00704 §2.8. Backward layout guidance takes gradient steps on the latent for the first 10 steps, each before the ordinary denoising step 2304.03373 §4.1. The guidance scale is not stated for every card; see each card's hyperparameter table.

Contradictions

• FID against human judgement. The HPS paper finds that IS and FID barely separate human-preferred from non-preferred dataset images: FID 38.2 against 37.7 2303.14420 tab. 1. The SDXL paper reports that its model had the worst FID of the three models compared, yet slightly higher CLIP score, and that humans preferred it 2307.01952 app. On FID Assessment. The PickScore paper reports an FID correlation of −0.900 with human win rates over 100 MS-COCO captions and 9 models, against 0.917 for PickScore 2305.01569 fig. 6. ImageReward's six-model ranking gives FID a Spearman of 0.09 against human ranks 2304.05977 tab. 1. These results are author-reported, with different model sets and prompt sets. The papers do not settle which holds in general.
• Baselines differ across human studies. The DALL-E paper compares against DF-GAN, and the unCLIP paper against GLIDE 2102.12092 §3.1 2204.06125 tab. 1. Their photorealism numbers are not comparable.
• Within SD-GAN. Table 4 and Table 6 give different numbers for the same setting without the Siamese term but with SCBN. Table 4 gives 4.49 ± .06 on CUB and 29.79 ± .61 on COCO 1904.01480 tab. 4. Table 6 labels the word-cue SCBN without Siamese as 4.45 ± .06 and 29.79 ± .61 [arxiv:1904.01480 tab. 6]. The cue level used in Table 4 is not stated.

Papers

• Semantics Disentangling for Text-to-Image Generation: SD-GAN, a stacked text-to-image GAN with a Siamese contrastive loss and semantic-conditioned batch norm.
• Zero-Shot Text-to-Image Generation: DALL-E, a 12-billion-parameter transformer over caption and discrete image tokens, with contrastive reranking.
• Hierarchical Text-Conditional Image Generation with CLIP Latents: unCLIP, a CLIP-embedding prior with a diffusion decoder.
• Muse: Text-To-Image Generation via Masked Generative Transformers: Muse, a masked generative transformer over VQGAN tokens, conditioned on frozen T5-XXL embeddings.
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis: SDXL, a larger latent diffusion model with size and crop conditioning and an optional refiner.
• Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models: SLD, a safety term added to classifier-free guidance during sampling.
• The Stable Signature: Rooting Watermarks in Latent Diffusion Models: a fixed 48-bit key fine-tuned into the latent diffusion decoder and read by a frozen extractor.
• T2I-Adapter: small adapters that add spatial control to a frozen Stable Diffusion model.
• One-step Diffusion with Distribution Matching Distillation: DMD, a one-step student trained to match a diffusion teacher's output distribution.
• LAION-5B: a web-scale dataset of image-text pairs filtered by CLIP similarity.
• Pick-a-Pic: an open dataset of user preferences and PickScore, a learned scorer trained on it.
• ImageReward: a BLIP-based preference scorer, and ReFL, which tunes Stable Diffusion with it.
• GenEval: an object-level benchmark that checks counts, colours, positions and object presence with a detector and a CLIP colour classifier.
• TIFA: a reference-free faithfulness metric that answers prompt-derived questions about the image with a VQA model, and the TIFA v1.0 benchmark.
• Generative Multimodal Models are In-Context Learners: Emu2 and its generation variant Emu2-Gen, a 37B autoregressive model over interleaved text and image embeddings.
• Training-Free Layout Control with Cross-Attention Guidance: backward guidance, a training-free latent update that moves cross-attention mass into user-given boxes on Stable Diffusion.
• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference: HPS, a CLIP ViT-L/14 scorer fitted to human picks, and a LoRA adapter trained on HPS-selected images.