--- kind: concept slug: text-to-image-generation title: Text-to-image generation summary: >- Generating images from captions. Nine generators in eight families, from GANs and token transformers to latent diffusion, adapters, layout steering, one-step distillation and a multimodal model. Also a safety steer, a reward-tuned generator, a watermark, a dataset, three scorers and two evaluators. Results are author-reported; compare within a row. tags: [text-to-image, gan, diffusion-models, autoregressive, masked-generative, distillation, adapters, conditioning, layout-control, safety, reward-modeling, preference-scoring, evaluation, faithfulness, multimodal, watermarking] papers: ["1904.01480", "2102.12092", "2204.06125", "2307.01952", "2302.08453", "2311.18828", "2312.13286", "2301.00704", "2211.05105", "2210.08402", "2305.01569", "2304.05977", "2310.11513", "2303.11897", "2303.15435", "2303.14420", "2304.03373"] related: [generative-adversarial-network, conditional-batch-normalization, contrastive-learning, diffusion-models, latent-diffusion, classifier-free-guidance, autoregressive-image-generation, masked-image-modeling, vector-quantization, variational-autoencoder, knowledge-distillation, parameter-efficient-fine-tuning, low-rank-adaptation, vision-language-pretraining, human-evaluation, generative-model-evaluation, dataset-curation, safety-filter, reward-modeling, rlhf, text-to-image-personalization, text-guided-image-editing, layout-to-image-generation, object-detection, benchmark-validity, multimodal-language-model, in-context-learning, image-watermarking] status: draft updated_at: 2026-11-01 --- ## Overview A text-to-image model takes a caption and returns an image that matches it. Captions for one picture vary in wording, so a model that treats one caption as the only truth can paint a different-looking object for each paraphrase [arxiv:1904.01480 §1]. The cards on this page reach the same goal with different machinery: a stacked GAN [arxiv:1904.01480 §3.1], discrete image tokens predicted by a transformer [arxiv:2102.12092 §2.2], masked token prediction in parallel [arxiv:2301.00704 §2.3], a diffusion decoder steered by CLIP image embeddings [arxiv:2204.06125 §2], latent diffusion [arxiv:2307.01952 §2], a frozen-model adapter for spatial control [arxiv:2302.08453 §3.3], a one-step student of a diffusion teacher [arxiv:2311.18828 §3.1], and one autoregressive model that writes both text and image embeddings [arxiv:2312.13286 §2.1]. Four cards change an existing diffusion model rather than offer a new generator: a safety term added to guidance [arxiv:2211.05105 §3], a reward model whose score is back-propagated into Stable Diffusion [arxiv:2304.05977 §3], a key fixed into the decoder so that every output carries a watermark [arxiv:2303.15435 §4.2], and a training-free term that moves attention into user-given boxes during sampling [arxiv:2304.03373 §3.2]. Five more cards support the field without being generators: a web-scale image-text dataset [arxiv:2210.08402 §3.1], two learned scorers of human preferences [arxiv:2305.01569 §3] [arxiv:2303.14420 §5], a benchmark that checks each prompt's objects, counts, colours and positions one by one [arxiv:2310.11513 §3.2], and a reference-free faithfulness metric that asks questions about the image [arxiv:2303.11897 §3]. For diffusion families, see [Diffusion models](/concepts/diffusion-models). ## Variants ### Stacked GAN with a Siamese contrastive term (SD-GAN) Three generator–discriminator stages, from a 64 × 64 initial image to higher resolutions. Each stage has its own discriminator, and the discriminators share no parameters [arxiv:1904.01480 §3.1]. Training uses two captions per step, one per Siamese branch. A contrastive loss on the discriminator features of the two branches, applied at every stage, pulls together pairs that describe the same image and pushes apart pairs that do not [arxiv:1904.01480 §3.1] [eq. 2]. Same-image pairs receive no pull once they are closer than a floor α = 0.1, and different-image pairs are pushed until a margin ε = 1.0 [arxiv:1904.01480 §3.1] [eq. 2]. The generator's batch normalization layers are replaced by semantic-conditioned batch norm. Its scale and shift come from the sentence vector (sentence cue) or from word-level attention (word cue) [arxiv:1904.01480 §3.2] [eq. 5] [eq. 6]. Author-reported results: Inception score 4.67 ± .09 on CUB against 4.36 ± .03 for AttnGAN, and 35.69 ± .50 on MS-COCO against 25.89 ± .47 [arxiv:1904.01480 tab. 1]. In the authors' ablation, removing SCBN (Siamese term kept) gives 4.51 ± .07 on CUB and 30.18 ± .47 on COCO; removing the Siamese term (SCBN kept) gives 4.49 ± .06 and 29.79 ± .61; removing both gives 4.11 ± .04 and 23.76 ± .40 [arxiv:1904.01480 tab. 4]. Card: [Semantics Disentangling for Text-to-Image Generation](/papers/1904.01480). ### Discrete image tokens with an autoregressive transformer (DALL-E) A discrete autoencoder maps each 256 × 256 image to a 32 × 32 grid of indices into a codebook of 8192 entries [arxiv:2102.12092 §2.1]. A 12-billion-parameter decoder-only transformer reads the caption tokens followed by those image tokens and predicts the next token [arxiv:2102.12092 §2.2]. Candidates are sampled at temperature 1, with no temperature reduction for the main results. The authors draw 512 candidates, decode each one, score it with a pretrained contrastive model, and keep the top k; k is not stated for the main results [arxiv:2102.12092 §2.6]. Author-reported zero-shot results on MS-COCO: raters preferred the images over DF-GAN's by majority vote 90% of the time for realism and 93% for caption match [arxiv:2102.12092 §3.1] [fig. 7]. FID is within about two points of the best prior method on MS-COCO, and about 40 points worse on CUB [arxiv:2102.12092 fig. 9]. Reranking gains flatten after 32 candidates [arxiv:2102.12092 fig. 9c]. The transformer is not released, and the official repository holds only the autoencoder. A reproduction must write the transformer from the paper's text and figures: see the card's pitfalls. Card: [Zero-Shot Text-to-Image Generation](/papers/2102.12092). ### Masked tokens filled in parallel (Muse) A VQGAN tokenizer turns each image into a grid of codes from a codebook of 8192 entries. The base model works on a 16 × 16 grid and the super-resolution model on 64 × 64 [arxiv:2301.00704 §2.2]. A transformer with cross-attention to frozen T5-XXL text embeddings is trained to predict hidden codes [arxiv:2301.00704 §2.1] [§2.3]. The share of hidden codes is drawn per example from a fixed density with mean about 0.64, so most training steps hide most of the grid [arxiv:2301.00704 §2.6]. At sampling time every code starts hidden. The model fills in its most confident codes over a fixed number of passes: 24 for the base grid and 8 for the super-resolution grid [arxiv:2301.00704 §2.8]. A token-by-token autoregressive model needs one pass per code, which is 256 passes for the base grid [arxiv:2301.00704 §2.8]. Author-reported: CC3M FID 6.06 for the base model with super-resolution, against 17.01 for LDM-4 on the same dataset [arxiv:2301.00704 tab. 1]. Zero-shot on MS-COCO at 256 × 256, the 3B model reaches FID 7.88 [arxiv:2301.00704 tab. 2], against 7.27 for Imagen and 10.39 for DALL-E 2. On PartiPrompts, raters preferred the 3B model on 70.6% of prompts, against 25.4% for Stable Diffusion v1.4 [arxiv:2301.00704 §3.2.1]. Each model's output was the best of 16 by CLIP score, so the same selection rule is needed to reproduce this figure. Inference takes 0.5 s per batch at 256 × 256 on the authors' TPUs, against 3.7 s for LDM with 50 steps on an external A100 benchmark. The batch size is not stated, and the hardware differs [arxiv:2301.00704 tab. 3]. No code or weights were released [arxiv:2301.00704 §5]. The cosine schedule, the sampling temperature and the guidance ramp are not stated, so a reimplementation must choose them [arxiv:2301.00704 §2.8] [§3.2]. Card: [Muse: Text-To-Image Generation via Masked Generative Transformers](/papers/2301.00704). ### Caption to CLIP image embedding, then a diffusion decoder (unCLIP) A prior maps a caption to a CLIP image embedding, and a diffusion decoder inverts the CLIP image encoder to render that embedding [arxiv:2204.06125 §2]. The authors try two priors: an autoregressive one over quantised principal components of the embedding, and a diffusion one that predicts the clean embedding directly [arxiv:2204.06125 §2.2]. At similar size, the authors find the diffusion prior gives better results [arxiv:2204.06125 §5.1] [§5.2]. The decoder is a 3.5-billion-parameter model built on GLIDE's architecture [arxiv:2204.06125 tab. 3]. Zero-shot FID on MS-COCO at 256 × 256 is 10.39 with the diffusion prior and 10.63 with the autoregressive prior [arxiv:2204.06125 tab. 2]. In human comparisons against GLIDE, the diffusion-prior model won on diversity 70.5% of the time, while GLIDE was preferred on photorealism (48.9% for unCLIP) and on caption similarity (45.3%) [arxiv:2204.06125 tab. 1]. The same CLIP space supports variations, interpolation between two images, and moving an image toward a text direction [arxiv:2204.06125 §3]. The image encoder was trained by the authors on about 650 million images. The paper does not say that its weights are released, and it does not evaluate public CLIP weights in its place [arxiv:2204.06125 app. C]. Card: [Hierarchical Text-Conditional Image Generation with CLIP Latents](/papers/2204.06125). ### Latent diffusion with size and crop conditioning (SDXL) A latent diffusion model whose UNet has about 2.6 billion parameters [arxiv:2307.01952 §2.1]. It reads the prompt with two text encoders, which together have 817 million parameters [arxiv:2307.01952 §2.1]. Rather than discarding small or oddly shaped training images, the model is given each image's original size and crop offset as extra inputs. At inference the crop offset is set to zero [arxiv:2307.01952 §2.2]. Training then uses buckets of several aspect ratios near 1024² pixels [arxiv:2307.01952 §2.3]. An optional refiner, trained in the same latent space, re-denoises the base output, starting from noise level 200 of 1000 [arxiv:2307.01952 §2.5]. Author-reported user study on four-way picks: refiner output won 48.44%, the base model 36.93%, SD 1.5 7.91% and SD 2.1 6.71% [arxiv:2307.01952 §2.5]. The model weights carry a licence separate from the paper and the code; check it before any commercial use (see the card's pitfalls). Card: [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](/papers/2307.01952). ### Training-free layout steering of Stable Diffusion (backward guidance) The Stable Diffusion weights stay fixed. During the first denoising steps the latent itself is updated by gradients of an energy. For each prompt word, the energy measures how little of that word's cross-attention mass falls inside its user-given box [arxiv:2304.03373 §3.2] [eq. 2] [eq. 3]. The main setting uses guidance on the first 10 steps, up to 5 updates per step, and a strength of 30 [arxiv:2304.03373 §4.1]. The paper's alternative, forward guidance, edits the attention maps directly; the authors find the latent update works better [arxiv:2304.03373 tab. 1]. Author-reported, on a 1,000-prompt sample of the VISOR protocol: conditional accuracy rises from 59.8% for Stable Diffusion to 96.9%, and the full protocol gives 95.95% against 62.98% [arxiv:2304.03373 tab. 1] [tab. 2]. The table caption for the full protocol gives no prompt count; the appendix describes 25,280 prompts [arxiv:2304.03373 app. B]. On COCO layouts scored with a detector, mAP rises from 19.2 to 35.7, while FID moves from 72.3 to 73.3 [arxiv:2304.03373 tab. 3]. Sampling takes about twice as long, roughly 8 s against 4 s per image in the authors' table, and the hardware is not stated [arxiv:2304.03373 tab. 1] [§4]. A large strength hurts fidelity: FID is 88.4 at 100 and 129.7 at 500 [arxiv:2304.03373 app. C]. For models trained on layouts, see [Layout-to-image generation](/concepts/layout-to-image-generation). Card: [Training-Free Layout Control with Cross-Attention Guidance](/papers/2304.03373). ### Sampling-time safety steering of Stable Diffusion (SLD) Safe latent diffusion keeps Stable Diffusion frozen and changes only the guidance step. Each step makes three noise predictions: unconditional, one conditioned on the prompt, and one conditioned on a text description of unwanted content [arxiv:2211.05105 §3] [eq. 3]. The prompt prediction is pushed away from the unwanted-content prediction, element by element. The push is applied only where the prompt prediction does not already exceed the unwanted-content prediction by a threshold [arxiv:2211.05105 §3] [eq. 4] [eq. 5]. The safety term starts only after a warm-up of some steps, and a momentum term smooths it over steps [arxiv:2211.05105 §3] [alg. 1]. The default concept is a list of about twenty words, from violence and nudity to drug use [arxiv:2211.05105 §4]. Author-reported on the I2P benchmark, the probability of an inappropriate image falls from 0.39 for Stable Diffusion v1.4 to 0.09 at the strongest preset, and to 0.18 with negative prompting [arxiv:2211.05105 tab. 2]. The cost shows in image quality: COCO FID rises from 14.43 to 18.28 at the strong preset [arxiv:2211.05105 tab. 3]. In a DrawBench user study, 62% to 64% of users rated quality as better or equal to the baseline [arxiv:2211.05105 tab. 3]. The authors note that the same mechanism could be used for censorship, and ask that any deployment state which content is suppressed [arxiv:2211.05105 §7]. The released code differs from the paper's equations in a few places; see the card before reproducing numbers. Card: [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](/papers/2211.05105). ### Watermark fixed into the decoder (Stable Signature) The method fine-tunes only the decoder of a latent diffusion model. The copy is trained for about 100 steps on fewer than 500 images, which the paper reports as about one minute on one GPU [arxiv:2303.15435 §4.2]. The result carries one fixed 48-bit key in every output. A separate extractor reads the key, and it is pre-trained with simulated edits and then whitened [arxiv:2303.15435 §4.1] [app. A.1]. Generation itself is unchanged [arxiv:2303.15435 §4.2]. Detection counts matching bits against a threshold. Identification compares the image with each user's key and raises the threshold as the number of keys grows [arxiv:2303.15435 §3] [§5.2]. Author-reported: 0.99 bit accuracy on unmodified text-to-image output, 0.92 after a combined crop, brightness and JPEG edit, and an FID change of −0.3 [arxiv:2303.15435 tab. 1]. With 10^5 users, identification is 98% on unmodified images and 40% after the combined edit [arxiv:2303.15435 §5.2]. The headline text-to-image numbers come from an internal 2.2B-parameter model, not from the public code [arxiv:2303.15435 §5]. The released code is CC BY-NC 4.0, so it is not cleared for commercial reuse [code: LICENSE#L1]. Card: [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](/papers/2303.15435). ### One-step student of a diffusion teacher (DMD) The student is a one-step generator, initialised from the teacher [arxiv:2311.18828 §3.1]. Its update combines the difference of two score estimates, one from the frozen teacher and one from a fake-score network trained alongside on the generator's own outputs [arxiv:2311.18828 §3.2]. A regression term to teacher outputs on paired noise, weighted 0.25 by default, is added to keep the sample modes covered [arxiv:2311.18828 §3.3]. Class-conditional ImageNet 64 × 64: FID 2.62 in one forward pass, against 2.32 for the EDM teacher with 512 [arxiv:2311.18828 tab. 1]. Text-to-image, distilled from Stable Diffusion v1.5 at guidance 3: 0.09 s against 2.59 s per image, with FID 11.49 against 8.78 for the teacher [arxiv:2311.18828 tab. 3]. At guidance 8 the teacher is better on FID and CLIP score (13.45 against 14.93; 0.322 against 0.320) [arxiv:2311.18828 tab. 4]. The authors report about 30 times the speed of the teacher [arxiv:2311.18828 §4.3]; the hardware for the latency figures is not stated. Cost: the LAION-based text-to-image run took about 36 hours on 72 A100 GPUs [arxiv:2311.18828 §4.3]. The method trains two networks at once, so memory use is high [arxiv:2311.18828 §5]. No official code exists; see the card. Card: [One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828). ### Multimodal language model that emits image embeddings (Emu2-Gen) Emu2 is one autoregressive model over interleaved text and images. Each image is written as 64 embeddings, and the model predicts either the next text token or the next image embedding [arxiv:2312.13286 §2.1] [§2.2.2]. A visual decoder built on SDXL-base turns those 64 embeddings into a 1024 × 1024 image [arxiv:2312.13286 §2.2.3]. Generation is a separate instruction-tuned variant, Emu2-Gen, trained on text phrases, bounding boxes and object images for controllable output [arxiv:2312.13286 §2.3.2]. Emu2-Chat, for dialogue and grounding, is a different variant [arxiv:2312.13286 §2.3]. Author-reported zero-shot results on MS-COCO, with CLIP-I comparing against reference images and CLIP-T comparing against the prompt: Emu2-Gen scores 0.686 and 0.297, SDXL 0.674 and 0.310, and SDv1.5 0.667 and 0.302 [arxiv:2312.13286 tab. 4]. On the authors' CLIP measures, Emu2-Gen is ahead on image similarity and behind SDXL on prompt similarity. The sampler uses 50 steps and a guidance scale of 3.0 [arxiv:2312.13286 app. E.3]. The code is Apache-2.0, but its README lists the model weights as non-commercial. Treat the weights as not cleared for commercial use; see the card's pitfalls. The released code also runs the full language model once per image slot, with no key/value cache, so generation is slow. Card: [Generative Multimodal Models are In-Context Learners](/papers/2312.13286). ## Data, scoring and evaluation These cards mostly do not generate images on their own. They supply training data, a way to judge outputs, or a benchmark. ImageReward's ReFL is the exception: it tunes a generator with a learned score. ### Web-scale image-text data (LAION-5B) LAION-5B is a dataset release, not a model. It keeps about 5.85 billion image-text pairs drawn from web alt-text. A CLIP ViT-B/32 similarity filter keeps English pairs at 0.28 or above and other pairs at 0.26 or above [arxiv:2210.08402 §3.1]. Potentially offensive pairs are tagged rather than removed, and about 3% of images are flagged NSFW [arxiv:2210.08402 §3.2] [§4]. Many captions are spam or keyword lists [arxiv:2210.08402 §6]. Stable Diffusion checkpoints and GLIDE fine-tunes were trained on LAION subsets [arxiv:2210.08402 app. F]. The paper's CC BY 4.0 licence covers the metadata; the authors say they do not own the images or the text, so check the rights for each source before reuse [arxiv:2210.08402 app. A]. Card: [LAION-5B: An open large-scale dataset for training next generation image-text models](/papers/2210.08402). ### Learned preference scorer (Pick-a-Pic, PickScore) PickScore is a CLIP ViT-H/14 model fine-tuned on user choices between two images for one prompt, with a two-way softmax loss and tie targets [arxiv:2305.01569 §3]. It predicts held-out picks with 70.5% accuracy, against 68.0% for expert annotators and 60.8% for CLIP-H [arxiv:2305.01569 tab. 1]. The authors report a Pearson correlation of 0.790 with real-user Elo ranks across 45 models, against 0.670 for HPS, 0.492 for ImageReward and 0.313 for CLIP-H [arxiv:2305.01569 fig. 7]. When 100 candidates per prompt are ranked by the scorer, human raters preferred PickScore's pick over CLIP-H's pick 71.3% of the time [arxiv:2305.01569 tab. 2]. The human baseline is a group of the authors' friends and colleagues, and the number of annotators is not stated [arxiv:2305.01569 §4]. Licences for the dataset and model are not stated in the paper or repository. Card: [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](/papers/2305.01569). ### Human-choice classifier on CLIP (HPS) HPS fine-tunes CLIP ViT-L/14 so that a prompt's text embedding sits close to the embedding of the image people picked, out of several images made from the same prompt. The score is 100 times the cosine similarity between the two embeddings [arxiv:2303.14420 §5]. The paper gives the loss in words only, so the exact form is not stated [arxiv:2303.14420 §5]. The training picks come from a public Discord channel of Stable Diffusion images: 25,205 prompts and 2,659 users [arxiv:2303.14420 §3]. The authors say the data are distributed under a non-commercial licence [arxiv:2303.14420 app. datasheet], and the dataset reflects a small, active community [arxiv:2303.14420 §8]. Author-reported: HPS picks the human choice 43.5% of the time on 5,000 user choices, against 42.0% for human participants, 32.9% for CLIP ViT-L/14 and 26.1% for random [arxiv:2303.14420 tab. 2]. On 398 Stable Diffusion against DALL·E pairs, HPS agrees with human picks 61.5% of the time, against 56.8% for CLIP and 63.5% between humans [arxiv:2303.14420 tab. 3]. These choices are not the Pick-a-Pic test set, so the numbers do not compare with the PickScore row above. The same score selects training images for a LoRA adapter on Stable Diffusion v1.4. The paper uses rank 32 and 10,000 steps [arxiv:2303.14420 §6]. Its user study shows 74% of the adapted model's images with more than ten positive votes, against 22% for the original, from 20 participants on 100 prompts; no variance is reported [arxiv:2303.14420 §7.2] [fig. 8]. The paper's released code differs from it in LoRA rank and score scale, so see the HPS card before reproducing these results. Card: [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](/papers/2303.14420). ### Learned preference scorer and reward tuning (ImageReward) ImageReward is a BLIP-based scorer trained on 136,892 pairwise comparisons, made by expert annotators on prompts from DiffusionDB [arxiv:2304.05977 §2.1] [§2.2]. Training uses only the order of images for each prompt, and the score is a single number for a prompt and image pair [arxiv:2304.05977 §2.2] [eq. 1]. The same score can also tune Stable Diffusion v1.4. ReFL runs most denoising steps without gradients, keeps gradients for one randomly chosen late step, and back-propagates the score of the resulting image [arxiv:2304.05977 §3]. On its own held-out comparisons, ImageReward reaches 65.14% preference accuracy. The same table gives 62.78% for PickScore, 60.79% for HPS, 57.76% for BLIP score and 54.82% for CLIP score [arxiv:2304.05977 tab. 3] [tab. 5]. These numbers use the authors' test set, not Pick-a-Pic's, so they do not compare directly with the PickScore row above. On six models and 100 prompts, the authors' ranking matches ImageReward with Spearman 1.00, against 0.60 for CLIP and 0.09 for FID [arxiv:2304.05977 tab. 1]. In human comparisons on real user prompts, ReFL beats Stable Diffusion v1.4 in 58.79% of comparisons [arxiv:2304.05977 tab. 4]. The number of raters for these studies is not stated. The released training script differs from the paper in its loss, its regularising term and its training length. See the card's pitfalls before reproducing ReFL. Card: [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](/papers/2304.05977). ### Object-level benchmark (GenEval) GenEval turns each prompt into a checklist of object classes, counts, colours and relative positions [arxiv:2310.11513 §3.1]. An object detector trained on the 80 COCO classes finds the objects, a CLIP ViT-L/14 classifier names their colours, and the relative positions are read from box centres. Each image gets a yes or no, and the six task scores are averaged without weights [arxiv:2310.11513 §3.2]. Author-reported: IF-XL scores 0.61 overall, followed by SD-XL at 0.55; the best position score is 0.15 (SD-XL) and the best attribute-binding score is 0.35 (IF-XL) [arxiv:2310.11513 tab. 2]. Overall scores move by 0.01 to 0.02 across random seeds, so smaller gaps are not meaningful [arxiv:2310.11513 tab. 2]. Against annotators, GenEval agrees on 83% of images, against 88% between annotators and 80% for thresholded CLIPScore [arxiv:2310.11513 §4]. Two of its thresholds (the counting threshold and the position dead-zone factor) were chosen on the same annotations; K-fold checks give 0.823 and 0.822 [arxiv:2310.11513 app. A.2]. The detector cannot verify fine details such as finger counts, and it generalises poorly to artistic renderings [arxiv:2310.11513 §6]. The released scorer differs from the paper's text in a few places, such as a per-class object cap and the position rule; see the card before reproducing numbers. Card: [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](/papers/2310.11513). ### Question-based faithfulness score (TIFA) TIFA does not generate images and needs no reference image. A language model writes multiple-choice and yes/no questions, each about one element of the prompt, with known answers [arxiv:2303.11897 §3.1]. A second model, UnifiedQA, keeps only the questions whose answers it reproduces [arxiv:2303.11897 §3.2]. A visual question answering (VQA) model then answers each question on the generated image, and the score is the share answered correctly [arxiv:2303.11897 §3] [eq. 1]. Author-reported Spearman correlation with human faithfulness ratings on 800 images: 59.7 for TIFA with mPLUG, against 33.2 for CLIPScore and 18.3 to 34.0 for the caption-based metrics [arxiv:2303.11897 tab. 2]. Of the five VQA models the authors tested, mPLUG had the highest VQA accuracy against human answers and the highest correlation [arxiv:2303.11897 tab. 3]. Models do poorly on shape, counting and spatial relations [arxiv:2303.11897 §5.3]. The authors release TIFA v1.0: 4,081 text inputs and 25,829 questions, with the questions generated once and then filtered [arxiv:2303.11897 tab. 1] [§4]. Users then need only a VQA model to score a new set of images. Questions written by a different language model change the wording and counts, so scores are not directly comparable [arxiv:2303.11897 §4.3]. Card: [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](/papers/2303.11897). ## When to use which | Situation | Use | Why | |---|---|---| | Training data has several captions per image, and you can tell which captions belong to the same image | SD-GAN style Siamese contrastive loss | The method assumes this grouping [arxiv:1904.01480 §4.1]. Without it there are no positive pairs. | | You can afford about two captions per training step and a GAN-style generator | SD-GAN setup | The Siamese variant costs more per step. The paper gives no timings or memory figures. Its ablations use the non-Siamese variant because it is cheaper [arxiv:1904.01480 §4.3]. | | You need a reproduction of SD-GAN without official code | Start from the cited AttnGAN setup, and fix the unstated settings yourself | The paper says it follows AttnGAN for losses, encoder and Inception features [arxiv:1904.01480 §4.1]. Optimizer, learning rate, batch size and the loss weight of the contrastive term are not stated [arxiv:1904.01480 §4.1]. | | You want to compare models on paraphrase consistency | Do not rely on the SD-GAN paper alone | The paper reports only the Inception score and a human ranking, with no FID and no caption-consistency metric [arxiv:1904.01480 §4.1]. | | You already have a Stable Diffusion checkpoint and need sketch, pose, depth or segmentation control | Frozen-model adapter (T2I-Adapter) | The base model is not retrained, and the adapter is about 77M parameters [arxiv:2302.08453 §1]. Composition weights must be tuned by hand [arxiv:2302.08453 §3.3]. | | You need a masked-token model with few sampling passes, and you will build the pipeline yourself | Muse-style masked generation | 24 passes for the base grid, against 256 for token-by-token decoding [arxiv:2301.00704 §2.8]. No code or weights were released [arxiv:2301.00704 §5], and the cosine schedule and temperature are not stated [arxiv:2301.00704 §2.8]. | | You must cut inappropriate output from a Stable Diffusion model during sampling, with no retraining and no external classifier | Safe latent diffusion (SLD) | The inappropriate probability falls from 0.39 to 0.09 at the strongest preset [arxiv:2211.05105 tab. 2]. Quality pays for it: COCO FID rises from 14.43 to 18.28 [arxiv:2211.05105 tab. 3]. The approach adds no classifier and no training [arxiv:2211.05105 §1]. The guidance scale for the main runs is not stated, so tune it yourself. | | You want to tune a Stable Diffusion v1.4 model toward a learned preference score | ReFL, with ImageReward as the score | ReFL is preferred over the baseline in 58.79% of real-prompt comparisons [arxiv:2304.05977 tab. 4]. The reward gradient passes through one denoising step, the decoder and the reward model [arxiv:2304.05977 §3]. The released script omits the paper's regulariser. | | You want every image from your own latent diffusion model to carry a provenance key, and you can change its decoder | Stable Signature | The generation process is unchanged and no post-generation step is needed [arxiv:2303.15435 §4.2]. After the combined edit its bit accuracy is 0.92, against 0.98 for the authors' post-generation HiDDeN, whose bits the authors say are not independent [arxiv:2303.15435 tab. 1] [§6.4]. The released code is non-commercial [code: LICENSE#L1]. | | You need objects in user-given boxes on a Stable Diffusion v1.5 model, and you cannot train | Backward layout guidance | VISOR conditional accuracy rises from 59.8% to 96.9% on a 1,000-prompt sample [arxiv:2304.03373 tab. 1]. Sampling takes about twice as long [arxiv:2304.03373 tab. 1], and a large strength hurts FID [arxiv:2304.03373 app. C]. For models trained with layouts, see [Layout-to-image generation](/concepts/layout-to-image-generation). | | Sampling latency matters more than training cost | One-step distillation (DMD) | 0.09 s against 2.59 s per image on SD v1.5, at a cost in FID (11.49 against 8.78 at guidance 3) [arxiv:2311.18828 tab. 3]. Training needs a teacher and a second network [arxiv:2311.18828 §5]. | | You need diverse samples from one caption, and exact caption fidelity matters less | unCLIP with the diffusion prior | Diversity was preferred over GLIDE 70.5% of the time, while caption similarity was 45.3% [arxiv:2204.06125 tab. 1]. You must train your own CLIP image encoder, since the paper does not say its weights are released [arxiv:2204.06125 app. C]. | | You want a high-resolution open-weight base model and can host two text encoders and a large UNet | SDXL, optionally with the refiner | Size and crop conditioning keep small and oddly shaped training images [arxiv:2307.01952 §2.2]. The refiner is optional, and the paper gives no timing figure for it [arxiv:2307.01952 §2.5]. The weights have a separate licence, so check it first (see the SDXL card). | | You want a CLIP-based human-preference score for ranking candidates from a Stable Diffusion-style model, and you accept a score fitted to one community's picks | HPS | 43.5% on its own 5,000 user choices, against 32.9% for CLIP ViT-L/14 [arxiv:2303.14420 tab. 2]. The choices come from one Discord community [arxiv:2303.14420 §8], and the dataset is non-commercial [arxiv:2303.14420 app. datasheet]. | | You want a human-preference score for candidates from any generator | PickScore, or ImageReward | PickScore reaches 70.5% on held-out picks [arxiv:2305.01569 tab. 1]. On the ImageReward test set, ImageReward reaches 65.14% against PickScore's 62.78% [arxiv:2304.05977 tab. 5]. DALL-E's own reranking gains flatten after 32 candidates [arxiv:2102.12092 fig. 9c]. | | You need to know which skill a model lacks (counting, position, colour binding) rather than one overall score | GenEval, with its per-task scores | Per-task scores separate the skills; the best position score is 0.15 and the best attribute-binding score 0.35 [arxiv:2310.11513 tab. 2]. Agreement with annotators is 83%, against 80% for thresholded CLIPScore [arxiv:2310.11513 §4]. | | You need a large open image-text corpus for pretraining | LAION-5B, with rights checked per source | About 5.85 billion pairs with a CLIP filter [arxiv:2210.08402 §3.1]. The paper's licence covers metadata only [arxiv:2210.08402 app. A]. | | You need a reference-free score of whether an image matches its prompt, broken down by element type | TIFA, with mPLUG as the VQA model | Spearman 59.7 against CLIPScore's 33.2 [arxiv:2303.11897 tab. 2]. The cost is one VQA call per question: about 0.25 s per question on an A40, or about 1.6 s per image for an average prompt [arxiv:2303.11897 §5.4] [tab. 1]. | | You want a score that is comparable with the published TIFA numbers | Use the released TIFA v1.0 questions, not newly generated ones | Regenerated questions change wording, categories and count [arxiv:2303.11897 §4.3]. | | You need one autoregressive model that writes both text and images, for research only | Emu2-Gen | Its MS-COCO CLIP-I is 0.686 against SDXL's 0.674 [arxiv:2312.13286 tab. 4]. The weights are listed as non-commercial, and generation runs a full language model pass per image slot, so it is slow (see the card's pitfalls). | ## Implementation sketch The first two blocks are SD-GAN's and cover its losses. The third is the shape shared by the diffusion and autoregressive cards. Pseudocode in our own words, following the papers' equations. ```pseudocode # Pair loss for one discriminator stage [arxiv:1904.01480 §3.1] [eq. 2] # Input: d (B,): feature distance between the two branches # same (B,): 1 if both captions describe one image, else 0 # Output: scalar def pair_loss(d, same, alpha=0.1, eps=1.0): # Same-image pairs: pull together, no gradient below floor alpha [eq. 2] pull = same * max(d, alpha) ** 2 # Different-image pairs: push apart until the margin eps [eq. 2] push = (1 - same) * max(eps - d, 0) ** 2 # The 1/(2N) scaling is not settled by the paper: the sum bound is # ambiguous (see the card's pitfalls) [arxiv:1904.01480 §3.1] return mean(pull + push) / 2 ``` ```pseudocode # Semantic-conditioned batch norm, sentence cue [arxiv:1904.01480 §3.2] [eq. 4-5] # Input: x (B, C, H, W) feature maps; s (B, D) sentence vector # Output: modulated features (B, C, H, W) def scbn_sentence(x, s): # Plain batch norm per channel [eq. 3] x_hat = normalize_per_channel(x) # Caption-derived scale and shift, broadcast over H and W [eq. 5] gamma_c = mlp_gamma(s) beta_c = mlp_beta(s) # Learned scale and shift plus the caption-derived terms [eq. 4] return (gamma + gamma_c) * x_hat + (beta + beta_c) ``` ```pseudocode # Guided sampling with optional reranking # Shape shared by the diffusion cards; see the notes below the block. # Input: caption; guidance strength w; number of candidates n # Output: one image def generate(caption, w, n, scorer=None): # Encode the caption; the null embedding serves the unconditional branch # [arxiv:2307.01952 app. Diffusion Models] cond = text_encoder(caption) uncond = null_embedding() candidates = [] for _ in range(n): # Guidance: (1 + w) * cond - w * uncond; the step rule is not shown # [arxiv:2307.01952 app. Diffusion Models] latent = sample_guided(cond, uncond, w) candidates.append(decode(latent)) if scorer is None: return candidates[0] # Keep the highest-scoring candidate, as in the reranking cards # [arxiv:2102.12092 §2.6] [arxiv:2305.01569 §6] return max(candidates, key=lambda img: scorer(caption, img)) ``` Where the variants differ: DALL-E's sampler draws its candidates at temperature 1 and reranks them as a separate step; the paper's description has no guidance term [arxiv:2102.12092 §2.6]. T2I-Adapter adds its control features once per sample instead of inside the loop [arxiv:2302.08453 §1]. DMD takes one step, so the loop is a single pass [arxiv:2311.18828 §3.1]. SLD adds a third noise prediction and subtracts a scaled concept term inside the loop [arxiv:2211.05105 §3] [eq. 3]. Stable Signature keeps the loop as it is and swaps in the fine-tuned decoder before decoding [arxiv:2303.15435 §4.2]. Muse replaces the noise loop with parallel token filling over a few passes [arxiv:2301.00704 §2.8]. Backward layout guidance takes gradient steps on the latent for the first 10 steps, each before the ordinary denoising step [arxiv:2304.03373 §4.1]. The guidance scale is not stated for every card; see each card's hyperparameter table. ## Contradictions - **FID against human judgement.** The HPS paper finds that IS and FID barely separate human-preferred from non-preferred dataset images: FID 38.2 against 37.7 [arxiv:2303.14420 tab. 1]. The SDXL paper reports that its model had the worst FID of the three models compared, yet slightly higher CLIP score, and that humans preferred it [arxiv:2307.01952 app. On FID Assessment]. The PickScore paper reports an FID correlation of −0.900 with human win rates over 100 MS-COCO captions and 9 models, against 0.917 for PickScore [arxiv:2305.01569 fig. 6]. ImageReward's six-model ranking gives FID a Spearman of 0.09 against human ranks [arxiv:2304.05977 tab. 1]. These results are author-reported, with different model sets and prompt sets. The papers do not settle which holds in general. - **Baselines differ across human studies.** The DALL-E paper compares against DF-GAN, and the unCLIP paper against GLIDE [arxiv:2102.12092 §3.1] [arxiv:2204.06125 tab. 1]. Their photorealism numbers are not comparable. - **Within SD-GAN.** Table 4 and Table 6 give different numbers for the same setting without the Siamese term but with SCBN. Table 4 gives 4.49 ± .06 on CUB and 29.79 ± .61 on COCO [arxiv:1904.01480 tab. 4]. Table 6 labels the word-cue SCBN without Siamese as 4.45 ± .06 and 29.79 ± .61 [arxiv:1904.01480 tab. 6]. The cue level used in Table 4 is not stated. ## Papers - [Semantics Disentangling for Text-to-Image Generation](/papers/1904.01480): SD-GAN, a stacked text-to-image GAN with a Siamese contrastive loss and semantic-conditioned batch norm. - [Zero-Shot Text-to-Image Generation](/papers/2102.12092): DALL-E, a 12-billion-parameter transformer over caption and discrete image tokens, with contrastive reranking. - [Hierarchical Text-Conditional Image Generation with CLIP Latents](/papers/2204.06125): unCLIP, a CLIP-embedding prior with a diffusion decoder. - [Muse: Text-To-Image Generation via Masked Generative Transformers](/papers/2301.00704): Muse, a masked generative transformer over VQGAN tokens, conditioned on frozen T5-XXL embeddings. - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](/papers/2307.01952): SDXL, a larger latent diffusion model with size and crop conditioning and an optional refiner. - [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](/papers/2211.05105): SLD, a safety term added to classifier-free guidance during sampling. - [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](/papers/2303.15435): a fixed 48-bit key fine-tuned into the latent diffusion decoder and read by a frozen extractor. - [T2I-Adapter](/papers/2302.08453): small adapters that add spatial control to a frozen Stable Diffusion model. - [One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828): DMD, a one-step student trained to match a diffusion teacher's output distribution. - [LAION-5B](/papers/2210.08402): a web-scale dataset of image-text pairs filtered by CLIP similarity. - [Pick-a-Pic](/papers/2305.01569): an open dataset of user preferences and PickScore, a learned scorer trained on it. - [ImageReward](/papers/2304.05977): a BLIP-based preference scorer, and ReFL, which tunes Stable Diffusion with it. - [GenEval](/papers/2310.11513): an object-level benchmark that checks counts, colours, positions and object presence with a detector and a CLIP colour classifier. - [TIFA](/papers/2303.11897): a reference-free faithfulness metric that answers prompt-derived questions about the image with a VQA model, and the TIFA v1.0 benchmark. - [Generative Multimodal Models are In-Context Learners](/papers/2312.13286): Emu2 and its generation variant Emu2-Gen, a 37B autoregressive model over interleaved text and image embeddings. - [Training-Free Layout Control with Cross-Attention Guidance](/papers/2304.03373): backward guidance, a training-free latent update that moves cross-attention mass into user-given boxes on Stable Diffusion. - [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](/papers/2303.14420): HPS, a CLIP ViT-L/14 scorer fitted to human picks, and a LoRA adapter trained on HPS-selected images.