Hierarchical Text-Conditional Image Generation with CLIP Latents

In plain words

Text-to-image models have to answer two questions: what the picture should contain, and how it should look. This paper splits the work in two. A first model takes the caption and writes a short summary of the picture in the language of CLIP, a model that already knows how images and text relate. A second model, a diffusion image generator, takes that summary and paints the picture. Because the summary is a compact description and not the full set of pixel-level choices, the painter can produce several different pictures from the same caption, which the authors show is a weakness of earlier systems.

The same summary can be reused for other tasks. Given a real photo, the system can encode it into this summary and decode new versions that keep the subject and style but change small details. It can also blend two photos by mixing their summaries, or nudge a photo toward a new text description by moving its summary in the direction of that text.

The authors report that the summary-first approach is close to an earlier strong model on photorealism and gives more variety. They also find that the summary-writing step is cheaper when it is itself a diffusion model rather than an autoregressive one.

TL;DR

• A text-to-image stack with two stages: a prior that maps a caption to a CLIP image embedding, and a diffusion decoder that inverts the CLIP image encoder to produce the image §2.
• Diffusion decoder: a 3.5B GLIDE-style model with the CLIP embedding added to the timestep embedding and projected into four extra context tokens §2.1 app. C.
• The diffusion prior is cheaper than the autoregressive prior for comparable size and compute, and it gives better results §5.1 §5.2.
• The CLIP embedding space allows image variations, interpolation between images and text-direction edits §3.
• The paper reports a zero-shot MS-COCO FID of 10.39 with the diffusion prior, and a human-preference result that is close to GLIDE on photorealism and better on diversity §5.4 tab. 1 tab. 2. All numbers are author-reported.

Problem

Models that generate directly from pixels with guidance give realistic images, but guidance lowers sample diversity §1 §5.3. The paper also argues that a text caption alone is not enough for the decoder to match captions as well as a model trained on the image representation §5.1.

Earlier text-to-image systems such as DALL-E and GLIDE were the points of comparison §1 §5.2. Autoregressive priors in CLIP space are expensive to train and sample from §2.2 §5.1.

Method

The model is a generative stack P(x|y) built from a prior P(z_i|y) and a decoder P(x|z_i, y), where x is the image, y its caption and z_i its CLIP image embedding. The decoder is trained to invert the CLIP image encoder, and the prior supplies embeddings from captions §2. The equation for the stacked model is unnumbered in the source §2.

Decoder §2.1. A diffusion model, built from GLIDE's architecture §2.1 app. C. The CLIP embedding is added to the timestep embedding and also projected into four context tokens that are appended to the GLIDE text tokens. The text pathway from GLIDE is kept, but the authors find it adds little §2.1 §7. Classifier-free guidance is trained by dropping the CLIP embedding and the caption at random §2.1. Two diffusion upsamplers take the 64×64 output to 256×256 and then 1024×1024. Each upsampler is trained on random crops at one quarter of the target size, with a corrupted conditioning image: Gaussian blur for the first, a BSR degradation for the second. The upsamplers use no attention, no caption and no guidance §2.1 app. C tab. 3.

Prior §2.2. Two options:

• Autoregressive prior. The CLIP image embedding is reduced to its first 319 principal components out of 1,024 with PCA. Each component is quantised into one of 1,024 buckets. A causal Transformer predicts the token sequence, conditioned on the caption and on the CLIP text embedding, which is prefixed to the sequence. A token for the quantised dot product between the text embedding and the image embedding is also prepended, and at sampling time the authors draw it from the top half of the distribution §2.2. PCA cuts the number of predicted tokens about threefold §2.2.
• Diffusion prior. A decoder-only Transformer with a causal mask reads, in order: the encoded text, the CLIP text embedding, the diffusion timestep, the noised image embedding, and a final output token that gives the prediction. The model is trained to predict the clean embedding directly, not the noise, with a mean-squared error §2.2. The dot-product condition is not used. Instead, two samples are drawn and the one with the higher dot product to the text embedding is kept §2.2.

The text conditioning is dropped 10% of the time for both priors §2.2.

# Text-to-image sampling: prior, then decoder  [§2] [§2.2] [§2.1]
# Input:  caption (string), decoder guidance scale
# Output: image at 1024 x 1024

def generate(caption, guidance_scale):
    # Embed the caption with the frozen CLIP text encoder  [§2]
    text_embedding = clip_text_encoder(caption)
    # Sample a CLIP image embedding from the prior  [§2.2]
    image_embedding = prior_sample(caption, text_embedding)  # (1024,)
    # Decode the embedding into a 64 x 64 image  [§2.1] [app. C]
    image_64 = decoder_sample(image_embedding, caption, guidance_scale)
    # Upsample 64 -> 256, then 256 -> 1024, without captions  [§2.1]
    image_256 = upsampler_256(image_64)
    return upsampler_1024(image_256)

# Decoder training: conditioning dropout  [§2.1]
# Input:  image, caption, image_embedding (CLIP image encoder output)
# Output: the conditioning inputs for one diffusion training step

def decoder_inputs(image, caption, image_embedding):
    # Zero the CLIP embedding (or use a learned vector) 10% of the time  [§2.1]
    if random() < 0.10:
        image_embedding = zeros_or_learned_vector()
    # Drop the caption 50% of the time  [§2.1]
    if random() < 0.50:
        caption = None
    return image, caption, image_embedding

# Diffusion prior, one training step  [§2.2] [app. C]
# Input:  caption, text_embedding (CLIP), image_embedding (CLIP, (1024,))
# Output: scalar loss

def prior_training_step(caption, text_embedding, image_embedding):
    # Scale the image embedding by 17.2 for the diffusion prior  [app. C]
    image_embedding = 17.2 * image_embedding
    # Drop the text conditioning 10% of the time  [§2.2]
    if random() < 0.10:
        caption, text_embedding = None, None
    # Sample a timestep t and noise the embedding to step t  [§2.2]
    t = uniform_int(1, T)
    noisy = noise_embedding(image_embedding, t)
    # Predict the clean embedding, not the noise  [§2.2]
    predicted = prior_model(noisy, t, caption, text_embedding)
    # Mean-squared error against the clean embedding  [§2.2]
    return mean((predicted - image_embedding) ** 2)

# Autoregressive prior: CLIP embedding to discrete tokens  [§2.2]
# Input:  image_embedding (1024,), pca (fitted on training embeddings)
# Output: 319 integer tokens, each in 0..1023

def to_tokens(image_embedding, pca):
    # Keep the first 319 principal components, in decreasing order  [§2.2]
    coefficients = pca.transform(image_embedding)[:319]    # (319,)
    # Quantise each component into one of 1024 buckets  [§2.2]
    return quantise(coefficients, buckets=1024)            # (319,)

Image manipulations §3. Each image x is encoded as a pair (z_i, x_T): z_i from the CLIP image encoder, and x_T from DDIM inversion of x with the decoder, conditioned on z_i §3.

# Variation of an image  [§3] [§3.1]
# Input:  image x, DDIM stochasticity eta > 0 (value not stated)
# Output: a new image with the same content

def make_variation(x, eta):
    # Encode the semantic content with CLIP  [§3]
    z_i = clip_image_encoder(x)
    # Invert x with the decoder, conditioned on z_i  [§3]
    x_T = ddim_invert(decoder, x, condition=z_i)
    # Sample with stochastic DDIM steps  [§3.1]
    return ddim_sample(decoder, noise=x_T, condition=z_i, eta=eta)

• Variations §3.1. Sampling with eta = 0 reconstructs x. Larger eta makes the variations vary more, and the variations stay centred on x §3.1 fig. 3.
• Interpolations §3.2. Spherical interpolation between the two CLIP image embeddings, with the DDIM noise either interpolated too, or fixed to one random value for the whole path. The authors use the fixed-noise option for their figure §3.2 fig. 4.
• Text diffs §3.3. The difference between a target text embedding and the embedding of a caption for the input image is normalised. The image embedding is rotated toward it with spherical interpolation, up to an angle typically between 0.25 and 0.50, with the base DDIM noise fixed §3.3 fig. 5.

# Text diff applied to an image embedding  [§3.3]
# Input:  z_i (image embedding), z_t (target text), z_t0 (caption of image)
# Output: CLIP embeddings to decode, one per step

def text_diff_path(z_i, z_t, z_t0, max_theta, steps):
    # Normalised difference of the two text embeddings  [§3.3]
    z_d = normalise(z_t - z_t0)
    # Rotate from the image embedding towards the text direction  [§3.3]
    for theta in linspace(0.0, max_theta, steps):
        # Number of steps is not stated  [§3.3]
        yield slerp(z_i, z_d, theta)

Probing §4. The decoder is used to inspect what the CLIP embedding keeps. Typographic attacks, where a text label is written over an object, do not stop the decoder from drawing the original object §4 fig. 6. PCA reconstructions with 20 to 320 dimensions show coarse semantics first and fine shape later §4 fig. 7.

Hyperparameters

Table 3 tab. 3 lists the settings for each model. These are the paper's research models. The production model differs in architecture, training length and data app. C.

Full settings for every model, as given in the paper's table of hyperparameters tab. 3 (blank cells are "-" in the source):

Results

Author-reported. Setup: text-to-image on MS-COCO for FID and on the authors' prompt sets for human evaluation. Human evaluations are against GLIDE §5.2.

Human evaluation against GLIDE tab. 1. Each value is the probability that the unCLIP model beats GLIDE. Values under 50% mean GLIDE is preferred.

Zero-shot FID on MS-COCO 256×256 tab. 2. Lower is better.

Small-model comparison §5.1. On a test set, FID was 9.16 for a small decoder that takes CLIP text embeddings, 7.99 for a small unCLIP stack, and 16.55 for feeding text embeddings to the unCLIP decoder zero-shot §5.1. Humans preferred the full unCLIP stack over the text-embedding decoder 57.0% ± 3.1% for photorealism and 53.1% ± 3.1% for caption similarity §5.1.

Diversity and guidance §5.3. The authors show that unCLIP keeps its content as decoder guidance rises, while GLIDE's semantics converge fig. 9. At the high guidance scales used for photorealistic output, unCLIP gives more diversity at similar photorealism and caption similarity fig. 10. FID rises less with guidance for unCLIP than for GLIDE fig. 11 §5.3.

Aesthetic quality §5.5. Guidance improves the mean aesthetic score for both GLIDE and unCLIP, and unCLIP keeps its recall under guidance fig. 13. The values are shown only in the figure.

Production model. Samples from the production model are shown in fig. 1, fig. 12 and fig. 18–fig. 20. The paper gives no numbers for it.

Paper vs code

No official code.

Searched: the paper's LaTeX source and the arXiv metadata contain no code or project link. A GitHub repository search for "unclip" under the openai organisation returned no results. A search for "dalle-2" under openai found openai/dalle-2-preview. Its top-level content is an assets folder and two system-card documents, with no model or training code, and the paper does not link it, so it was not used. A search for "unCLIP dalle2" found only third-party reimplementations (e.g. LiYinqi/un2CLIP), which were not used.

Pitfalls

• Conflicting CLIP-embedding dropout. §2.1 says the CLIP embedding is zeroed 10% of the time during decoder training, but §5.1 says it is dropped 5% of the time, and the fig. 8 caption repeats 5% for the text-to-image-only case. Pick one and document it. The source does not explain the difference.
• Equation numbers. The equations in the LaTeX source are unnumbered, so this card cites them by section. Check the compiled PDF before citing an equation number.
• Figure numbers. Figures are numbered in source order. Float placement in the compiled PDF may shift them. Figs 14–17 cover the limitations text and fig. 18–20 are the appendix samples.
• Reproduction requires training CLIP. The image encoder is trained by the authors on about 650M images, and the paper does not state that its weights are released. Using public CLIP weights would change the embedding space, and the paper does not evaluate that.
• Data. The paper names its training sets (CLIP, DALL-E) but not their filtering. The production model uses a larger set filtered for aesthetics and safety app. C.
• Diffusion prior scaling. The CLIP embedding is scaled by 17.2 for the diffusion prior app. C. The paper does not say whether the scale is undone before decoding, and it does not say how the MSE in §2.2 is normalised. Check against the output scale before using the loss.
• Embedding dimension. The PCA step reduces 1,024 dimensions to 319 §2.2. The paper does not say how quantisation bounds are set.
• Dot-product conditioning. The 50th-percentile choice is for the AR prior only §2.2. The sampling count of 2 is for the diffusion prior only.
• Guidance scales. The paper sweeps guidance scales for human evaluation, but gives only the FID value (1.25) in tab. 2. Do not treat 1.25 as the optimal scale for other goals.
• Decoder text path. The decoder keeps GLIDE's text conditioning, which the authors find helps little §2.1 §7.
• Variation noise. The value of eta used for variations is not stated §3.1. The text diff step count is also not stated §3.3.
• Compute. Sampling uses 250 decoder steps and 27 or 15 steps per upsampler, on a 3.5B-parameter decoder tab. 3. The paper gives no hardware or wall-clock figures.
• Known failures. The model binds attributes to objects poorly, renders text poorly, and lacks detail in complex scenes §7 fig. 14–fig. 17. The authors link the binding and text problems to the CLIP embedding and to BPE tokens §7.
• Risks. §7 discusses misuse and refers to a separate risk analysis of DALL·E 2 Preview that is not in this wiki.
• Annotation of tab. 1. The table gives the probability that unCLIP beats GLIDE, not GLIDE's own score. Read values under 50% as GLIDE being preferred.

Lineage

• Builds on: CLIP, arXiv:2103.00020, for the embedding space and the text encoder §2 app. C. GLIDE, arXiv:2112.10741, for the decoder architecture and text pathway §2.1 app. C. Diffusion models and guidance: Dhariwal and Nichol, arXiv:2105.05233, for ADMNet and classifier guidance §2.1 app. C. Improved DDPM, arXiv:2102.09672, for learned variance and sampling app. C. DDIM, arXiv:2010.02502, for image manipulations §3 app. C. Analytic-DPM, arXiv:2201.06503, for the diffusion prior sampler app. C. SR3, arXiv:2104.07636, for the upsampler degradation §2.1. Latent diffusion, arXiv:2112.10752, for crop training and the BSR degradation §2.1 app. C. SAM, arXiv:2010.01412, for the CLIP training optimiser app. C.
• Compared against: GLIDE §5.2 §5.3, DALL-E, arXiv:2102.12092 §5.4, LAFITE, arXiv:2111.13792 §5.4, and Make-A-Scene and GAN baselines tab. 2. The paper also cites CLIP-GEN, arXiv:2203.00386, for the text-embedding-as-image approach §5.1.
• The ids above come from the paper's bibliography. None of these works has a card in this wiki yet.