VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

In plain words

This paper shows how to make a picture from a sentence, or to change a picture by describing the change in words. It trains no new image model. It reuses two pretrained parts: a generator that builds a picture from a grid of learned codes (VQGAN), and a model that scores how well a picture fits a sentence (CLIP).

The process starts from random noise, or from a photo you want to edit. It then adjusts the codes step by step so that CLIP rates the decoded picture as a better match for the text. CLIP's signal is noisy on a single picture, so each step scores many randomly cropped and tweaked copies of it and averages the result.

Generating and editing use the same procedure; only the starting point differs. Because nothing is trained, a run is a short optimisation loop, and the authors report that it runs on modest hardware, including a free notebook service.

TL;DR

• Text-to-image generation and text-guided editing without training, by combining a pretrained VQGAN decoder with a pretrained CLIP encoder §1 §2.
• The latent grid is optimised with Adam. The loss is a squared spherical distance between CLIP embeddings of augmented crops and the prompt §2 §2.3.
• Editing differs from generation only in the starting latent: an encoded input image instead of random codes §2.
• Author-reported: on 12 prompts, human raters gave a mean of 4.6 out of 5, against 3.3 for the best baseline. Baselines were best-of-five and this method was best-of-one tab. 1 §3.2.
• The official code is MIT-licensed. Several of its defaults differ from the values the paper states (see Paper vs code).

Problem

Generating and editing images from open-domain text had required expensive, specially trained models §1. Open-Edit allowed only semantically simple edits and no generation. DALL-E and GLIDE generate from text but do not edit. All three train their models from scratch §1.

The method assumes a pretrained image generator and a pretrained joint image-text encoder are available, and that combining them needs no further training §1.

Method

The generator is VQGAN, a convolutional autoencoder with a discrete codebook. Its decoder maps a latent grid of codebook vectors to an image §2.1. The encoder is CLIP. It embeds text and images separately, and the cosine similarity between the two embeddings is turned into a loss §2.2.

The loss is differentiated through CLIP and the decoder down to the latent grid, which the paper calls the z-vector §2. Forward, each latent vector is replaced by its nearest codebook entry. Backward, the gradient is passed straight through to the continuous latent §2.1 grad.py.

Each optimisation step decodes the latent, takes many random crops, augments them, and embeds them with CLIP. Averaging over crops reduces the noise in each update §2.3. An optional L2 penalty on the latent, applied only in the regularised variant, is described in §2.4. Multiple prompts are handled by adding their losses §2.5 main.py. Masking is described in the appendix app. G.2.

1. Initialise the latent grid with random codebook vectors, or encode the input image for editing §2 §2.1.
2. At each iteration, decode the quantised latent to an image §2.1.
3. Take random crops of that image, augment them and add Gaussian noise §2.3.
4. Embed the crops with CLIP's image encoder. Embed each prompt once with its text encoder §2.2.
5. For each prompt, compute the squared spherical distance to the crop embeddings, scaled by the prompt weight. Average over crops §2 main.py.
6. Add the optional regulariser and sum all terms §2.4 §2.5.
7. Take one Adam step on the latent §2.
8. Repeat for a fixed number of iterations §2.

Key equations. The paper's displays are not numbered, so they are cited by section.

z_q = \argmin_{z_k \in \mathcal{Z}} \|z_{i,j} - z_k\| §2.1, unnumbered display

\mathit{Loss}=\mathit{L}_\mathit{CLIP}+\alpha\cdot \frac{1}{N}\sum_{i=0}^{N}Z_{i}^{2} §2.4, unnumbered display; α and N are not given numerically

The paper names a "squared spherical distance" but gives no equation for it. The code computes, for normalised image embedding x and text embedding y, d = 2\arcsin^2(\lVert x - y\rVert / 2) main.py.

Pseudocode below is written for this card from §2 and the code. Code paths are relative to the repository at code_commit.

# VQGAN+CLIP latent optimisation  [§2]
# Input:  prompts as (text, weight, stop); image size (S, S)
# Output: image (3, S, S), values in [0, 1]

def generate(prompts, steps):
    # Start from random codebook vectors on a latent grid. Editing
    # encodes an input image here instead  [§2] [§2.1]
    # [code: main.py#L299-L306]
    latent = random_codebook_grid(S // 16, S // 16)   # (1, D, H, W)
    latent.requires_grad = True
    # CLIP text embeddings stay fixed, so encode each prompt once
    # [§2.2] [code: main.py#L317-L321]
    targets = [
        (clip_text_embed(text), weight, stop)
        for text, weight, stop in prompts
    ]
    # Adam with learning rate 0.15 in the paper  [§2]
    # [code: main.py#L77] [code: optimizers.py#L8-L10]
    optimiser = Adam([latent], lr=0.15)
    for step in range(steps):
        # Snap each latent vector to its nearest codebook entry; the
        # gradient passes straight through  [§2.1]
        # [code: grad.py#L58-L62]
        image = decoder(nearest_codebook(latent))     # (1, 3, S, S)
        # Many random crops, each augmented, then CLIP-normalised  [§2.3]
        # [code: masking.py#L146-L202]
        crops = augment(random_crops(image, num_crops))   # (N, 3, C, C)
        embeds = clip_image_embed(normalise(crops))
        # Per prompt: squared spherical distance to the crops, scaled by
        # the prompt weight  [§2]  [code: main.py#L145-L149]
        losses = []
        for target, weight, stop in targets:
            losses.append(spherical_loss(embeds, target, weight, stop))
        # Optional L2 penalty that pulls the latent toward zero  [§2.4]
        # (the decay schedule is not the paper's; see Paper vs code)
        losses.append(latent_penalty(latent, step))
        loss = sum(losses)
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()
        # Keep every latent channel inside the codebook's min/max range
        # [code: main.py#L236-L237]
        clamp_to_codebook_range(latent)
    return decoder(nearest_codebook(latent))

Hyperparameters

Values marked "paper" come from the text. "Code" values are the CLI defaults in main.py or the settings in the notebooks. Where the paper is silent, the card says so.

Results

Author-reported. Setup: a human study on 12 prompts (A to L), rated 1 (worst) to 5 (best). Number of raters not stated. Baselines are minDALL-E (1.3 B parameters) and GLIDE in two variants (783 M without CLIP guidance, 941 M with CLIP guidance) §3.2. Hardware for runtime: K80, P100 and V100 GPUs, 10 runs per cell tab. 2.

Paper vs code

The official repository is EleutherAI/vqgan-clip at e4c78e72e9743182fc0584a318842a5011738727. The abstract links it. The paper names EleutherAI-affiliated authors, and the repository's MIT license is copyright EleutherAI LICENSE.

• Learning rate. Paper: 0.15 §2. CLI default: 0.1 main.py. MSE notebook: 0.95 VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb. z+quantize notebook: 0.05 VQGAN+CLIP_(z+quantize_method).ipynb. The only 0.15 in the repository is in a style-transfer notebook, not the main experiments Semantic_Style_Transfer_with_CLIP+VQGAN_(Gumbel_VQGAN).ipynb.
• Iterations. Paper: 400 §2. CLI default: 500 main.py. Notebooks have no paper-matching cap; the MSE notebook sets max_itter = -1 VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb.
• Initial latent. Paper: "random pixel values" §2. Code: random codebook vectors, so the initial image is a random sample from the codebook, not noise pixels main.py.
• Augmentations. Paper lists flips, affine, perspective, colour jitter and Gaussian noise §2.3. CLI default set is Hf, Af, Pe, Ji, Er, which includes random erasing Er, not described in the paper main.py masking.py. Under the default latest cut method, Hf (horizontal flip) has no branch, so it is silently skipped masking.py. The Gn Gaussian-noise augmentation exists but is not in the default set masking.py. Separately, uniform-scale Gaussian noise is always added after augmentation masking.py.
• Regularisation formula. Paper: a mean over N of Z_i², decayed "by 0.005" §2.4. MSE notebook: F.mse_loss(z, z_orig) * mse_weight / 2, which has a factor 1/2 the paper does not show VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb. Its weight decreases linearly by a fixed step, not by 0.005 VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb. The CLI uses 1/(2i+1) decay, but only if the weight is set, and the default is 0 main.py main.py.
• Latent clamping. The paper does not mention it. The CLI clamps the latent to the codebook's per-channel min and max after each step main.py. The MSE notebook does not clamp VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb.
• Crop sampling. Paper: random crops, no distribution given §2.3. CLI default: a normal distribution around 0.8 of the shorter side masking.py. MSE notebook: uniform VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb.
• Masking. The paper describes masking by zeroing gradients, plus a self-masking method §2.5 app. G.2. The CLI does not call any masking code. vqgan_clip/masking.py holds the cutout code plus helper functions for self-masking, and main.py does not use them main.py.

Pitfalls

• Unnumbered equations. The paper's display equations are unnumbered, so cite them by section. The paper gives no alpha value, no N, and no numeric decay §2.4.
• The regularised variant is a different run. Regularisation is off in the CLI default. It is only in the MSE notebook, with its own schedule and learning rate main.py VQGAN+CLIP_(MSE regularized z+quantize_method).ipynb. An implementation based only on the paper's text will not match either.
• Cut count drives cost. The loss averages over the crops main.py masking.py. Each crop is an extra CLIP forward and backward pass, so run time and memory grow with the number of crops (inferred from the code). The paper does not state the number of crops it used.
• Prompt losses are summed without normalisation. Each prompt term is a mean over crops, scaled by its weight, and the terms are added main.py main.py. Adding prompts raises the total loss, and the relative weights set how much each prompt counts.
• Negative and stop values. A negative weight flips the sign of the distance, which pushes the image away from that text main.py. A stop value is applied with replace_grad, so the gradient is zero once the distance is below the stop threshold main.py grad.py. This is derived from the code and not described in the paper.
• Straight-through hides the snapping. The optimiser moves continuous latent values, but the image is built from nearest codebook entries grad.py. Inferred from the code: small latent changes can leave the decoded image unchanged until a value crosses a boundary.
• Clamping changes the optimisation. The CLI clamps the latent after every step main.py, which the paper does not describe. Removing it changes the trajectory.
• Break-even numbers conflict. The text gives 19,200 V100-hours for GLIDE training §6.2. The break-even discussion and the Table 2 caption give 20,200 §6.2 tab. 2.
• Hardware figures conflict. The text gives a runtime for an RTX 2080 Ti §6. Table 2 has K80, P100 and V100 rows and no 2080 Ti row tab. 2.
• Human study is small and selected. The study uses 12 prompts tab. 1. Baselines were picked best-of-five and this method was not §3.2. The paper does not report the number of raters or any variance.
• Adoption figures are not measured. The counts of NFTs and of uses on commercial sites come from the authors' observation §7.

Lineage

• Builds on: VQGAN [esser2021taming] and CLIP [Radford2021LearningTV], both as cited in the paper's bibliography. The paper's bibliography does not give arXiv ids for them, so none are recorded here. Adam is cited as [kingma2014adam].
• Compared against: minDALL-E [kakaobrain2021minDALL-E], GLIDE [arXiv:2112.10741] and Open-Edit [liu2020open].
• Superseded by: not stated in this wiki.