GLIGEN: Open-Set Grounded Text-to-Image Generation

In plain words

Text-to-image models draw whatever the caption says, but a caption is a poor way to say where things go. "A teddy bear and a bird" leaves open which one is on the left, how big each is, and whether the bird sits on the bear. GLIGEN lets you point at the answer: you give each phrase a rectangle, and the model draws that object inside it.

The trick is to avoid retraining the expensive picture model from scratch. The original model stays exactly as it was, with all its learned knowledge of objects and styles. GLIGEN adds a few new layers beside it. These layers read the rectangles and their phrases as extra hints. A gate on each new layer starts closed, so at the beginning the model behaves as the original did, and training slowly opens the gate as the new layers learn to use the hints.

Because the phrase is encoded by the same text encoder as the caption, the model can place objects it never saw with a box during training. The same mechanism takes other hints too, such as a reference picture of an object, a set of body keypoints, or an edge, depth or semantic map. A sampling option lets you use the hints only in the early steps, when the layout is decided, and then let the original model finish the details.

TL;DR

• GLIGEN adds grounding inputs (boxes first, plus keypoints, reference images and spatial maps) to a frozen pretrained latent diffusion model, using gated self-attention layers whose gate starts at zero §4.2.
• Phrases are encoded with the caption's text encoder and fused with their box, so the model can ground concepts not seen with boxes in training §4.1 §5.2.
• Author-reported: fine-tuned FID 5.6 to 6.4 on COCO2014 with boxes tab. 1; zero-shot LVIS GLIP-score 6.4 for a COCO-only model, above supervised LAMA at 2.0 tab. 3.
• Relevant if you need controllable layout in a diffusion model without retraining it; the main cost is the grounding data and the training run.

Problem

Text-only diffusion models cannot be told where an object goes, or which rectangle belongs to which phrase §1. Layout-to-image methods can take boxes, but they are trained from scratch and only handle a fixed set of categories (for example the 80 COCO classes) §1 §2. Retraining a large pretrained generator to accept boxes risks losing the concept knowledge it learned from web-scale data §1 §4.2. The paper therefore asks how to add new conditioning to a pretrained text-to-image model while keeping its weights §1.

Method

Inputs. An instruction is a caption c plus a list of grounded entities, each an entity e (text or image) and a spatial condition l (a box, keypoints or a map) §4.1 eq. 2 eq. 3 eq. 4. The paper's main setting is text entities with boxes §4.1. Keypoints use the same Fourier encoding per point; spatial maps are encoded by a small network into h x w tokens §4.1.

Grounding tokens. The caption goes through the frozen text encoder as usual §4.1. For each box-with-text entity, the phrase is encoded by the same text encoder, the box coordinates are Fourier-embedded, and an MLP maps the concatenation to one grounding token §4.1 eq. 5. Because the phrase lives in the same text space as the caption, the grounding is not restricted to a closed vocabulary §4.1. Image entities use an image encoder in place of the text encoder §4.1 app. A.

Gated self-attention. A pretrained Latent Diffusion Model (LDM) block is self-attention over visual tokens, then cross-attention from caption tokens §4.2 eq. 6 eq. 7. GLIGEN freezes both and inserts a new self-attention layer between them. That layer attends over the concatenation of visual and grounding tokens, keeps only the visual outputs, and adds them back through a gate beta * tanh(gamma), where gamma is a learned scalar initialized to 0 and beta is 1 during training §4.2 eq. 8 fig. 3. The gate starts at zero, so the first forward pass equals the pretrained model §4.2. The paper prefers this gated self-attention to gated cross-attention, which performs worse in its ablation app. B.

Training. All original weights are frozen. The new parameters are the grounding MLP and the gated layers. The objective is the original LDM denoising loss eq. 1 applied to the grounded model §4.2 eq. 9. The paper says the caption and grounding tokens are randomly dropped with 10% probability, which enables classifier-free guidance app. A.

Data. The paper trains on three box data types: grounding data (captions with boxes for noun phrases), detection data (COCO-style categories with a null caption), and detection plus caption data app. A. Scaling uses Objects365 and GoldG (Flickr and VG) and pseudo-labelled CC3M and SBU, with GLIP producing the pseudo-boxes §5.2.

Scheduled sampling. At inference the grounding is on (beta = 1) for the first tau * T denoising steps and off (beta = 0) afterwards, with tau in [0, 1] §4.2 eq. 10. The paper uses tau = 0.2 for its qualitative figure §5.4. Early steps decide the rough layout; later steps refine detail with the original model §4.2 §5.4.

# Grounding token for one entity  [§4.1] [eq. 5] [app. A]
# Input:  text_feature (768,)  CLIP text feature of the phrase
#         box (4,)             (x_min, y_min, x_max, y_max)
# Output: grounding_token (768,)

def grounding_token(text_feature, box):
    # Fourier-embed each of the four coordinates  [app. A]
    box_feature = fourier_embedding(box)          # (64,)
    # Concatenate the two inputs and run the MLP  [eq. 5] [app. A]
    joined = concat(text_feature, box_feature)    # (832,)
    return mlp(joined)                            # (768,)

# Gated self-attention, inserted between the frozen self-attention and
# cross-attention of each transformer block  [§4.2] [eq. 6-8] [app. A]
# Input:  visual (B, M, d), grounding (B, N, 768), gamma (learned, init 0)
# Output: visual (B, M, d)

def gated_self_attention(visual, grounding, gamma, beta):
    num_visual = visual.shape[1]                    # M
    # Project grounding tokens to the visual width d  [app. A]
    grounding = linear_projection(grounding)        # (B, N, d)
    # Joint self-attention over visual and grounding tokens  [eq. 8]
    joint = concat(visual, grounding, axis=1)       # (B, M + N, d)
    attended = self_attention(joint)[:, :num_visual]
    # Gated residual; tanh(0) = 0 at initialization  [eq. 8]
    return visual + beta * tanh(gamma) * attended

# One training step for the grounding layers  [§4.2] [eq. 9] [app. A]
# Frozen: all original weights. Trained: grounding MLP and gated layers.
# Input:  latents (B, 4, H, W), captions, phrases and boxes per image

def training_step(latents, captions, phrases, boxes):
    # Sample Gaussian noise and a random timestep  [eq. 9]
    noise = sample_normal_like(latents)
    t = sample_timestep()
    noisy = add_noise(latents, noise, t)
    caption_tokens = encode_caption(captions)
    grounding = build_grounding_tokens(phrases, boxes)  # [§4.1]
    # Randomly drop caption and grounding with probability 0.1  [app. A]
    if random() < 0.1:
        caption_tokens = null_caption_tokens
    if random() < 0.1:
        grounding = null_grounding_tokens
    predicted = unet(noisy, t, caption_tokens, grounding)
    # Same denoising objective as the pretrained model  [eq. 9]
    return squared_error(predicted, noise)

# Scheduled sampling at inference  [§4.2] [eq. 10] [§5.4]
# tau in [0, 1]: fraction of denoising steps that use grounding

def grounding_weight(step, total_steps, tau):
    # beta = 1 for the first tau * T steps, then 0  [eq. 10]
    if step <= tau * total_steps:
        return 1.0
    return 0.0

Extensions use the same grounding path with a different encoder. For keypoints, one learned embedding per person ties the keypoints of that person together app. A. For spatial maps, a ConvNeXt-tiny backbone turns the map into 64 tokens, and the map is also concatenated to the UNet input, which the paper says speeds up training §4.1 app. A. Image-conditioned generation and image grounded inpainting are described in §5.3 and app. C.

Hyperparameters

Results

Author-reported. Setups differ between rows; each table's source is cited.

COCO2014 val, text-to-image (30K generated images; FID for generation, YOLO-v4 AP for grounding) tab. 1. Baseline numbers are copied from their papers, and LDM* is the authors' COCO-fine-tuned LDM.

COCO2017 val, layout-to-image tab. 2. Trained on COCO2017 detection annotations only §5.1.

LVIS validation, zero-shot, GLIP-score tab. 3. GLIGEN was trained on COCO only for the first row group; LAMA was trained on LVIS (fully supervised) with the official code §5.2.

Supplementary results.

• LVIS after fine-tuning on LVIS, GLIGEN-LDM from the largest data: AP 14.9, FID 6.25 app. E.
• COCO2017, pretrained then zero-shot: FID 27.03, AP 19.1; after fine-tuning: FID 21.58, AP 30.8 app. E.
• Text inpainting, YOLO AP by object size (1-3%, 5-10%, 30-50% of the image): LDM 25.9, 23.4, 14.6; GLIGEN-LDM 29.7, 30.9, 25.6 app. C.
• Keypoints on COCO2017: pix2pixHD FID 142.4, AP 15.8; GLIGEN without caption FID 31.02, AP 31.8; with caption FID 27.34, AP 31.5 app. D.
• Ablations on COCO2014CD: gated cross-attention gives FID 5.8 and AP 16.6, versus AP 21.7 for gated self-attention app. B. A pseudo-caption made of entity names raises FID from 5.61 to 7.40 for detection data app. B. Replacing the Fourier box embedding with a plain MLP keeps FID (5.82 vs 5.80) but drops AP from 21.7 to 3.2 app. B.

Scheduled sampling is shown qualitatively only §5.4 fig. 7.

Paper vs code

The code is the repository linked from the project page printed on the title page of the paper (https://gligen.github.io/); it is the code cloned at code_commit. Checked at that commit:

• Grounding MLP depth. The paper says three hidden layers app. A. The code has two hidden layers of width 512 with a final projection text_image_grounding_net.py.
• Feed-forward gate. Equation 8 gates only the attention output eq. 8. The code adds a second gated residual around the feed-forward block, with its own learnable scalar attention.py.
• Normalization. The code applies a LayerNorm to the concatenated tokens before the gated attention attention.py. Equation 8 has no normalization.
• Padding tokens. The code replaces padded grounding slots with learned null embeddings text_image_grounding_net.py text_image_grounding_net.py. The paper does not describe padding.
• Separate MLPs for text and image. The code runs separate MLPs for text and image entities, giving two tokens per box text_image_grounding_net.py. The paper does not state whether they share weights.
• Scheduled sampling. The paper's Equation 10 is a hard switch at tau * T. The code's schedule has three stages: grounding on, a linear decay from 1 to 0, then off gligen_inference.py. The example settings use a hard switch with [0.3, 0.0, 0.7] gligen_inference.py.
• Null-alpha step. At alpha 0 the sampler calls a method that restores the first conv layer to its Stable Diffusion weights ddim.py openaimodel.py. The paper does not describe this.
• Caption and token dropout. The paper says 10% app. A. The box config sets prob_use_caption: 0.5 GoldG+SBU+CC3M+O365_box_text_image.yaml. The dataset also drops either the text or the image feature per box with probability 0.5 tsv_dataset.py.
• Text feature for training. The box config comment calls the text feature a "penultimate" CLIP feature GoldG+SBU+CC3M+O365_box_text_image.yaml. Inference uses the pooler output (EOS) gligen_inference.py, which matches the paper app. A. The training-time feature extraction is not in this repository.
• Optimizer. The paper says Adam app. A. The code uses AdamW with weight decay defaulting to 0 trainer.py main.py.

Pitfalls

• Box coordinate convention. The paper gives the box as top-left and bottom-right coordinates §4.1 but does not say whether they are normalized or in pixels. The example inference settings use values in [0, 1] gligen_inference.py.
• Delayed grounding. The gate starts at zero and the paper reports the gated layer starts attending to the boxes only around 60k to 70k iterations, so short runs may show no grounding app. F.
• Magic constant for image tokens. Image embeddings are projected and rescaled to norm 28.7, the average text-feature norm the authors observed app. A. Using raw CLIP image embeddings gives different scales.
• Projected CLIP text features converge slowly. The paper reports slow convergence when it used the projected CLIP text embedding instead of the EOS feature app. A.
• Fourier embedding is essential. Replacing it with a plain MLP keeps FID but collapses box accuracy app. B. Do not drop it when implementing.
• Null captions for detection data. Using a null caption is a deliberate choice; a pseudo-caption of entity names is worse app. B.
• Maximum entities. Images with more than 30 entities are not described; the paper gives a maximum without saying what happens above it.
• Evaluation uses GLIP, which is also a data source. GLIP produces the pseudo-boxes for COCO2014G and CC3M and SBU §5.1 §5.2, and the LVIS GLIP-score uses GLIP as the detector §5.2 tab. 3. The metric and some training labels share one detector.
• Baseline numbers come from other papers. Table 1's baselines are copied from their papers tab. 1 caption. The paper does not say whether protocols match.
• Comparison is not like for like. LAMA is trained on LVIS, while GLIGEN is zero-shot on LVIS §5.2.
• Compute. Scaling runs use 500k iterations at batch 32 for Stable Diffusion app. A. Inference cost of the extra tokens is not stated.
• Inpainting needs extra input channels. The paper adds five trainable input channels for inpainting app. C. The code README says they are initialized to zero when converting a checkpoint README.md.
• Checkpoint license. The code README states that model checkpoints and the demo are restricted to uses that follow the Latent Diffusion and Stable Diffusion license terms README.md. This affects any use of released weights, not the paper's method.

Lineage

• Builds on: the pretrained latent diffusion model (LDM), with Stable Diffusion used for the larger runs §3 app. A. Its CLIP text encoder provides the phrase features app. A. Gated attention is adapted from Flamingo's gating idea, which uses cross-attention instead §4.2.
• Compared against: layout-to-image GANs (LostGAN-V2, LAMA, TwFA, OCGAN, HCSS) on COCO and LVIS §5.1 §5.2 tab. 2 tab. 3; text-to-image baselines in tab. 1; pix2pixHD for keypoints app. D.
• Concurrent work noted by the paper: ReCo, which fine-tunes the Stable Diffusion weights rather than freezing them §2.
• Superseded by: not stated in this wiki.