LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

In plain words

Image generators often get the details of a request wrong, such as the number of objects or where they sit in the picture. One way to fix this is to tell the generator where each object goes. Drawing those boxes by hand is slow, though, so the paper asks whether a language model can draw them from the sentence alone.

The method, LayoutGPT, shows the language model a few solved examples first. Each example lists the objects in a page-layout style notation, one line per object, with its width, height and position. The examples are picked because they look like the new request. The model then continues the pattern for the new sentence. No training is needed; the model only sees the examples in its prompt.

The same idea works for a 2D picture and for the furniture in a room. A second model draws the picture from the boxes. The reason it matters is that a text model can take over the planning step, so users need not draw the boxes.

TL;DR

LayoutGPT is a training-free prompt method. It writes each object as a CSS-style rule (category as selector, size and position as declarations), prompts a GPT-family model with retrieved examples, and parses the reply back into boxes. With a layout-to-image model, the authors report better counting and spatial accuracy than end-to-end text-to-image models on their NSR-1K benchmark. For 3D bedrooms they report fewer out-of-bounds placements than the supervised ATISS baseline. All headline numbers are author-reported.

Problem

Text-to-image models often fail on object counts, spatial relations and attributes §1. Layout-guided generators can fix this, but a user must then supply boxes by hand §1. Autoregressive layout models usually treat a layout as a flat list of numbers, which is hard for an LLM to interpret §3.2. For 3D indoor scenes, supervised models must keep furniture inside the floor plan §1, §5.1.

Assumptions: the method needs an LLM that accepts long completion prompts and, for 2D retrieval, CLIP features of the demonstrations §3.3. The 2D numbers depend on a layout-to-image model (GLIGEN in the main results) and GLIP for evaluation §4.1.

Method

Overview. The task maps a condition to a set of objects. In 2D, a condition is a text prompt and each object has a category, a top-left position and a size §3.1. In 3D, the condition is the room type and room size, and each object has a category, a 3D location, a 3D size and an orientation §3.1.

Prompt parts. A LayoutGPT prompt has three parts: task instructions, in-context exemplars written as CSS, and normalized values §3.2 fig. 2. The exact instruction text used for 2D and 3D is in app. A.

1. CSS structure. Each object is one rule. The category is the selector, and the position and size are declarations (height, width, top, left in 2D; the 3D version adds length, depth and orientation) §3.2 fig. 2 app. A.
2. Task instructions. The instruction names the task, the format of each line, the unit and the canvas size §3.2 app. A. The 3D instruction also lists the available furniture and their overall frequencies app. A.
3. Normalization. Values are rescaled so the largest canvas side maps to 256 px §3.2. Integers are used in the exemplars app. B run_layoutgpt_2d.py.
4. Exemplar retrieval. Demonstrations are ranked by similarity to the query and the top k are used §3.3. In 2D the score is the cosine between a CLIP text feature of the query caption and a CLIP image feature of each demonstration §3.3. In 3D the score is a squared difference of room length and width §3.3 (see Paper vs code for what the code does).
5. Ordering. The retrieved exemplars are put in reverse order, so the most similar one comes last, right before the query §3.3.
6. Generation. The prompt is sent to the LLM with the sampling settings in app. A. Five layouts are sampled per prompt for the 2D evaluation app. A.
7. Parsing. Each generated line is turned back into a category and a box. Invalid boxes are dropped or zeroed, see Pitfalls utils.py.
8. 2D rendering. The layout is passed to a layout-to-image model, GLIGEN in the main table or a training-free method in the ablation §4.1 tab. 4.
9. 3D rendering. For each predicted object, a 3D model is retrieved by category, location, orientation and size, following ATISS, and the scene is rendered §3.4 §5.1.

Spatial relation (2D evaluation). The benchmark assigns one of four relations from the angle between two box centres, dividing the circle into four sectors around the reference object app. B.1 eq. 1:

\text{Rel}(A,B) = \begin{cases} B \text{ above } A & \text{if } \frac{y_B-y_A}{d} \geq \sin(\pi/4) \\ B \text{ below } A & \text{if } \frac{y_B-y_A}{d} \leq \sin(-\pi/4) \\ B \text{ on the left of } A & \text{if } \frac{x_B-x_A}{d} < \cos(3\pi/4) \\ B \text{ on the right of } A & \text{if } \frac{x_B-x_A}{d} > \cos(\pi/4) \end{cases} app. B.1 eq. 1

Here d is the Euclidean distance between the two centres app. B.1.

3D scene difference (retrieval analysis). The paper measures how far a generated room is from its closest demonstration with the sum, over generated objects, of L1 distances in location and size to the nearest same-category object app. C.1 eq. 2:

D(S^t|S^g) = \sum_{i=1}^{n} \min_{j,\ c^t_j = c^g_i} \left( \lVert t^t_j - t^g_i \rVert_1 + \lVert s^t_j - s^g_i \rVert_1 \right) app. C.1 eq. 2

Locations and sizes are measured in meters app. C.1. Thresholds of 1.0 and 6.0 separate duplication, modification and generation in that analysis app. C.1.

Prompt construction and parsing, as pseudocode. The paper has no algorithm block; the code below follows §3.2 to §3.3 and app. A, with the code cited.

# Prompt for one query  [§3.2] [§3.3] [app. A]
# Input:  query (text); demos (list of caption and objects); k (int)
# Output: prompt (text) for the LLM

def build_prompt(query, demos, k, canvas_size):
    # Task instruction: task, CSS format and canvas size  [§3.2] [app. A]
    header = instruction_text(canvas_size)
    # Keep the k demos whose features are closest to the query  [§3.3]
    # Sorted with the most similar demo first
    neighbours = top_k_similar(query, demos, k)
    # Most similar demo goes last, next to the query  [§3.3]
    exemplars = list(reversed(neighbours))
    prompt = header
    for demo in exemplars:
        # One block per demo: caption, then one CSS rule per object  [§3.2]
        prompt += f"\nPrompt: {demo.caption}\nLayout:\n"
        for category, box in demo.objects:
            # Box is (x, y, w, h) in [0, 1], scaled to integer pixels
            left, top, width, height = to_pixels(box, canvas_size)
            prompt += css_rule(category, height, width, top, left)
    # Query block is left open for the model to complete  [§3.2]
    prompt += f"\nPrompt: {query}\nLayout:\n"
    return prompt


# Scale a normalized box to integer pixels  [§3.2] [code: run_layoutgpt_2d.py#L74]
def to_pixels(box, canvas_size):
    return [int(value * canvas_size) for value in box]

# Read one generated line back into (category, box)
# Output box is (x1, y1, x2, y2) in [0, 1]  [code: utils.py#L112-L115]

def parse_line(line, canvas_size):
    # Category is the text before " {", without digits  [code: utils.py#L82]
    category = strip_digits(text_before(line, " {"))
    # Declarations as integers, with "px" removed  [code: utils.py#L101-L105]
    values = read_declarations(line)
    # Missing keys: the object is kept with a zero box  [code: utils.py#L107-L110]
    if set(values) != {"height", "left", "top", "width"}:
        return category, [0, 0, 0, 0]
    # Far corner is clamped to the canvas  [code: utils.py#L112]
    x1 = values["left"]
    y1 = values["top"]
    x2 = min(values["left"] + values["width"], canvas_size)
    y2 = min(values["top"] + values["height"], canvas_size)
    # Start outside the canvas: the line is dropped  [code: utils.py#L113-L114]
    if x1 >= canvas_size or y1 >= canvas_size:
        return None, None
    return category, [v / canvas_size for v in (x1, y1, x2, y2)]

# Numerical scores for one prompt  [app. B.2]
# gt and pred map each category to its count

def count_scores(gt, pred):
    # Overlap: per ground-truth category, the smaller of the two counts
    overlap = sum(min(gt[c], pred.get(c, 0)) for c in gt)
    # Precision over predicted objects, recall over ground-truth objects
    precision = overlap / sum(pred.values())
    recall = overlap / sum(gt.values())
    # Accuracy is 1 only if categories and counts match exactly  [app. B.2]
    # (the comparison subtask uses its own rule, not shown here)
    accuracy = float(gt == pred)
    return precision, recall, accuracy

# Per-room scale used with --normalize  [code: run_layoutgpt_3d.py#L62-L68]
# Scale factor for every size and position of one room

def room_scale(room_length, room_width):
    # The shorter room side maps to 256 px  [code: run_layoutgpt_3d.py#L63-L67]
    # Paper: one fixed scalar per prompt [§3.2]. Code: this value per room.
    shorter_side = min(room_length, room_width)
    return 256 / shorter_side

Hyperparameters

Training-free: no learning rate or training schedule applies. Settings below are the paper's unless the Notes say otherwise.

Results

Author-reported. Setup: NSR-1K for 2D (prompts from MSCOCO captions and templates, with ground-truth boxes) §4.1 app. B.1; 3D-FRONT-derived bedroom and living room splits following ATISS §5.1. Image metrics use GLIGEN for rendering and GLIP for detection §4.1 tab. 2.

2D, main comparison tab. 2:

Rows 5 to 11 use GLIGEN for image generation tab. 2. The text says LayoutGPT beats the end-to-end models by 20 to 40% on GLIP-based accuracy and by 1 to 6% relative on CLIP similarity §4.2. Layout accuracy is close to human layouts on both tasks: GPT-3.5 numerical 86.33 vs 92.56, GPT-4 spatial 91.73 vs 91.17 §4.2 tab. 2.

Attribute binding, with image generators tab. 3:

Colour binding accuracy in percent, on HRS-Bench prompts tab. 3. The text reports 100% binding at the layout level on HRS colour prompts, without a table §4.2.

2D ablation, GPT-3.5 on spatial prompts tab. 4:

Row 8 equals the GPT-3.5 spatial row in tab. 2. Row 9 is the paper's test of model-agnosticism §4.2. The text names CSS and instructions as the most important parts for 2D §4.2.

2D further results (appendix): For template-based numerical prompts the paper reports about 86% for single-category prompts, 66% accuracy for two-category prompts and about 78% for comparison prompts app. B. On HRS size-comparison prompts, accuracy is 98.0%, 93.1% and 92.1% for 2, 3 and 4 objects, against a best of 31.1%, 0.2% and 0% for nine text-to-image models quoted by the paper app. B.

3D, bedroom and living room tab. 5:

OOB is the share of scenes with furniture outside the floor plan (%), KL is the furniture-category KL divergence, and FID is over four rendered views per scene §5.1 tab. 5. The text says ATISS has the best KL §5.2. The paper attributes the 3D gains to the normalization and CSS format §5.4.

3D ablation, bedroom split tab. 6:

The paper's conclusion for 3D is that normalization matters most for keeping furniture in bounds, and that CSS helps most in combination with it §5.4.

Other reported results: Keypoint planning with GPT-3.5 and GLIGEN is a preliminary demonstration, with no metrics app. D. Random in-context exemplars change 2D accuracy only slightly app. B, but raise 3D out-of-bound to 85.58% for bedroom app. C.1.

Paper vs code

Code: official repository weixi-feng/LayoutGPT at commit fc31954962553e5b65bf267a904a6930d50b1f5e, MIT LICENSE. It is the authors' own repository: the paper's author block links it, and its owner is the first author.

• 2D exemplar count. Paper: 16 for numerical, 8 for spatial app. A. Code: one default K = 8 for every setting run_layoutgpt_2d.py. The README counting command passes --K 8 README.md. The code does not reproduce the 16-exemplar numerical setting by default; pass --K 16 for that.
• 3D exemplar count for living rooms. Paper: 8 for bedrooms, 4 for living rooms app. A. Code: one K for both rooms run_layoutgpt_3d.py and top_k = args.K with no room override run_layoutgpt_3d.py. Check the living-room setting separately.
• 3D retrieval. Paper: distance is a squared difference of room length and width §3.3. Code: by default the distance is a mean squared difference between 64 by 64 floor-plan images run_layoutgpt_3d.py run_layoutgpt_3d.py run_layoutgpt_3d.py run_layoutgpt_3d.py. The length and width variant exists only behind the floor_plan=False branch, which is not called by default.
• 3D normalization. Paper: one fixed scalar, with the largest side at 256 px §3.2. Code: each room is scaled by its own shorter side, which is mapped to 256 run_layoutgpt_3d.py run_layoutgpt_3d.py.
• 3D rectangular-only restriction. Paper: only test-set floor plans are restricted to rectangles §5.1. Code with --regular_floor_plan: train and test both use the rectangular splits run_layoutgpt_3d.py run_layoutgpt_3d.py.
• 3D gpt-3.5-chat path. Paper reports GPT-3.5-chat for 3D tab. 5. Code: the chat call branch tests args.gpt_type in ['gpt3.5', 'gpt4'], so gpt3.5-chat falls through to NotImplementedError run_layoutgpt_3d.py. Treat the 3D GPT-3.5-chat row as not reproducible from this code without a fix.
• 2D canvas size. Paper: 256 px maximum §3.2. The instruction text in the appendix says 64 px app. A. Code default is 256 run_layoutgpt_2d.py, but the README output names use px_64 README.md. Use one value consistently and state which.
• Models. Paper: four GPT variants including Codex (code-davinci-002) app. A. Code: no Codex entry in the model map run_layoutgpt_2d.py.
• Retrieval encoder. Paper: CLIP, variant not stated §3.3. Code: ViT-L/14 run_layoutgpt_2d.py.
• Spatial relation. Paper: four sectors with boundaries at sin and cos of π/4, and the pair (A, B) fixed by the definition app. B.1 eq. 1. Code: boundaries at ±√2/2 on the normalized centre difference, plus a diagonal fallback utils.py, with an image-coordinate sign convention that the code does not document utils.py. Do not reuse the relation code without checking the argument order.
• Sampling and penalties. Paper: temperature 0.7, no penalties app. A. Code: 0.7 and zero penalties in both settings run_layoutgpt_2d.py run_layoutgpt_3d.py run_layoutgpt_3d.py. Matches.
• Chat prompts. Paper: in-context exemplars become multi-turn dialogue for chat models app. A. Code: matches, for 2D and 3D run_layoutgpt_2d.py run_layoutgpt_3d.py.
• Metrics. Paper defines precision, recall and accuracy app. B.2. The metric script (eval_counting_layout.py) was not read for this card; the formulas above are the paper's.
• Not checked. The 3D evaluation script (eval_scene_layout.py) and the GLIP evaluation were not read for this card.

Pitfalls

• Model access and variance. The results depend on API model snapshots named in app. A, and sampling is at temperature 0.7. The paper averages five samples per prompt app. A, but the tables show no variance or confidence intervals. Expect different numbers from current models; re-run before quoting.
• Rendering and detection. The image metrics depend on GLIGEN rendering and GLIP detection §4.1 tab. 2. The paper notes that the gap between layout accuracy and GLIP accuracy points to rendering and detection as the bottleneck §4.2. A layout that is right can still score low on image accuracy.
• Hard 2D cases. Two-category and comparison prompts are harder, with small boxes and GLIGEN struggling to fit them app. B, failure cases.
• Retrieval is essential in 3D. Random exemplars raise bedroom out-of-bound from 43.26% to 85.58% app. C.1. In 2D the effect is small app. B.
• Parser behaviour (2D). Missing declarations keep the object with a zero box utils.py. Boxes that start outside the canvas are dropped utils.py. Digits are stripped from category names, so chair2 becomes chair utils.py. Spaces in category names become hyphens utils.py.
• Corner versus size format. Exemplars use (left, top, width, height) run_layoutgpt_2d.py. The parser returns corner coordinates (left, top, right, bottom) utils.py. Do not feed parser output back as exemplars without converting it.
• Prompt budget. When the budget is reached, the loop stops and the least similar exemplars are left out run_layoutgpt_2d.py run_layoutgpt_3d.py. The budgets (3000 and 7000 tokens) are code defaults, not paper values.
• 2D retry bug. On an InvalidRequestError, the 2D script calls the prompt builder without the tokenizer argument, which fails run_layoutgpt_2d.py. Fix before relying on that path.
• Table rows mixed in the text. §5.2 gives living-room out-of-bound as 64.16%, which is the GPT-4 row (64.15% in tab. 5). It gives bedroom 43.26% and living FID 76.34, both GPT-3.5 rows §5.2 tab. 5. Use the table values.
• 3D data and licence. The 3D evaluation needs ATISS-processed 3D-FRONT and 3D-FUTURE data, downloaded separately README.md. The data licences are not checked for this card.
• Scene copying. The paper's own analysis places 34 of 423 bedroom scenes as near duplicates of an exemplar, 111 as modifications and 278 as generation, using distance thresholds 1.0 and 6.0 app. C.1. The thresholds are the authors' empirical choice.
• Reported claims without numbers. The 100% colour binding at layout level §4.2 has no table, and the keypoint planning result has no metrics app. D.

Lineage

• Builds on: retrieval of in-context examples for prompting, as in the Flamingo-style selection cited in §3.3. No card in this wiki for that work.
• Downstream rendering: GLIGEN [/papers/2301.07093] for the main 2D rows §4.1 tab. 2. Layout-Guidance, a training-free method, for the ablation tab. 4.
• Compared against: Stable Diffusion, Attend-and-Excite tab. 2; LayoutTransformer tab. 2 row 5; ATISS for 3D tab. 5. No card in this wiki for ATISS.
• Superseded by: not known from this wiki.