GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment

In plain words

Text-to-image models draw a picture from a sentence. The usual scores either judge overall picture quality or give one number for how well the picture matches the sentence, so they cannot say which part of the request went wrong.

This paper treats the prompt as a checklist instead. It separates the request into objects, how many of each, their colours, and where they sit relative to each other. A ready-made object detector looks for those objects in the picture, and a colour classifier checks the colours. Each picture gets a yes or no, and the reasons for a "no" are recorded. Averaged over six kinds of request, the scores show which skills a model is missing: counting, placing objects relative to each other, and keeping each colour on the right object are the hardest for the models tested.

TL;DR

GenEval is an automated benchmark for text-to-image models. Each prompt carries metadata (object classes, counts, colours, relative positions) that a COCO instance-segmentation model and a CLIP colour classifier check per image. The authors report that its verdicts agree with human annotators better than thresholded CLIPScore on compositional tasks. Author-reported: IF-XL scores 0.61 overall, and relative position and attribute binding stay low for all models. Use this card together with the Paper vs code section: the official code differs from the paper in a few places that change the numbers.

Problem

Automated text-to-image metrics measure either image quality, such as FID, or one holistic image-text alignment score, such as CLIPScore §1 §2. The authors argue that neither suits fine-grained, instance-level analysis of compositional prompts §1. Human preference comparisons are the gold standard but are costly to scale §1.

The approach rests on one assumption: the objects named in a prompt can be recognised by a detector trained on the 80 MS COCO classes §3.1 §6. The authors state that detectors trained on COCO cannot verify fine details, such as the number of fingers on a hand, and that they generalise poorly to artistic renderings §6 fig. 4.

Method

The authors build six prompt tasks from templates, generate images, and score each image with detection and colour-classification models.

1. Prompt generation §3.1 app. C.1. Each prompt is filled from a template with a random object name, colour, count or relative position. Object names come from the 80 MS COCO classes, with a few renamed to remove ambiguity. Colours are 10 basic colour terms, with gray left out. Counts are 2, 3 or 4. Relative positions are "left of", "right of", "above" and "below". The colour tasks exclude the "person" class. Each prompt is stored with metadata that the scorer uses app. C.1.
2. Instance detection §3.2 app. C.3. A Mask2Former model with a Swin-S backbone, trained on COCO for instance segmentation, returns boxes, classes, confidences and masks. Only detections above a confidence threshold are kept: 0.3 by default and 0.9 for the counting task §3.2.
3. Object presence §3.2. Each required class must have at least the requested number of detections. For the single-object and two-object tasks this is the whole score.
4. Counting §3.2. The number of detections must equal the requested count. The paper says "matches"; the code enforces this with a second clause that forbids count + 1 create_prompts.py (see Paper vs code).
5. Colour §3.2 app. C.3. Each detected object is cropped to its box, pixels outside its mask are replaced with a solid gray, and a zero-shot CLIP ViT-L/14 classifier picks one of 10 colours using three text templates.
6. Relative position §3.2 app. C.3. The relation between two objects is read from their box centres. A dead zone, proportional to the box sizes, stops overlapping or near-aligned objects from being called offset. The factor is c = 0.1.
7. Scoring §3.2 app. C.3. Each image gets a binary score: 1 if every element of the prompt is present, 0 otherwise. The score is averaged per task and then averaged over the six tasks. The scorer also writes the reason for each failure (missing object, wrong count, colour or position).

Per-image check, following the metadata clauses:

# Per-image check: are all elements of the prompt present?  [§3.2]
# Input:  image; metadata with "include" and "exclude" clauses [app. C.1]
# Output: correct (bool)

def score_image(image, metadata):
    # 1. Detect instances of the 80 COCO classes  [app. C.3]
    detections = detect_instances(image)
    # 2. Keep detections above 0.3, or above 0.9 for counting  [§3.2]
    threshold = 0.9 if metadata.tag == "counting" else 0.3
    objects = filter_by_confidence(detections, threshold)
    correct = True
    matched = []
    # 3. Include clauses: count, then colour, then position  [§3.2]
    for clause in metadata.include:
        found = most_confident(objects[clause.class], clause.count)
        ok = len(found) >= clause.count
        if ok and "color" in clause:
            colors = [classify_colour(image, obj) for obj in found]
            ok = colors.count(clause.color) >= clause.count
        if ok and "position" in clause:
            relation, target_index = clause.position
            targets = matched[target_index]
            ok = targets is not None and all(
                relation in relations(obj.box, t.box)
                for obj in found
                for t in targets
            )
        correct = correct and ok
        matched.append(found if ok else None)
    # 4. Exclude clauses: the class must stay below `count`  [app. C.1]
    for clause in metadata.exclude:
        if len(objects[clause.class]) >= clause.count:
            correct = False
    return correct

Colour of one object:

# Colour of one detected object  [§3.2] [app. C.3]
# Input:  image, object with box, mask and class name
# Output: one of the 10 candidate colours  [app. C.1]

def classify_colour(image, obj):
    # 1. Crop to the box, then grey out pixels outside the mask  [app. C.3]
    crop = crop_to_box(image, obj.box)
    crop = replace_outside_mask(crop, obj.mask, gray)
    # 2. Average the normalised text embeddings of three templates per colour
    #    [app. C.3]
    text = {}
    for color in COLORS:
        prompts = [template(color, obj.name) for template in TEMPLATES]
        text[color] = mean(normalize(encode_text(prompts)))
    # 3. Pick the colour whose embedding is closest to the crop  [app. C.3]
    crop_vector = encode_image(crop)
    return argmax(cosine(crop_vector, text))

Relative position with a dead zone:

# Relations of object A to object B with a dead zone  [app. C.3]
# Input:  boxes as (x0, y0, x1, y1); dead-zone factor c = 0.1 [app. C.3]
# Output: subset of {"left of", "right of", "above", "below"}

def relations(box_a, box_b, c):
    # 1. Centres (x, y) and sizes (w, h) of both boxes  [app. C.3]
    x_a, y_a, w_a, h_a = center_and_size(box_a)
    x_b, y_b, w_b, h_b = center_and_size(box_b)
    found = set()
    # 2. A is left of B if B's centre is right of A's by the dead zone
    #    [app. C.3]
    if x_b > x_a + c * (w_a + w_b):
        found.add("left of")
    # 3. A is right of B if B's centre is left of A's by the dead zone
    #    [app. C.3]
    if x_b < x_a - c * (w_a + w_b):
        found.add("right of")
    # 4. Image y grows downward: A is above B if B's centre is lower  [app. C.3]
    if y_b > y_a + c * (h_a + h_b):
        found.add("above")
    # 5. A is below B if B's centre is higher by the dead zone  [app. C.3]
    if y_b < y_a - c * (h_a + h_b):
        found.add("below")
    return found

Note that the appendix prints the "above" line with the wrong inequality sign; see Pitfalls.

Two proxy scores come from the same images. The CLIPScore baseline is the cosine similarity between the prompt and image embeddings, clipped at 0 and scaled to 0-100 app. C.3. The human study compares both with annotations §4.

Hyperparameters

Results

All numbers are author-reported. Setup: 553 prompts over six tasks, 4 images per prompt, default generation settings, scored with the detector and CLIP classifier above §5.1 tab. 1.

Tasks

tab. 1. The article "a" or "an" follows the next word's first letter app. C.1. Plurals are formed by appending "s" and are not always grammatical app. C.1.

Benchmark scores

Main models, with the paper's CLIPScore and human columns tab. 2:

Overall scores vary by 0.01-0.02 across random seeds tab. 2. The paper states that IF-XL has the best overall score and SD-XL the second, and that CLIPScore cannot separate SD-XL from IF-XL §5.2 tab. 2.

All evaluated versions app. A, tab. 3:

The paper's headline findings §1 §5.2 §7: recent models (IF-XL, SD-XL) improve on the hard tasks; counting, position and attribute binding still have much room for improvement; the best model generates 15% (position) and 35% (attribute binding) of images correctly. The position task is not monotonic with model size fig. 5. Within Stable Diffusion, v2 (new text encoder) gains clearly, and v1.1 to v1.5 does not improve consistently fig. 5.

Agreement with human judgment

• Overall, GenEval agrees with annotators on 83% of images, against 88% between annotators and 80% for thresholded CLIPScore §4.
• On 860 images where all five annotators agree, GenEval agrees 91% of the time, against 87% for CLIPScore §4.
• For counting, GenEval improves human agreement over CLIPScore by 22 points §4.
• Agreement with annotators by CLIP model used for CLIPScore tab. 5:

tab. 5 (percent agreement; the "ImageNet zero-shot" column is a CLIP property, not an agreement score). The paper says OpenCLIP ViT-H/14 is the best CLIPScore variant overall but is clearly worse on position app. C.3.

Ablations

Colour-classifier preprocessing, Cohen's kappa against annotators tab. 4:

Counting threshold: kappa rises from 0.37 at 0.3 to 0.65 at about 0.9 app. A.2. The position dead-zone factor c = 0.1 gives the best human agreement app. C.3 fig. 8.

Paper vs code

Official repository: djghosh13/geneval, MIT licence, commit af4902f, cloned for this card. It is linked from the paper's abstract and carries the first author's name README.md. The repo contains the prompt generator, the scorer, the summary script, a Stable Diffusion sample script, and annotation files.

Matches the paper: detection thresholds 0.3 and 0.9 evaluate_images.py; c = 0.1 evaluate_images.py; CLIP ViT-L/14 with OpenAI weights evaluate_images.py; the 10 colours and 3 templates evaluate_images.py evaluate_images.py; counts 2 to 4 create_prompts.py; the 553-prompt set evaluation_metadata.jsonl; the overall score as the mean of task scores summary_scores.py.

Differences:

1. Object cap. The appendix says that, except for counting, there is no upper limit on objects app. C.3. The code keeps at most 16 detections per class by default for every task evaluate_images.py evaluate_images.py. Counts up to 4 are checked, so the cap changes results only when many more objects are present.
2. Position rule. The paper's rule compares each axis against the dead zone c(size_A + size_B) app. C.3. The code subtracts the same dead zone, then divides each component by the length of the raw offset, and counts a relation only if that quotient exceeds 0.5 in magnitude evaluate_images.py. The code therefore drops relations the paper formula keeps. Example: two 40 x 40 px boxes whose centres are 100 px apart horizontally and 20 px apart vertically. Dead zone per axis = 0.1 x 80 = 8 px. The paper gives "left of" (100 > 8) and "below" (20 > 8). The code's vertical quotient is (20 - 8) / 102 = 0.12, below 0.5, so it gives "left of" only.
3. Counting. The paper says the detected count "matches" the requested number §3.2. The code implements this with an include clause (at least N) and an exclude clause (not N + 1) create_prompts.py evaluate_images.py. The result is the same exact-count check, but an implementer must add the exclude clause.
4. Most confident objects only. For colour and position checks, the code uses the most confident detections up to the requested count evaluate_images.py evaluate_images.py. The paper does not say this.
5. Background value. The paper says "solid gray" app. C.3; the code default is #999 evaluate_images.py.
6. Colour averaging. Averaging the normalised template embeddings is done by an external clip_benchmark function evaluate_images.py evaluate_images.py. That function is not in this repository, so this compile did not check it against the paper.
7. GPU only. The scorer asserts CUDA evaluate_images.py.
8. Generation defaults. The sample script uses Stable Diffusion v1.5 by default, 50 steps, guidance 9.0 and seed 42 diffusers_generate.py diffusers_generate.py. The paper does not state these settings for each model app. C.2. Treat them as the sample script's values, not the paper's.
9. Not checked. The repo ships annotation CSV files for SD v2.1, IF-XL and CLIP retrieval . This compile did not recompute the agreement numbers from them.

The paper's Table 2 does not say which CLIP model its CLIPScore column uses (the human comparison uses OpenCLIP ViT-H/14 §4).

Pitfalls

• Position formula printed wrongly. The appendix's "above" line reads y_B > y_A - c(h_A + h_B), which overlaps the "below" line and contradicts the rest of the rule app. C.3. The card's pseudocode uses the consistent form (B's centre must be lower than A's by the dead zone to call A "above"). Image y grows downward, so "above" means a smaller y.
• Direction of the relation. The prompt "A [position] B" asks for A relative to B, but the appendix's formulas are written as B relative to A app. C.3. Get the sign convention right before testing; the pseudocode above is A relative to B.
• Thresholds were fitted on the reported annotations. The counting threshold and the dead-zone factor were chosen on the same human annotations used to report agreement; K-fold checks give 0.823 ± 0.013 and 0.822 ± 0.013 app. A.2. The CLIPScore threshold was also tuned per task on the annotations §4. The comparison therefore favours the tuned method, and the held-out numbers are the ones to quote.
• Kappa is unstable for single-object. Almost all images are correct, so kappa for the single-object task is near 0 app. A.1 and should not be compared across methods.
• Seed noise. Overall scores move by 0.01-0.02 across random seeds tab. 2. Compare models only when the gap exceeds that, and state the seeds.
• Count sensitivity to detector confidence. A low threshold (0.3) inflates counts when several instances of one class appear app. A.2 fig. 8. The 0.9 threshold for counting is tuned to COCO-trained detections.
• Detector and colour failure modes. Holes in masks mislead the colour classifier; overlapping objects of one class merge boxes; clip art and artistic renders are out of distribution §4 fig. 4. The paper's own figure shows the position bias of IF-XL and colour swapping in SDv2.1 fig. 6 fig. 10. These are model findings, not bugs in the scorer.
• Unstated generation settings. Sampler, steps, guidance scale, resolution and seed are not given per model app. C.2. The paper says extra prompt wording ("a photo of") and a universal negative prompt did not help in preliminary tests app. C.2. Results with other settings are not comparable.
• Cost. Generating 4 images for 553 prompts plus the detector and one CLIP pass per object is GPU work; the scorer will not run without CUDA evaluate_images.py.
• Table numbers are counted in source order. The paper uses no equation numbers, and its algorithm-free formulas are unnumbered. Figure and table numbers here are counted from the LaTeX source in order of appearance, which is how they are compiled.

Lineage

• Builds on: CLIPScore (reference-free alignment baseline) §1 §2; CLIP embeddings for the colour classifier app. C.3; Mask2Former and MMDetection for detection app. C.3; COCO classes §3.1.
• Compared against: CLIPScore with several CLIP backbones app. C.3; the LAION-5B CLIP-retrieval baseline §5.1 app. C.2.
• Related, not used as content: Dall-Eval and VISOR (object-detection and spatial-reasoning metrics) §2; TIFA and LLM/VLM-based evaluators §2.
• Superseded by: not stated.