CREPE: Can Vision-Language Foundation Models Reason Compositionally?

In plain words

Vision-language models learn to match pictures with captions. A caption is built from parts: objects, their colours or other attributes, and how they relate ("a black dog on a red couch"). People combine such parts freely, so they can understand a new combination they have never heard before. This paper asks whether the models do the same.

The authors build a test called CREPE. For each picture, the model must pick the correct caption out of a list. The list also contains near-miss captions that differ by one part, such as a wrong attribute or relation, a swapped word order or a negation. A model that only matches loose word patterns gets fooled by these.

The test checks two things. The first is whether a model handles combinations it saw in training, compared with combinations that were rare or absent there. The second is whether it still works as captions grow longer and have more parts. The authors report that the models they tested lose accuracy on both counts, and that the drop is clearest for the models trained on the largest dataset.

TL;DR

• CREPE is a zero-shot image-to-text retrieval benchmark that scores compositionality on two axes: systematicity and productivity §3.
• Systematicity splits each test set by whether its atoms and compounds appeared in the training captions of CC-12M, YFCC-15M or LAION-400M §3.2 §4.1.
• Productivity uses captions with 4 to 12 atoms and three hard-negative types: atomic, swap and negation §3.3 §4.2.
• Author-reported: on the combined systematicity set, Recall@1 drops from seen to unseen compounds for all seven models, most for LAION-400M models on atom-foil negatives (up to 12.57 points for ViT-B-32) §5.1 app. B.1.
• Author-reported: on productivity, Recall@1 falls as complexity rises for the atom-foil and swap sets, and most models sit near the 16.67% random line at high complexity §5.2 app. B.1.

Problem

Vision-language models are scored mostly by image-text retrieval. Those datasets either have no controlled negatives or only vary one axis, such as word order or a single substitution, and they do not test unseen combinations or growing complexity §1. The paper argues there is no large benchmark that measures compositional behaviour directly for such models §1.

The evaluation assumes that a model's retrieval score on a hard negative reflects its representation, not dataset artefacts. The authors point out that unlabelled alternatives can be true captions, so they generate hard negatives that are verified as incorrect §3.4.

Method

Overview. The benchmark is built from Visual Genome scene graphs (atoms are objects, relations or attributes; compounds are attribute-object or object-relation-object pairs) §3.1. Two test families are built from them.

• Systematicity takes three training corpora (CC-12M, YFCC-15M, LAION-400M). Each corpus is parsed into atoms and compounds. Every test caption then goes into one of three splits: Seen Compounds (SC), Unseen Compounds (UC) or Unseen Atoms (UA) §3.2 §4.1. Hard negatives are of two types: HN-Atom replaces one atom with a foil; HN-Comp joins two single-compound foils §4.1.
• Productivity samples subgraphs of n atoms, n = 4 to 12, and writes a caption for each. Hard negatives are HN-Atom, HN-Swap (permutes atoms of the same kind) and HN-Neg (negates an atom or the whole caption) §3.3 §4.2 app. A.3 app. A.4.

Evaluation is zero-shot image-to-text retrieval. For each image, the candidates are its ground-truth caption and the hard negatives of one type. A query is scored by the rank of the ground truth §3.4 §5.

# Assign each test pair to one systematicity split  [§3.2]
# Input:  pairs: list of (image, graph) from the test set
#         train_atoms, train_compounds: sets seen in the training captions
# Output: dict from split name to list of pairs

def split_pairs(pairs, train_atoms, train_compounds):
    splits = {"SC": [], "UC": [], "UA": []}
    for image, graph in pairs:
        # Any atom never seen in training: unseen atoms  [§3.2]
        if any(atom not in train_atoms for atom in graph.atoms):
            splits["UA"].append((image, graph))
        # All atoms seen, but some compound never seen  [§3.2]
        elif any(c not in train_compounds for c in graph.compounds):
            splits["UC"].append((image, graph))
        # Every compound seen in training  [§3.2]
        else:
            splits["SC"].append((image, graph))
    return splits

# Make one productivity caption of n atoms from a scene graph  [§4.2]
# Input:  graph: scene graph of one image; n: atom count, 4 to 12
# Output: (caption, crop box), or None if the sample is discarded

def make_caption(graph, n):
    # Random walk that grows a connected subgraph of n atoms  [app. A.3]
    subgraph = random_walk_subgraph(graph, n_atoms=n)
    # Discard walks that end with too few atoms  [app. A.3]
    if subgraph is None:
        return None
    # Template for n = 4, few-shot GPT-3 for n >= 5  [§4.2] [app. A.3]
    if n == 4:
        caption = template_caption(subgraph)
    else:
        caption = gpt3_caption(subgraph, n)
    # Drop the caption if GPT-3 leaves out an atom  [§4.2]
    if omits_atom(caption, subgraph):
        return None
    # Crop the image to the union of the subgraph boxes  [§4.2]
    box = union_of_boxes(subgraph)
    return caption, box

# Rank the ground truth in one retrieval set, image-to-text  [§3.4]
# Input:  image_emb (d,); text_embs (1 + h, d), ground truth at row 0
# Output: rank of the ground truth, 0 = top

def ground_truth_rank(image_emb, text_embs):
    # Unit-normalised embeddings; dot product as similarity  [§3.4]
    scores = text_embs @ image_emb
    # Candidates sorted from highest to lowest score  [§3.4]
    order = argsort(scores, descending=True)
    # Where row 0 (the ground truth) lands in that order  [§3.4]
    return position_of(0, order)


# Recall@K over all queries of one retrieval set  [§5]
def recall_at_k(ranks, k):
    return mean(ranks < k)

Filtering and quality control.

• Region crops must cover at least 40K pixels, at least 10% of the image, and have a width-to-height ratio between 0.5 and 2.0 §4.1.
• Text must contain at least 2 atoms and 1 compound; duplicates are removed by scene graph §4.1.
• For productivity, crops overlapping by 75% or more are de-duplicated app. A.3 §4.2.
• WordNet picks antonym or cousin atoms as foils; BERT filters systematicity foils §4.1. RoBERTa entailment filters HN-Swap foils §4.2.
• Human check: 2% of GPT-3 productivity captions and 1% of queries, two annotators each. 87.9% of sampled captions are faithful, 83.7% of productivity hard negatives and 86.0% of systematicity hard negatives are genuine negatives §4.3.

Retrieval sets. Systematicity sets have 5 candidates for HN-Atom and 7 for HN-Comp; productivity sets have 6 per hard-negative type app. A.5. The overall productivity set pools five negatives of each of the three types, so 16 candidates app. B.3.

Models. Systematicity uses seven OpenCLIP models pretrained on CC-12M, YFCC-15M or LAION-400M: RN50 and RN101 on YFCC, RN50 on CC-12M, and ViT-B-32, ViT-B-16, ViT-B-16-plus-240 and ViT-L-14 on LAION-400M §5. Productivity adds OpenAI CLIP (RN50, RN101, ViT-B/32, ViT-B/16, ViT-L/14), CyCLIP, ALBEF and FLAVA §5.

Hyperparameters

Results

Author-reported. Setup: CREPE test sets built from Visual Genome; open_clip models pretrained on CC-12M, YFCC-15M and LAION-400M; zero-shot retrieval; no training on CREPE. Random chance is 9.09% for the combined systematicity set, 20.00% for HN-Atom, 14.29% for HN-Comp and 16.67% for each productivity type app. B.1 app. B.2.

Systematicity, Recall@1 for image-to-text, SC → UC, with change (UC − SC) in points. Values from app. B.1. Combined = HN-Atom + HN-Comp.

The text also reports the human oracle at 100% on the systematicity set §5.1.

Productivity, image-to-text Recall@1 at complexity 4 and 12. Values from app. B.1 (full nine-level tables are there). Random = 16.67 for all types.

Source for the productivity table: app. B.1 (tables for HN-Atom, HN-Swap and HN-Neg). The headline claims are: retrieval near random chance for HN-Atom at high complexity and below it for HN-Swap §5.2; no dataset-size effect on productivity §5.2; no clear model-size trend §5.3; ImageNet zero-shot accuracy correlates with systematicity Recall@1 (R² 0.95 and 0.80 for SC and UC on HN-Atom, 0.91 and 0.97 on HN-Comp) but not with productivity on HN-Swap (R² 0.07, 0.08, 0.47 at n = 4, 8, 12) §5.4.

Paper vs code

The repository is https://github.com/RAIVNLab/CREPE, linked in the paper's footnote §1. The paper links it, which meets the official-code rule. Commit 1fa81c4 was cloned; it has no LICENSE file, so no code is quoted and the card cites code only by path and line.

• Dataset generation is not in the repository. The paper says the authors release "code to generate and evaluate" the test sets §1. The repository holds evaluation scripts and two zip archives of hard negatives syst_hard_negatives.zip prod_hard_negatives.zip. The parser, the filters, WordNet/BERT/RoBERTa checks and the GPT-3 prompts are not present. The thresholds in §4 cannot be checked against code.
• Raw-set evaluation is not in the code. The K-fold retrieval on D_RAW app. B.2 has no matching script. The scripts only read hard-negative CSVs crepe_compo_eval_open_clip.py.
• Checkpoint variants differ from the names in the paper. The paper names RN50, RN101 and ViT-B-32 §5. The code loads the quickgelu checkpoints for RN50, RN101 and ViT-B-32 crepe_compo_eval_open_clip.py. The paper does not mention this variant.
• Ground truth is always row 0, and ties are not handled. The code places the caption at index 0 and ranks with argsort crepe_eval_utils.py. Equal scores are ordered by argsort, which the paper does not discuss. The ground truth can then gain or lose rank on exact ties, for example if a hard negative equals the caption.
• Truncation differs between scripts. The OpenAI CLIP script calls clip.tokenize(..., truncate=True) crepe_prod_eval_clip.py. The open_clip script calls tokenize without a truncate flag crepe_compo_eval_open_clip.py crepe_compo_eval_open_clip.py. The paper does not say how over-long captions are handled. FLAVA and ALBEF pad every caption to max_length crepe_prod_eval_flava.py crepe_prod_eval_albef.py.
• Averages in the tables are not computed by the code. The code returns R@1, 3, 5 and 10 crepe_eval_utils.py crepe_eval_utils.py. In the tables, "Avg R@K" equals (R@1 + R@3) / 2 on every row checked, although the captions say R@1, 3, 5. This is not in the code.
• Matching checks. Crop coordinates and the one-to-many retrieval setup match the paper crepe_compo_eval_open_clip.py crepe_compo_eval_open_clip.py.

Pitfalls

• Headline drop is in absolute points on one subset. The abstract says Recall@1 drops "by up to 12%" [abstract]. The 12.57 points is the HN-Atom subset for ViT-B-32 (55.32 → 42.75) app. B.1. On the combined set the largest drop is 6.28 points. Report the subset and the unit.
• HN-Comp is not "no difference" for LAION models. The text says HN-Comp shows "little (1–3%) to no difference" §5.1. The table shows the four LAION models score 2.5 to 3.7 points higher on unseen compounds than on seen ones app. B.1. The direction is reversed, not flat.
• Random baselines depend on set size, and the combined set is not explained. Random is 20.00 for HN-Atom and 14.29 for HN-Comp, which are 1/5 and 1/7. The combined random of 9.09 equals 1/11, but the text gives no combined set size app. B.1 app. A.5.
• Table 1 text counts do not match the appendix. Table 1 gives 325,523 systematicity hard-negative texts for CC-12M tab. 1. The HN-Atom and HN-Comp counts in app. A.5 sum to 295,930 for CC-12M SC plus UC. The gap is not explained. The productivity totals do match (183,855 hard-negative texts; 17,553 ground truths).
• Complexity confounds caption source and foil type. Captions at n = 4 come from templates; n ≥ 5 come from GPT-3 §4.2. For HN-Swap at n = 4, any permutation of same-type atoms is allowed app. A.4, so the n = 4 swap set is a different task. Treat the n = 4 column as a different distribution.
• Hard-negative quality is not perfect. The discussion says about 70% of HN-Swap and HN-Neg negatives are rated correct as negatives §6. Section 4 reports 83.7% for all productivity negatives §4.3. Both numbers apply, and the swap and negation sets are noisier than HN-Atom app. A.5.
• Parser recall is low on LAION. Recall for triplets is 64.04%, 81.11% and 39.55% for CC-12M, YFCC-15M and LAION-400M app. A.2. The authors argue that a poor recall would make UC easier, not harder app. A.2. Recall that UC/SC membership is parser-dependent.
• Some evaluated models were trained on Visual Genome images. The paper notes FLAVA and ALBEF include Visual Genome in training §5.2. The productivity crops come from Visual Genome, so those results are not held out.
• Zero-shot ImageNet correlation is not a generalisation claim. The high R² values are for absolute recall; the SC-UC gap and the productivity gap are not explained by ImageNet accuracy §5.4.
• Seven checkpoints. The systematicity claim rests on seven checkpoints, and the RN models use the quickgelu weights (see Paper vs code). No variance across seeds or runs is reported for the HN sets §5.
• Raw-set numbers have a known false-negative issue. Simple captions are likely true for several images, so low-complexity raw scores are unreliable app. B.2.

Lineage

• Builds on: LAION-400M, the largest training corpus here §5; LAION-400M (in this wiki).
• Builds on: Visual Genome scene graphs as the compositional language §3.1; not a card in this wiki.
• Compared against: image-text retrieval sets without controlled negatives, and narrow-negative sets such as VALSE, Winoground and FOIL §1; not cards in this wiki.
• Evaluated models: CyCLIP (arXiv:2205.14459) and FLAVA (arXiv:2112.04482), as cited in §5; no cards in this wiki.
• Superseded by: not stated.