Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

In plain words

Picture two photos and two short sentences. Both sentences use exactly the same words, but in a different order, so they describe different scenes. One sentence might say a tree sits inside a shopping cart, and the other that the cart sits inside the tree. A person pairs them up at a glance. This paper asks whether today's AI models that read both pictures and text can do the same.

The authors built a small, carefully hand-made test set for this, called Winoground. They then ran many popular models on it. Almost all of them did about as well as guessing, while people did very well. The test is meant to be a yardstick for future models, not a new model itself.

TL;DR

Winoground is a 400-example, expert-written probe in which two images and two captions share one bag of words, so only word order and visual grounding can separate the pairs §3.1. Scores are text, image and group accuracy §3.2. Ten transformer families and two RNN families were evaluated, and most sit near or below chance. Humans score 85.50 on the group score, against a best model group score of 14.50 tab. 3. Use it as a diagnostic for compositional image-text matching; the numbers are author-reported.

Problem

Vision-language transformers are strong on many multimodal tasks but poorly understood. The question is whether they can reason about composition: the same words in a different order should give a different picture §1. Earlier probes either use synthetic images, or change the set of words between captions, so a bag-of-words model can cheat. Winoground keeps the word set fixed, so that shortcut is closed §2.

Method

The paper proposes a benchmark and an evaluation protocol rather than a model.

Dataset construction §3.1. Four expert annotators with vision-language and computational-linguistics experience wrote the examples using licensed Getty Images. An example counts as valid only if the annotator prefers the two correct image-caption pairs over the two swapped ones, and the two captions have the same words or morphemes in a different order §3.1. The result has 1600 image-text pairs, 800 correct and 800 incorrect, built from 400 examples with 800 unique captions and images §3.1. Each example is tagged with linguistic swap types (object, relation, both), with the number of main predicates (one or two), and with three visual tags (pragmatics, symbolic, same image series) §3.1 tab. 1. Example items per tag are shown in a figure in §3.1.

Scoring §3.2. Let s(caption, image) be the model's score for a pair. Three binary per-example flags are defined:

• text score f: each image scores higher with its own caption than with the other caption eq. 1;
• image score g: each caption scores higher with its own image than with the other image eq. 2;
• group score h: both f and g hold eq. 3.

The group score is stricter than the other two, following the argument that scoring twin sentences individually overestimates performance §3.2.

# Winoground flags for one example  [eq. 1] [eq. 2] [eq. 3]
# Input:  score(caption, image) -> float, from the model under test
#         caption_0, caption_1, image_0, image_1
# Output: text_ok, image_ok, group_ok (booleans)

def winoground_flags(score, caption_0, caption_1, image_0, image_1):
    # Text score: each image prefers its own caption  [eq. 1]
    text_ok = (
        score(caption_0, image_0) > score(caption_1, image_0)
        and score(caption_1, image_1) > score(caption_0, image_1)
    )
    # Image score: each caption prefers its own image  [eq. 2]
    image_ok = (
        score(caption_0, image_0) > score(caption_0, image_1)
        and score(caption_1, image_1) > score(caption_1, image_0)
    )
    # Group score: all four pairings must be right  [eq. 3]
    group_ok = text_ok and image_ok
    return text_ok, image_ok, group_ok

# Aggregate score per metric, in percent  [tab. 3]
# Input:  one boolean per example, 400 examples  [§3.1]
# Output: percentage of examples flagged correct

def percent_correct(flags):
    # Reported values are multiples of 0.25, i.e. counts out of 400 (derived)
    return 100 * sum(flags) / len(flags)

Figures. The teaser example fig. 1 shows a pair in which every tested model fails, except one large checkpoint by a very small margin fig. 1. Heatmaps for ViLT, ViLLA and UNITER are in §6.1 and app. D; the source does not give their scoring details beyond the caption.

Hyperparameters

Winoground is evaluation-only: no model is trained on it. The table lists the evaluation settings that are stated.

Results

Author-reported. Setup: 400 Winoground examples, the listed pretrained models, human baseline from Amazon Mechanical Turk workers §4.3. Bold text in the source marks values above chance (25.00 %) for text score.

Key findings, in the paper's words §5.1:

• Text score: about 50 points separate humans from the best models, and the remaining models are below chance §5.1.
• Image score: even the best model, FLAVA ITM, trails humans by about 70 points §5.1.
• Group score: all models are below chance §5.1.
• Tag breakdown §5.2 tab. 4 tab. 5: humans do best on object swaps; models do worst on the same-image-series tag, where most get a 0 % group score tab. 5. CLIP's "both" text score exceeds the human one (80.77 against 76.92) tab. 4.

Confidence intervals app. A are wide because each group has about 100 examples. Example: human text score 89.50 with interval [80.83, 98.17]; FLAVA ITM text score 32.25 with interval [20.04, 44.46].

Paper vs code

No official code.

Searched: the paper text and the LaTeX source contain no code link, only the dataset URL (https://huggingface.co/datasets/facebook/winoground). The project-page step does not apply, since none is linked. A GitHub repository search for "winoground" returned 17 repositories, none owned by the authors, by Meta AI's facebookresearch organisation or by the authors' lab; the analysis and reimplementation repositories among them were not used. A search restricted to org:facebookresearch returned nothing, and so did a search restricted to the authors' user names.

Pitfalls

• The score function is not specified in the compiled text. Section 3.2 defines s only as "the model's score", and the model section does not say which score each model uses. The LaTeX source has a commented-out sentence (winoground_arxiv.tex line 405) saying CLIP uses cosine similarity and the other models use the softmax probability of an image-text-matching head. That sentence is not published, so implement it only as a guess and test the choice. Only UniT's fine-tuning is stated §4.1.
• Strict inequalities. The equations use >, so exact ties count as failures for every model eq. 1 eq. 2.
• Chance-level wording differs. The abstract says none of the models does much better than chance. Section 5.1 says six models exceed chance on the text score and only that score §5.1 tab. 3.
• Visual-tag denominators do not match the tag counts. Table 1 gives Symbolic 41, Series 31 and Pragmatics 24 examples tab. 1. The human percentages in Table 5 imply other denominators: 96.43 = 27/28 for symbolic, 95.65 = 22/23 for same series and 58.82 = 10/17 for pragmatics tab. 5. The paper does not explain this. Do not compute per-tag statistics from the visual-tag table without checking.
• Small set. 400 examples; the authors say expert curation limits size, and the confidence intervals are wide §7 app. A.
• Human baseline is not a ceiling. Crowdworkers judge one pair at a time, and the authors suspect that showing both images and captions together, or using more raters, would raise human pragmatics scores §5.2.
• Image licences. The images were licensed from Getty Images for research use §3.1, and the paper's acknowledgement lists the Getty copyright holders §7. This card does not reproduce any image. Check the dataset's own terms before redistributing it.
• English only §7. Translation to other languages is noted as non-trivial.
• Author interpretations. The "richer features" explanation and the claim that ViLT ignores adjectives are the authors' hypotheses, from heatmaps and discussion §6.1.
• Models are evaluated as published checkpoints. The compiled text names the variants (e.g. ViT-B/32) but not every training detail tab. 3 §4.1.

Lineage

• Builds on: the Winograd Schema Challenge, which uses minimally different sentence pairs to test commonsense coreference §2. Group-score reasoning follows Elazar et al. on Winograd evaluation §3.2.
• Compared against: ten transformer families and two RNN families, listed in tab. 2 and tab. 3 §4.
• Superseded by: not stated in the compiled paper.