What does CLIP know about a red circle? Visual prompt engineering for VLMs

In plain words

CLIP matches pictures with text. People usually steer such a model by changing the words they give it. This paper changes the picture instead. Draw a plain red circle around one object, and CLIP starts to attend to that spot while still seeing the rest of the picture. You can then ask about the circled object, for example by naming it, or pick the circled object that a sentence describes.

The authors compare this with the older trick of cutting out the region and feeding only the crop. A crop throws away the surroundings, and the paper finds the surroundings matter. A circle keeps them, and in the tests on this card it does better. No training is needed; the circle is the only change.

The paper also finds a downside. Circles can push the model toward labels such as "criminal" or "missing person" for the people inside them, so the method must not be used on sensitive data.

TL;DR

• Drawing a red circle on an image steers CLIP to the circled region without cropping away the context §1 §3.2.
• Scored zero-shot, circled images beat crops on keypoint naming (CUB 58.2% vs 25.5%, both with Sinkhorn-Knopp) and keypoint localisation (CUB 45.2% vs 21.3%, both with the saliency mask), author-reported tab. 1 tab. 4.
• Zero-shot referring-expression comprehension reaches state of the art on most benchmarks, e.g. 63.9% top-1 on RefCOCO+ testA tab. 5.
• The same marking raises the rate at which people are labelled criminal, so it must not be used on sensitive data §4.4 tab. 6.

Problem

CLIP scores how well an image and a text match. It does not map a point in the image to a word, so a region must be encoded in the input somehow §3.1. The standard way is to crop the image around the region. Cropping loses the surrounding context, which the paper argues matters §3.2 §1.

The paper asks whether an edit in pixel space, a fixed visual prompt, can carry location information instead. It contrasts this with visual prompt tuning, where learnable tokens are trained for the task §1, footnote.

Assumptions stated in the paper:

• The model is a zero-shot CLIP with no task training §3.1.
• Referring-expression proposals come from MAttNet §4.3.
• Keypoint localisation uses an unsupervised saliency mask (TokenCut) to drop background locations §3.3.
• The authors conjecture that CLIP learned the circle from annotated images in its training data. They found about 70 such images among 10k candidates in YFCC15M, which they call rare (about 0.001%) §4.1. This is a hypothesis the paper does not test directly.

Method

Scoring a marked image §3.1 eq. 1

The paper casts zero-shot prediction as question and answer. For a question q and a candidate answer a, the score is computed on engineered inputs: s(q,a|i,t) = s(i_qa, t_qa) eq. 1. For keypoints and referring expressions, the image is marked at candidate a and the text encodes the question q §3.1.

1. Pick candidate locations a. For each one, make i_a by drawing a red circle at a on the image. The marked image keeps the rest of the image §3.2.
2. Encode the question as text t_q, for example "an image of" plus a keypoint name §3.3.
3. Score each pair with s(i_a, t_q) and choose the best location or match §3.1 §3.3.

Marking is the paper's name for overlaying a circle, a box or an arrow at the location. Circles work best in its tests §3.2 §4.1 tab. 2.

Naming keypoints §3.3 eq. 2

Given m keypoint names and m locations, predict the permutation Π that matches each name to one location. The cost of matching name q to location a is C_qa = s(i_a, t_q) §3.3. The paper decodes the cost with optimal transport: the permutation that maximises the sum of Π_qa exp(−τ C_qa) eq. 2. The decoding is solved with the Sinkhorn-Knopp algorithm, which renormalises C §3.3.

# Match keypoint names to marked locations  [§3.3] [eq. 2]
# Input:  image; names (m strings); locations (m points)
# Output: permutation matrix (m, m)

def match_names(image, names, locations):
    # One marked image per candidate location  [§3.2] [app. Keypoint tasks]
    marked = [red_circle(image, loc) for loc in locations]  # m images
    # Text prompt for each name  [§3.3]
    prompts = ["an image of " + name for name in names]     # m strings
    # Cost C[q, a] = s(i_a, t_q)  [§3.3]
    cost = clip_similarity(marked, prompts)                 # (m, m)
    # Soft assignment: Sinkhorn-Knopp on exp(-tau * C)  [eq. 2] [§3.3]
    plan = sinkhorn(exp(-tau * cost))                       # (m, m)
    # Hard permutation from the soft plan  [not stated in the paper]
    return round_to_permutation(plan)

Localising keypoints §3.3 app. Keypoint tasks

Here the locations are unknown. Candidates are a regular M × M grid, restricted to a salient region from TokenCut §3.3. The keypoint with name q is then the argmax over the remaining candidates of s(i_a, t_q) §3.3. The paper's argmax has no temperature.

# Localise one named keypoint  [§3.3] [app. Keypoint tasks]
# Input:  image; name (str); saliency (H, W) in {0, 1}
# Output: (row, col) of the best candidate

def localise(image, name, saliency, grid_size=30):
    # Regular grid of M x M candidate locations, M = 30  [app. Keypoint tasks]
    candidates = regular_grid(grid_size)                    # (900, 2)
    # Drop candidates outside the salient region  [§3.3] [app. Keypoint tasks]
    candidates = [c for c in candidates if saliency[c] == 1]
    # Dataset-specific template for the name  [app. Keypoint tasks]
    text = template.format(part=name)
    # Score each candidate by its circled image  [§3.2] [§3.3]
    scores = [clip_score(red_circle(image, c), text) for c in candidates]
    # Best candidate: argmax over a  [§3.3]
    return candidates[argmax(scores)]

Referring-expression comprehension §3.3 eq. 3

Proposals A come from MAttNet §4.3. For each query q, the best proposal is the argmax over a of s(i_a, t_q) minus the mean score of a over the other queries eq. 3. The paper says this subtraction downweights proposals, such as faces, that respond strongly to every query §3.3. The circle variants and the backbone ensemble are in the appendix app. Referring Expressions Detection.

# Pick the box that matches a referring expression  [§3.3] [eq. 3]
# Input:  image; boxes (N, 4) from MAttNet; query (str)
#         other_queries (Q strings; the paper uses Q = 500)
# Output: index of the chosen box

def comprehend(image, boxes, query, other_queries):
    # Prefix the query with "This is"  [app. Referring Expressions Detection]
    text = "This is " + query
    best_index = None
    best_score = -inf
    for index, box in enumerate(boxes):
        # Three marked views of the box  [app. Referring Expressions Detection]
        views = [red_circle(image, box),
                 red_circle_blur_outside(image, box),
                 red_circle_gray_outside(image, box)]
        # Score the query, minus the mean over the other queries  [eq. 3]
        # How the views and the two backbones are combined is not stated.
        score = clip_score(views, text)
        score = score - mean(clip_score(views, t) for t in other_queries)
        if score > best_score:
            best_score = score
            best_index = index
    return best_index

Hyperparameters

Values as stated in the paper. "Not stated" means the paper does not give the value in the text or appendix checked.

Results

Author-reported. Setup for keypoints: CUB-200-2011 and SPair71k, with names added by the authors to SPair71k §4.1 app. Datasets. Setup for referring expressions: RefCOCO, RefCOCO+, RefCOCOg, MAttNet proposals §4.3. Zero-shot comparisons, with the supervised Pseudo-Q row marked as not zero-shot.

Naming keypoints, % correct tab. 1. Name-to-keypoint (left) and keypoint-to-name (right). SK = Sinkhorn-Knopp normalisation.

The paper reports that red circles about double the crop baseline on naming §4.1.

Marker shape and colour, CUB naming tab. 2. Mean ± std over sizes, with the best value. The table caption describes these columns differently from the rows; see Pitfalls.

Backbones for naming tab. 3. Name-to-keypoint and keypoint-to-name, % correct, with the appendix sample sizes. ★ = evaluated at 336 × 336 input.

Data: 400M = WIT-400M, 2B = LAION-2B, 15M = YFCC-15M, 88M = DeCLIP-88M. †, ‡: the authors evaluated the models trained by the cited SLIP and DeCLIP papers tab. 3 caption. The paper says marking gains grow with pretraining set and encoder size, and that LAION-2B pretraining gives worse results than WIT-400M, which it attributes to filtering §4.1 fig. 6.

Keypoint localisation, PCK (α = 0.1) tab. 4. "Mask" = saliency mask.

Referring expressions, top-1 accuracy % tab. 5. ZS = zero-shot.

The paper says red circles beat the other zero-shot methods on most columns, including ReCLIP‡, which adds hand-written relation rules. ReCLIP‡ is better on RefCOCO testB and RefCOCOg test tab. 5 §4.3.

Referring-expression backbones, Red Circle rows only app. Referring Expressions Detection. The full table, including ReCLIP's rows, is in the paper.

The paper picks RN50×16 + ViT-L/14@336 as the ensemble app. Referring Expressions Detection. Its row in the table above matches the headline numbers in tab. 5.

Referring-expression ablations app. Referring Expressions Detection. RN50×16 and ViT-L/14@336 backbones, RefCOCO val / testA / testB:

• Red circle alone: 42.01 / 48.58 / 36.90.
• Plus mean subtraction: 43.67 / 50.20 / 38.59.
• Plus ensemble of the three marked views: 49.84 / 58.57 / 39.96.

The paper reports that a thin circle scores best among its marker sizes, shapes and colours, and that blurring or greying the outside helps [app. Referring Expressions Detection, Annotations].

Bias, criminal-category rate % tab. 6. Zero-shot classification into positive (honest man/woman/person), neutral (man/woman/person) and criminal (criminal/thief/suspicious person) categories. The bracketed values are the change in the criminal rate against the same model without circles, as printed in the table.

* The source table reads "505.%" for this cell. The row sums to 100 only if it is 50.5%, so 50.5% is used here.

The paper's reading: for every ViT encoder, people are labelled criminal more often with circles, while the RN50×16 model on synthetic faces goes the other way §4.4 tab. 6. Also see Pitfalls.

Paper vs code

No official code. The paper body gives no code link, and the arXiv metadata we fetched has none. The paper's ethics statement links a VGG publication page, which links no code. A GitHub search returned suny-sht/clip-red-circle, under the first author's account. At commit bcb15c7370604eddc6a555efb4c8ff37010c0293 it contains a README, an example image and the note "Code coming soon!". It has no LICENSE file and no code. A third-party repository, YangYY-Liu/CircleCLIP, is not an author repository and is not used.

What we checked: the paper's links, its project page, the author repository. Therefore no paper-vs-code comparison is possible. Implement from this card and the paper, and treat the parameters in the table above as the paper's own.

Pitfalls

• Two marker sizes for keypoints. The main setting is r = 0.06 H and t = 0.01 H, which is 20 px and 3 px at H = 336 app. Keypoint tasks. The appendix figure caption says the best size in its sweep is radius 12 px and thickness 4 px app. Keypoint tasks. The paper does not say which of these generated the headline numbers; the 20 px and 3 px values are the ones it states for its setup.
• Referring-expression marker size uses a different unit. The size unit there is 0.5% of the larger image side, so "1 pixel for an image of size 224" app. Referring Expressions Detection, Annotations. This is not the same rule as the keypoint radius.
• Temperature τ is ambiguous. The paper states τ = 1/150 in the localisation paragraph of its appendix app. Keypoint tasks. Eq. 2 for naming also uses τ, and the paper does not say whether the same value applies. The Sinkhorn-Knopp iteration count is not stated.
• Hard permutation not stated. Eq. 2 defines the soft optimal-transport plan, and the paper names Sinkhorn-Knopp for solving it §3.3. How the plan is turned into a single permutation is not stated.
• Ensemble rules not stated. The paper ensembles three marked views per box and two backbones, but does not give the combination rule (sum, mean, or weights). Re-implementing the headline REC result needs this decision, and the card cannot make it for you.
• Crop baseline size not stated. The crop baselines tab. 1 tab. 4 do not give the crop size or margin, so the baseline may differ from a re-implementation.
• Blur and grey strengths not stated. The blur and grey-out variants app. Referring Expressions Detection, Annotations give no strength or kernel size.
• Sampling pool for Q is not stated. The mean subtraction uses Q = 500 randomly sampled expressions app. Referring Expressions Detection. The paper does not say which split they come from, or the random seed.
• Keypoint names are authors' own. The SPair71k names were added manually by the authors. The paper says which keypoints were dropped or merged (nostrils, tails, ears, birds' knees) and gives the name tables app. Datasets. Whether the names are released is not stated, so rebuilding this exact benchmark may need your own labels.
• Table caption mismatch. The caption for the shape and colour table says the table compares sizes for shape and colour for circles, which does not match its rows tab. 2. Read the rows as the data.
• Bias evidence is limited. The bias study uses 1000 FaceSynthetics faces, 1352 COCO person crops and binary gender only. The paper calls the analysis limited §4.4. Its own table caption describes the bracketed values as differences from "the crop" without a circle; they are differences from the same model without circles.
• Pretraining explanation is a hypothesis. The paper attributes the effect to rare annotated images in the training data, and suspects that filtering explains the weaker LAION-2B result §4.1. Neither is tested directly.
• Cost scales with proposals. REC needs three marked images per proposal and an extra mean over Q queries per proposal, on two backbones. The paper gives no run time or FLOPs for the method.
• Not for sensitive data. The paper says marking must not be used to analyse sensitive data, because the model labels marked people more often as criminal or missing §4.4.

Lineage

• Builds on: CLIP, used as the scorer for both prompt types §3.1 §4.1. The REC setup follows ReCLIP's backbone choice and its crop baseline §4.3 app. Referring Expressions Detection.
• Builds on: MAttNet, for the referring-expression proposals §4.3. TokenCut, for the keypoint saliency mask §3.3. Sinkhorn-Knopp, for matching §3.3.
• Compared against: crop-based prompting tab. 1 tab. 4; zero-shot ReCLIP, CPT and DTWREG, and the weakly supervised Pseudo-Q tab. 5 §4.3. Visual prompt tuning is named as a different setting §1.
• Superseded by: not stated in this wiki.