CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching

In plain words

A normal object detector only finds the kinds of things it was trained on. Open-vocabulary detection asks for a detector that can also find things whose names it never saw during training, and you name those things when you run it. CORA gets this by starting from CLIP, a model that already links pictures to words, and using it as the part that names each object.

Two problems make this harder than it sounds. CLIP was trained on whole pictures, but a detector shows it small crops, so the crops look slightly different from what CLIP expects. CORA handles this with a small set of learned adjustments, called region prompts, applied to the features of each box, so the box is judged in the way CLIP expects. The second problem is finding boxes around objects of unseen kinds. CORA's box finder first guesses which named class each candidate box belongs to, and then lets that guess shape where the box goes. Both changes are trained on ordinary detection data with the base classes, and the detector can then be pointed at any list of new names.

TL;DR

• CORA is a DETR-style detector (DAB-DETR as localizer) with a frozen CLIP image encoder and a CLIP text encoder §3.1 §4.2.
• Region prompting learns a prompt added to pooled region features before CLIP's attention pooling, and trains only that prompt with cross-entropy on base-class boxes §3.2 eq. 1.
• Anchor pre-matching labels each anchor with its best CLIP class and conditions the query on that class's embedding, so one decoder pass covers all classes §3.3 eq. 2 eq. 3.
• Author-reported on COCO OVD (generalized, AP50 on novel classes): 35.1 for RN50 and 41.7 for RN50x4 without extra data, 43.1 for CORA+ tab. 1.

Problem

Open-vocabulary detection trains on base-class boxes and must localise and classify boxes of novel classes at test time §3. Two obstacles are named: (1) CLIP is trained on whole images, but detection classifies image regions, which causes a distribution gap §1 fig. 2; (2) a proposal network trained on base classes localises novel objects poorly §1 §2. Cropping each region and classifying it as a separate image is the usual workaround. The paper says this is slower for overlapping boxes and loses context §3.2. Methods that need novel class names during training must be retrained when the vocabulary changes §4.3.

Method

Overview §3.1 fig. 1. A CLIP ResNet backbone gives a shared feature map. A DETR-style localizer decodes object queries into boxes. Each box is then classified by a CLIP-based region classifier. Localisation and classification are decoupled and run in sequence §3.1.

Region prompting §3.2 eq. 1. Each box is pooled from the feature map with RoIAlign after CLIP's first three blocks, then passed through the last block, then classified by CLIP's attention pooling §3.2. The paper adds a learnable prompt p \in \mathbb{R}^{S\times S\times C} to the region feature before the attention pooling P eq. 1. Only p is trained, by cross-entropy on ground-truth boxes with the class-name embeddings as fixed classifier weights §3.2. The paper states that p has under 1M parameters §3.2.

v_\mathrm{prompt} = P(f_\mathrm{region} \oplus p) eq. 1

Anchor pre-matching §3.3 eq. 2 eq. 3. The localizer is DAB-DETR: each query carries an anchor box §3.3. Before box regression, each anchor's region feature is compared with every class embedding and the query takes the class with the highest cosine similarity eq. 2. The query is then replaced by an MLP of that class's embedding, so box regression is class-aware eq. 3 §3.3. Unlike conditional matching, all classes are decoded in one pass with a fixed number of queries §3.3 fig. 3.

\hat{c}_i = \underset{c \in C^B}{\arg\max}\ \mathrm{cosine}(v_i, l_c) eq. 2

q_i = \mathrm{MLP}(l_{c_i}) eq. 3

Matching and loss §3.3 eq. 4 eq. 5 eq. 6. Each ground-truth box may only be matched to a prediction with the same pre-matched label, and matching is done per class §3.3. The matching cost is a focal classification term plus an L1 box term and a GIoU term eq. 5. The total loss is a weighted sum of focal, L1 and GIoU terms eq. 6. The paper states the matching cost coefficients equal the loss coefficients, and auxiliary losses are applied to every decoder layer while pre-matching runs once app. A.

\mathcal{L}_\mathrm{cost}(y, \hat{y}) = \mathcal{L}_\mathrm{match}(p,\hat{p}) + \mathcal{L}_\mathrm{box}(b,\hat{b}) eq. 5

\mathcal{L} = \lambda_\mathrm{focal} \mathcal{L}_\mathrm{focal} + \lambda_\mathrm{L_1} \mathcal{L}_\mathrm{L_1} + \lambda_\mathrm{GIoU} \mathcal{L}_\mathrm{GIoU} eq. 6

Inference §3.3 eq. 7. Boxes are re-classified by the CLIP region classifier. The paper scores each box as its matching probability times the cosine similarity to the class embedding eq. 7. Class-agnostic NMS with IoU 0.5 is applied §4.2.

\mathrm{P}(\hat{b}_i \in c) = \hat{p}_i\, \mathrm{cosine}(\hat{v}_i, l_c) eq. 7

Training techniques §3.3 fig. 4. Class dropout: with probability p per iteration, the base classes are split into two complementary random halves and the same image is trained twice, once per half §3.3. CLIP-aligned labeling: ground-truth boxes are relabelled with the region classifier, so boxes whose label disagrees with the classifier are not dropped from matching §3.3. The paper's pseudo-labelling for CORA+ uses CORA to label extra boxes §4.1.

# Region classification with region prompts  [§3.2] [eq. 1]
# Input:  image (3, H, W); boxes (N, 4); class_embeds (C, d), L2-normalised
# Output: cosine scores (N, C)

def classify_regions(image, boxes, class_embeds, prompt):
    # Encode the whole image with CLIP's first three blocks  [§3.2]
    feature_map = clip_blocks_1_to_3(image)           # (C_in, H/16, W/16)
    # Pool each box to an S x S grid with RoIAlign  [§3.2]
    region = roi_align(feature_map, boxes, size=S)    # (N, C_in, S, S)
    # Last CLIP block on the pooled region  [§3.2]
    region = clip_block_4(region)                     # (N, C, S, S)
    # Add the learned region prompt before attention pooling  [eq. 1]
    tokens = flatten(region) + prompt                 # (N, S*S, C)
    embed = attention_pool(tokens)                    # (N, d)
    embed = embed / norm(embed)
    return embed @ class_embeds.T                     # (N, C)

# Anchor pre-matching  [§3.3] [eq. 2] [eq. 3]
# Input:  region features of the Q anchor boxes (Q, d); class_embeds (C, d)
# Output: pre-matched class per query (Q,), query embeddings (Q, h)

def pre_match(anchor_feats, class_embeds, text_mlp):
    # Cosine similarity of every anchor to every class  [eq. 2]
    sims = anchor_feats @ class_embeds.T              # (Q, C)
    # Each query takes its most similar class  [eq. 2]
    pre_classes = argmax(sims, axis=-1)               # (Q,)
    # Replace the query by an MLP of its class embedding  [eq. 3]
    queries = text_mlp(class_embeds[pre_classes])     # (Q, h)
    return pre_classes, queries

# Per-class matching cost, then Hungarian assignment  [§3.3] [eq. 4] [eq. 5]
# Input:  predictions (Q, 4) and their pre-matched classes (Q,);
#         ground truth (G, 4) with labels (G,)
# Output: one-to-one pairs of query and ground-truth box

def match(pred_boxes, pred_scores, pred_classes, gt_boxes, gt_labels):
    # Focal classification cost plus L1 and GIoU box costs  [eq. 5]
    cost = cost_focal(pred_scores, gt_labels)         # (Q, G)
    cost = cost + cost_l1(pred_boxes, gt_boxes)       # (Q, G)
    cost = cost + cost_giou(pred_boxes, gt_boxes)     # (Q, G)
    # A ground-truth box may only match queries with its own label  [§3.3]
    allowed = pred_classes[:, None] == gt_labels[None, :]
    cost[~allowed] = large_value                      # (Q, G)
    # Minimise the total cost over one-to-one pairs  [eq. 4]
    return hungarian(cost)

Hyperparameters

Values stated in the paper are cited to the paper. Values read from the official configs or code are marked as such and cited to the code.

Results

Author-reported. Setup: COCO OVD benchmark, generalized setting (predictions over base and novel classes), evaluated on novel classes with AP50 §4.1. LVIS: mean AP over novel-class boxes (APr) on the validation set §4.1. Region classification: mAP on COCO ground-truth boxes §4.1. Baselines are grouped by CLIP backbone and listed with their extra data and whether they need novel class names tab. 1.

COCO OVD, AP50 tab. 1 (as printed; "Require novel" means the method needs novel class names during training)

The text reports CORA (RN50x4) at +2.4 AP50 novel over the previous best, which is RegionCLIP RN50x4 §4.3 tab. 1. That RegionCLIP row lists CC3M as extra data tab. 1. The paper's claim that CORA uses no extra data is correct for CORA itself. CORA (RN50) is +0.7 over VL-PLM, which needs novel class names §4.3 tab. 1.

LVIS OVD, APr tab. 2.

Region classification, mAP on COCO ground-truth boxes tab. 3. Novel and base columns as printed.

Text: region prompting gives +6.9 mAP novel over CLIP for RN50 and +10.2 for RN50x4 §4.4.

Region features vs cropped whole images, RN50 tab. 4. As printed; see Pitfalls for the row mismatch.

Ablation of anchor pre-matching and CLIP-aligned labeling, COCO novel AP50 tab. 5.

Localisation only, COCO novel vs base AP50 tab. 6 app. B. RegionCLIP 81.7 / 88.2; CORA 83.4 / 88.1 (novel / base). This evaluation replaces labels and scores with ground-truth-IoU based values to isolate box quality app. B.

Paper vs code

Checked at commit c334a6d87bb19b23fa8c3374e7ff59213dc87e49 of tgxs002/CORA (Apache-2.0). The repository is the official code linked from the abstract. Its README states that the region-prompt training code is on a separate region branch and that the CLIP-aligned labeling code is "released soon" in another branch, so neither is in this checkout. We did not clone the branches. Relabelled annotations and region prompts are provided separately and are not in the repository.

• Anchor pre-matching uses the layer-4 map in training and inference, while the final classifier at inference uses the layer-3 map. Pre-matching always pools from the layer-4 feature map with 7×7 RoIs [code: models/fast_detr.py#L188-L201]. At inference the final region classifier pools from layer 3 with 14×14 RoIs and then applies layer 4 [code: models/fast_detr.py#L338-L354]. The paper describes one path (first three blocks, RoIAlign, last block) §3.2.
• Region prompt placement. The paper's text puts the last CLIP block after pooling and before the prompt §3.2, but its equation adds the prompt inside the attention pooling eq. 1. The code adds a learned positional-embedding table of size S²+1 to the tokens inside CLIP's attention pool, after layer 4 model.py, loaded from region_prompt_path backbone.py. The extra row is a mean token; the paper does not mention it.
• Region prompt extraction, not training. The repository provides a script that copies the positional-embedding entries from a trained detector checkpoint into a file extract_attn_pool.py. The prompt-training loop described in §3.2 and §4.2 is not in this checkout.
• Class-agnostic detector head. The paper's classification is a matching probability to the pre-matched class §3.3. The code trains a single objectness output for every query: training sets all target labels to zero before the loss engine.py and the head has one output fast_detr.py. Class identity enters only through the per-query pre-matched class matcher.py.
• Matching is one Hungarian pass with masks, not one per class. Pairs whose pre-matched class differs from the ground-truth label get cost 99 [code: models/matcher.py#L127-L129]. A candidate must also have GIoU > 0 with at least one correctly classified query matcher.py. Neither rule is in the paper.
• Inference score. The paper scores a box as matching probability times cosine eq. 7. The code takes a softmax over cosine × 100 across classes with a background logit of 0, then multiplies by objectness [code: models/fast_detr.py#L416-L419]. Temperature 0.01 app. A matches the eval_tau default of 100 fast_detr.py.
• ×8 novel scaling applies to pre-matching only. The code multiplies the softmax probabilities of a fixed list of 17 COCO class indices by 8 during inference, and only in the pre-matching classifier [code: models/fast_detr.py#L237-L246]. The final classifier at inference is not scaled fast_detr.py. The paper says the logit is multiplied app. A. The code does not name the classes behind the indices.
• Matching equation. Equation 4 is written as an arg max of a cost eq. 4, while the text says the cost is minimised §3.3. The code minimises, via Hungarian assignment matcher.py.
• Class dropout. The code draws a random half of the class embeddings for each class independently with probability 0.5 and only on split iterations fast_detr.py engine.py. Split iterations duplicate the batch engine.py and repeat the features fast_detr.py, so they cost roughly twice the compute of an ordinary iteration. The paper does not state this cost.
• Ground-truth class oracle. The flag --class_oracle restricts pre-matching to ground-truth classes fast_detr.py. It is off by default engine.py and the configs do not set it.
• Detection output. The top 100 class-score pairs are kept per image [code: models/fast_detr.py#L799]. The paper does not state an output limit.
• Pseudo-labels for CORA+. The code keeps boxes with score at least 0.2 by default export_pseudo_label.py. The paper does not state a threshold.
• LVIS and CORA+ configs. The repository has only COCO configs [code: configs/COCO/]. The LVIS 100-class sampling is available as --num_label_sampled engine.py, and the cosine relaxation as --matching_threshold, which is off by default matcher.py. We could not confirm the 0.7 value or that these flags reproduce the LVIS run.
• Region features in training. The training-time classifier and the decoder's region features use torch.no_grad fast_detr.py. The prompt is therefore only learned in the separate prompt stage, which is consistent with the paper's frozen-backbone claim §3.2.

Pitfalls

• Result text does not match the tables in three places. (a) The text says the full anchor pre-matching adds 10.8 AP50 over the previous row §4.5. Table 5 gives 29.9 → 40.9 (+11.0) or 29.9 → 41.7 (+11.8), so neither gives 10.8 tab. 5. (b) The introduction says LVIS gives "22.2/28.1 APr with/w.o. extra data" §1. Table 2 has 22.2 without extra data and 28.1 with IN-21k tab. 2, so the order is reversed. (c) Table 4 lists regional features at 58.6 novel and 58.2 base tab. 4. Table 3 lists the same CLIP RN50 region classifier at 58.2 novel and 58.6 base, and the text uses 58.2 for novel §4.4. Treat the two tables as inconsistent and use the cited number from the text if you need one.
• Ambiguous placement of the region prompt: it can be added before or after the last CLIP block (see Paper vs code). The code's placement is the attention-pool positional table.
• Pre-matching is not run with the same features at train and test time. See Paper vs code. A reimplementation must reproduce both paths or it will drift between training and inference.
• The ×8 novel scaling needs the exact class list. The code hard-codes 17 indices for COCO and a separate index list for LVIS [code: models/fast_detr.py#L238-L243]. The paper does not name the classes, so without those lists the scaling cannot be reproduced.
• The class-agnostic objectness head, not the class head, is what is trained. A reimplementation that adds a per-class head, following eq. 7 literally, will not reproduce the code's behaviour.
• Relabelled annotations and region prompts are provided separately (README, Data Preparation). The relabelling rule is not in the default branch; the included relabel.py only applies exported labels [code: relabel.py].
• Cost: the paper's efficiency claim is for anchor pre-matching against conditional matching §3.3 fig. 3. The region classifier still runs on every query, and class-dropout iterations double the batch [code: engine.py#L113-L114].
• Numbers are author-reported on COCO and LVIS with CLIP backbones of different sizes and different extra data. Table 1 mixes RN50, RN50x4 and ViT-B/32 backbones; compare within a backbone row group.
• Dataset split and novel-class names follow the cited ZSD convention §4.1. The class names of the 17 novel classes are not listed in the paper text.

Lineage

• Builds on: CLIP for the text-image embedding and the frozen image encoder §1; DAB-DETR for the localizer §3.3; DETR for the set-prediction framework §2.
• Compared against: OVR-CNN, Detic, RegionCLIP, VL-PLM, ViLD, OV-DETR, MEDet, CLIP-Adapter and CoOp tab. 1 tab. 3.
• Relates to conditional matching in OV-DETR, which CORA replaces with anchor pre-matching §3.3 fig. 3.
• Superseded by: not stated in the paper, and not known from content already in this wiki.