---
kind: concept
slug: open-vocabulary-detection
title: Open-vocabulary detection
summary: >-
  Detecting and naming object classes that were not in the training label set, by classifying boxes against text embeddings of class names given at test time. Covers one paper so far (CORA, frozen CLIP region classifier with region prompts and anchor pre-matching); its numbers are author-reported.
tags: [object-detection, open-vocabulary, clip, detr]
papers: ["2303.13076"]
related: [object-detection, set-prediction, bipartite-matching, focal-loss, open-vocabulary-segmentation, vision-language-pretraining, zero-shot-learning]
status: draft
updated_at: 2026-11-08
---

## Overview

A standard detector only outputs the classes it was trained on. Open-vocabulary
detection trains on boxes of base classes and must localise and classify boxes
of novel classes whose names are supplied at test time [arxiv:2303.13076 §3].

The invariant shared by the approaches is that class names enter as text
embeddings, and each box is scored by similarity to those embeddings. In CORA,
classification uses cosine similarity between a region feature and each class
embedding [arxiv:2303.13076 §3.3 eq. 2], so changing the list of names changes
the embeddings, not the trained weights.

Two obstacles recur. First, CLIP is trained on whole images, while a detector
classifies image regions, which gives a distribution gap [arxiv:2303.13076 §1]
[arxiv:2303.13076 fig. 2]. Second, a proposal network trained on base classes
localises novel objects poorly [arxiv:2303.13076 §1] [arxiv:2303.13076 §2]. The usual workaround for the first
obstacle is to crop each region and classify it as a separate image; the paper
says this is slower for overlapping boxes and loses context [arxiv:2303.13076
§3.2].

## Variants

Only one paper with a card is available for this page, so the variants below
are all CORA. Other open-vocabulary detectors are not covered here until their
papers have cards.

### Frozen CLIP region classifier with region prompts

The localiser is a DETR-style detector (DAB-DETR) whose queries are
pre-matched to a class before box regression. Each box is classified by a
frozen CLIP image encoder and text encoder [arxiv:2303.13076 §3.1]. Two
additions carry the method. Region prompting adds a learned prompt to pooled
region features before CLIP's attention pooling; only the prompt is trained,
by cross-entropy on base-class boxes [arxiv:2303.13076 §3.2] [eq. 1]. Anchor
pre-matching gives each query its most similar class, and the query is then
conditioned on that class's embedding, so one decoder pass covers all classes
[arxiv:2303.13076 §3.3] [eq. 2] [eq. 3]. Card:
[CORA](/papers/2303.13076).

The paper reports 35.1 novel AP50 on COCO for a RN50 backbone and 41.7 for
RN50x4, both without extra data [arxiv:2303.13076 tab. 1].

### Pseudo-label extension

The same paper's CORA+ uses a second detector to produce boxes on extra images
with COCO captions, and then labels those boxes with CORA [arxiv:2303.13076
§4.1]. The author-reported result is 43.1 novel AP50 with RN50x4 [arxiv:2303.13076
tab. 1]. On LVIS, CORA+ reports 28.1 APr with extra IN-21k data, against 22.2
for CORA without extra data [arxiv:2303.13076 tab. 2].

## When to use which

Choose by constraint. The paper reports no latency or memory figures for the
variants, so cost claims below are limited to what the paper says.

| Situation | Use | Why |
|---|---|---|
| Vocabulary changes at run time and retraining per vocabulary is not wanted | Class-embedding classifier, as in CORA | Names enter as text embeddings [arxiv:2303.13076 §3.3] [eq. 2]. The paper notes that methods needing novel names during training must be retrained when the vocabulary changes [arxiv:2303.13076 §4.3]. |
| Only base-annotated detection data, no extra images | CORA without extra data | Author-reported 41.7 novel AP50 with RN50x4 and no extra data [arxiv:2303.13076 tab. 1]. |
| Extra unlabelled images and a second detector are affordable | CORA+ | Author-reported 43.1 novel AP50 with RN50x4 [arxiv:2303.13076 tab. 1]. Pseudo-labels come from CORA itself [arxiv:2303.13076 §4.1]. |
| Smaller CLIP backbone is required | RN50 instead of RN50x4 | Novel AP50 falls from 41.7 to 35.1 [arxiv:2303.13076 tab. 1]. |
| Latency-critical with many candidate boxes | Plan for the cost of classifying every box | Boxes are re-classified by the CLIP region classifier at inference [arxiv:2303.13076 §3.3]. The paper's efficiency claim is relative to conditional matching, not an absolute speed [arxiv:2303.13076 §3.3] [arxiv:2303.13076 fig. 3]. |
| Limited engineering time for training | Budget for the prompt stage | Region prompts are trained in a separate stage, 5 epochs in the paper's setup [arxiv:2303.13076 §4.2]. The card records that the code for this stage is not in the public checkout, so the stage must be rebuilt from the paper. |

## Implementation sketch

Language-agnostic outline of the common path, with the CORA-specific step
marked. Shapes are in parentheses.

```pseudocode
# Open-vocabulary detection, shared path  [arxiv:2303.13076 §3.1]
# Input:  image; class names given at test time
# Output: boxes with a label and a score

def detect(image, class_names):
    # Embed each class name with the text encoder, normalised  [§3.3]
    class_embeds = normalise(text_encoder(class_names))   # (C, d)
    # Localiser proposes boxes; trained on base classes only  [§3.1]
    boxes, match_prob = localiser(image)                  # (N, 4), (N,)
    # Region feature for each box, with the learned region prompt  [§3.2]
    region_embeds = normalise(region_encoder(image, boxes))  # (N, d)
    # Cosine similarity to every class  [eq. 7]
    sims = region_embeds @ class_embeds.T                 # (N, C)
    scores = match_prob[:, None] * sims                   # (N, C)
    return boxes, scores
```

The step that differs between variants is the query conditioning. CORA
replaces the query with a learned projection of its pre-matched class
embedding [arxiv:2303.13076 §3.3] [eq. 3].

## Contradictions

The card for CORA is the only source here, so these are disagreements inside
that paper, and between the paper and its released code. Each side is cited.
None is resolved here.

- **Region prompt placement.** The text places the last CLIP block after
  pooling and before the prompt [arxiv:2303.13076 §3.2]. Equation 1 adds the
  prompt inside the attention pooling [arxiv:2303.13076 §3.2] [eq. 1]. The
  released code adds the prompt to the attention-pool positional table after
  the last block [arxiv:2303.13076 §3.2] (code-level detail is on the paper
  card).
- **Pre-matching feature path.** The paper describes one feature path for
  region features: first three CLIP blocks, RoIAlign, last block [arxiv:2303.13076
  §3.2]. The released code pre-matches from the last-stage map in training and
  inference, and its final classifier at inference pools from an earlier stage
  (code-level detail is on the paper card).
- **Novel-class scaling.** The paper says the novel-class logit is multiplied
  at inference [arxiv:2303.13076 app. A]. The released code applies the
  factor to the pre-matching classifier only, and the class list is not named
  in the paper [arxiv:2303.13076 app. A].
- **Matching equation.** Equation 4 is written as an arg max, while the text
  says the cost is minimised [arxiv:2303.13076 §3.3] [eq. 4].
- **Result text vs tables.** The text gives a +10.8 AP50 gain from full anchor
  pre-matching [arxiv:2303.13076 §4.5]. Table 5 gives +11.0 or +11.8 between
  the relevant rows [arxiv:2303.13076 tab. 5]. The introduction lists the LVIS
  numbers in the reverse order of Table 2 [arxiv:2303.13076 §1] [tab. 2].
- **Region classification numbers.** Table 4 lists regional features at 58.6
  novel and 58.2 base [arxiv:2303.13076 tab. 4]. Table 3's CLIP RN50 row lists
  58.2 novel and 58.6 base [arxiv:2303.13076 tab. 3]. The text uses 58.2 for
  novel [arxiv:2303.13076 §4.4].

## Papers

- [CORA](/papers/2303.13076): frozen CLIP region classifier with region prompts and anchor pre-matching for DETR-style open-vocabulary detection; author-reported COCO and LVIS results.