Open-vocabulary detection

Overview

A standard detector only outputs the classes it was trained on. Open-vocabulary detection trains on boxes of base classes and must localise and classify boxes of novel classes whose names are supplied at test time 2303.13076 §3.

The invariant shared by the approaches is that class names enter as text embeddings, and each box is scored by similarity to those embeddings. In CORA, classification uses cosine similarity between a region feature and each class embedding 2303.13076 §3.3 eq. 2, so changing the list of names changes the embeddings, not the trained weights.

Two obstacles recur. First, CLIP is trained on whole images, while a detector classifies image regions, which gives a distribution gap 2303.13076 §1 2303.13076 fig. 2. Second, a proposal network trained on base classes localises novel objects poorly 2303.13076 §1 2303.13076 §2. The usual workaround for the first obstacle is to crop each region and classify it as a separate image; the paper says this is slower for overlapping boxes and loses context [arxiv:2303.13076 §3.2].

Variants

Only one paper with a card is available for this page, so the variants below are all CORA. Other open-vocabulary detectors are not covered here until their papers have cards.

Frozen CLIP region classifier with region prompts

The localiser is a DETR-style detector (DAB-DETR) whose queries are pre-matched to a class before box regression. Each box is classified by a frozen CLIP image encoder and text encoder 2303.13076 §3.1. Two additions carry the method. Region prompting adds a learned prompt to pooled region features before CLIP's attention pooling; only the prompt is trained, by cross-entropy on base-class boxes 2303.13076 §3.2 eq. 1. Anchor pre-matching gives each query its most similar class, and the query is then conditioned on that class's embedding, so one decoder pass covers all classes 2303.13076 §3.3 eq. 2 eq. 3. Card: CORA.

The paper reports 35.1 novel AP50 on COCO for a RN50 backbone and 41.7 for RN50x4, both without extra data 2303.13076 tab. 1.

Pseudo-label extension

The same paper's CORA+ uses a second detector to produce boxes on extra images with COCO captions, and then labels those boxes with CORA [arxiv:2303.13076 §4.1]. The author-reported result is 43.1 novel AP50 with RN50x4 [arxiv:2303.13076 tab. 1]. On LVIS, CORA+ reports 28.1 APr with extra IN-21k data, against 22.2 for CORA without extra data 2303.13076 tab. 2.

When to use which

Choose by constraint. The paper reports no latency or memory figures for the variants, so cost claims below are limited to what the paper says.

Implementation sketch

Language-agnostic outline of the common path, with the CORA-specific step marked. Shapes are in parentheses.

# Open-vocabulary detection, shared path  [arxiv:2303.13076 §3.1]
# Input:  image; class names given at test time
# Output: boxes with a label and a score

def detect(image, class_names):
    # Embed each class name with the text encoder, normalised  [§3.3]
    class_embeds = normalise(text_encoder(class_names))   # (C, d)
    # Localiser proposes boxes; trained on base classes only  [§3.1]
    boxes, match_prob = localiser(image)                  # (N, 4), (N,)
    # Region feature for each box, with the learned region prompt  [§3.2]
    region_embeds = normalise(region_encoder(image, boxes))  # (N, d)
    # Cosine similarity to every class  [eq. 7]
    sims = region_embeds @ class_embeds.T                 # (N, C)
    scores = match_prob[:, None] * sims                   # (N, C)
    return boxes, scores

The step that differs between variants is the query conditioning. CORA replaces the query with a learned projection of its pre-matched class embedding 2303.13076 §3.3 eq. 3.

Contradictions

The card for CORA is the only source here, so these are disagreements inside that paper, and between the paper and its released code. Each side is cited. None is resolved here.

• Region prompt placement. The text places the last CLIP block after pooling and before the prompt 2303.13076 §3.2. Equation 1 adds the prompt inside the attention pooling 2303.13076 §3.2 eq. 1. The released code adds the prompt to the attention-pool positional table after the last block 2303.13076 §3.2 (code-level detail is on the paper card).
• Pre-matching feature path. The paper describes one feature path for region features: first three CLIP blocks, RoIAlign, last block [arxiv:2303.13076 §3.2]. The released code pre-matches from the last-stage map in training and inference, and its final classifier at inference pools from an earlier stage (code-level detail is on the paper card).
• Novel-class scaling. The paper says the novel-class logit is multiplied at inference 2303.13076 app. A. The released code applies the factor to the pre-matching classifier only, and the class list is not named in the paper 2303.13076 app. A.
• Matching equation. Equation 4 is written as an arg max, while the text says the cost is minimised 2303.13076 §3.3 eq. 4.
• Result text vs tables. The text gives a +10.8 AP50 gain from full anchor pre-matching 2303.13076 §4.5. Table 5 gives +11.0 or +11.8 between the relevant rows 2303.13076 tab. 5. The introduction lists the LVIS numbers in the reverse order of Table 2 2303.13076 §1 tab. 2.
• Region classification numbers. Table 4 lists regional features at 58.6 novel and 58.2 base 2303.13076 tab. 4. Table 3's CLIP RN50 row lists 58.2 novel and 58.6 base 2303.13076 tab. 3. The text uses 58.2 for novel 2303.13076 §4.4.

Papers

• CORA: frozen CLIP region classifier with region prompts and anchor pre-matching for DETR-style open-vocabulary detection; author-reported COCO and LVIS results.