Towards Open World Object Detection

In plain words

A normal object detector is trained on a fixed list of classes. When it sees something it was never taught, it usually squeezes the object into one of the known classes with high confidence. This paper asks for something more useful: the detector should say "this is unknown" for objects it has not learned, and then learn the real name once someone labels a few examples.

The proposed system, ORE, makes that possible in three steps. During training it pulls the internal features of each known class into a tight group and keeps the groups apart. It treats some background boxes that look like objects as stand-ins for "unknown" so that the unknown case is learned at all. At test time it compares how strongly the classifier responds against a learned range of responses for known and unknown objects. When new labels arrive, it retrains on a small saved set of old examples so older classes are not lost.

The authors also report that the same ideas help in a standard incremental detection benchmark, even though that was not the design goal.

TL;DR

• Open world object detection: detect unseen objects as "unknown", then learn them incrementally when labels arrive §3.
• ORE = Faster R-CNN (ResNet-50, C4) + contrastive clustering of RoI features §4.1 + pseudo-labelled unknowns from RPN objectness §4.2 + energy-based unknown identification with Weibull fits §4.3 + replay of a balanced exemplar set §4.4.
• Author-reported gains over Faster R-CNN on Pascal VOC + MS-COCO task splits: lower Wilderness Impact and A-OSE, and far better mAP on previously learned classes after incremental steps tab. 2.
• The paper's numbers are the authors'; several hyperparameters disagree between paper and code (see Paper vs code).

Problem

Standard detectors assume every class is known at training time. Instances of unseen classes are part of the training data as background, so the detector learns to call them background and, at test time, often assigns them to a known class with high confidence §1 §2. Earlier open set detection work mostly estimates uncertainty and does not learn new classes afterwards §2.

The paper formalises the open world setting as a sequence of tasks. At step t, classes from earlier tasks are known and classes of later tasks are unknown. Test instances of unknown classes must be labelled as unknown (label 0), the unknowns are sent to a human who names n new classes, and the model is updated without retraining from scratch §3. While learning task t, classes of earlier tasks and of task t count as known, and classes of later tasks count as unknown §5.1.

Method

ORE re-uses a standard Faster R-CNN §4 and changes three things: the RoI features are clustered, the RPN and RoI classifier are trained with pseudo unknowns, and the classifier logits are converted into an energy score §4.1 §4.2 §4.3. Incremental learning uses a fixed-size exemplar memory §4.4.

1. Feature store. For each class (including unknown) keep a fixed-length queue of recent RoI features §4.1. The queue length is Q; the paper uses Q = 20 app. A.
2. Prototypes. After a burn-in of I_b iterations, prototypes are class-wise means of the queues. Every I_p iterations they are refreshed with a momentum step controlled by eta §4.1 alg. 1 app. F.
3. Contrastive clustering loss. For a feature f of class c, the loss sums a distance to its own prototype with a hinge term that pushes every other prototype beyond a margin Delta eq. 1 §4.1. The loss is added to the usual Faster R-CNN losses and trained end to end §4.1 app. E.
4. Unknown pseudo-labels from the RPN. The RPN is class-agnostic. Among the background proposals that do not overlap a ground-truth object, the top-k by objectness are labelled as unknown and used for the clustering loss and the classifier §4.2. The paper uses k = 1 app. F.
5. Energy score. The energy of a feature is the negative temperature-scaled log-sum-exp of the known-class logits eq. 2 eq. 4 §4.3. The paper connects softmax outputs to this energy through a Gibbs distribution eq. 3.
6. Unknown decision. Two shifted Weibull distributions are fitted, one to the energies of known and one to the energies of unknown samples. A detection is labelled unknown when the unknown density is larger at its energy §4.3 fig. 3.
7. Incremental step. Keep a balanced exemplar set with at least N_ex instances per class. Fine-tune the model on the new labels plus the exemplars. N_ex = 50 was chosen on validation §4.4 §5.2 §6.2 tab. 5.

Pseudocode for the clustering loss, following alg. 1 and eq. 1:

# Clustering loss for one batch of RoI features  [alg. 1] [eq. 1]
# Input:  roi_features (N, 2048); gt_classes (N,), 0 = unknown
# Output: scalar loss

def clustering_loss(roi_features, gt_classes, iteration):
    # Nothing is computed during burn-in  [alg. 1] [app. F]
    if iteration < I_b:
        return 0
    if iteration == I_b:
        # First prototypes are plain class-wise means of the queues  [alg. 1]
        prototypes = class_means(feature_store)
    elif iteration % I_p == 0:
        # Momentum refresh: old prototypes keep weight eta  [alg. 1]
        new_prototypes = class_means(feature_store)
        prototypes = eta * prototypes + (1 - eta) * new_prototypes
    loss = 0
    for feature, label in zip(roi_features, gt_classes):
        # Pull to the own prototype, push the others out to Delta  [eq. 1]
        for i, prototype in enumerate(prototypes):
            distance = D(feature, prototype)
            if i == label:
                loss += distance
            else:
                loss += max(0, Delta - distance)
    return loss

Pseudocode for the unknown pseudo-labels and the decision rule:

# Pseudo-label unknowns and decide unknown at test time  [§4.2] [§4.3]
# Input:  objectness (M,); overlaps_gt (M,) bool; energy_value (scalar)

def autolabel_unknowns(objectness, overlaps_gt, k):
    # Ignore boxes that overlap a ground-truth object  [§4.2]
    candidates = where(overlaps_gt, -inf, objectness)
    # Top-k remaining boxes by objectness become unknown  [§4.2] [app. F]
    return top_k_indices(candidates, k)

def is_unknown(energy_value, known_density, unknown_density):
    # Compare the two fitted densities at this sample's energy  [§4.3]
    return unknown_density(energy_value) > known_density(energy_value)

Key equations, as printed in the paper:

\mathcal{L}_{cont}(\bm{f}_c) = \sum_{i=0}^{C} \ell(\bm{f}_c, \bm{p}_i) eq. 1

E(\bm{f}; g) = -T \log \sum_{i=1}^{C} \exp\left(\frac{g_i(\bm{f})}{T}\right) eq. 4

Here C is the number of known classes, g_i the i-th classification logit and T the temperature. In eq. 1, ell is D(f, p_i) when i = c and max(0, Delta - D(f, p_i)) otherwise, where D is any distance and Delta a margin eq. 1 §4.1. The paper's free-energy definition (before the logit form) is eq. 2 and the softmax correspondence is eq. 3.

Hyperparameters

Values stated in the paper, with the code value next to them where it differs. "not stated" means the paper does not give the value.

Results

Author-reported. Setup: four-task protocol from Pascal VOC 2007 (task 1) and the remaining 60 MS-COCO classes (tasks 2 to 4), Faster R-CNN R-50 C4 as the baseline, evaluation on VOC 2007 test and COCO val §5.1 tab. 1. WI and A-OSE measure how often unknown objects are confused with known classes (lower is better); mAP measures known-class detection (higher is better) §5.1.

Main comparison tab. 2:

The table also lists an Oracle row that has all labels at each step. It is a reference, not a method tab. 2. The paper states that WI and A-OSE cannot be measured for task 4 because there are no unknown ground-truth boxes left §5.3.

Incremental object detection, Pascal VOC 2007 mAP at IoU 0.5 tab. 3:

Ablation on task 1 tab. 4: the full model (CC, ALU and EBUI on) gives WI 0.02193 and A-OSE 8234 at mAP 56.34. Removing CC and keeping ALU and EBUI gives WI 0.04539 and A-OSE 9011 tab. 4. Removing EBUI and keeping CC and ALU gives WI 0.05614 and A-OSE 12064 tab. 4.

Closed-set comparison with a dropout-sampling open-set detector (mAP on VOC 2007, with and without COCO unknown images) tab. 6: Faster R-CNN 81.86 and 77.09, dropout sampling 78.15 and 71.07, ORE 81.31 and 78.16.

Other reported numbers: training costs an extra 0.1349 s per iteration and inference 0.009 s per iteration over Faster R-CNN; the exemplar memory for N_ex = 50 takes about 34 MB app. H.

Paper vs code

Repository: https://github.com/JosephKJ/OWOD at commit 23890f188cd1a6801c6ac0e3dacd78b8572b8c29, linked from the paper's first page footnote §1. The repository is a Detectron2 fork with the ORE changes inside detectron2/. The code has the discrepancies below.

• Distance is not Euclidean and the margin is not Delta. The paper uses Euclidean distance and Delta = 10 app. F. The code computes torch.cdist(..., p=self.margin) with margin = 10.0 [code: detectron2/modeling/roi_heads/fast_rcnn.py#L600] [code: detectron2/config/defaults.py#L613]. That is an L10 norm, not Euclidean. The hinge margin is a hard-coded nn.HingeEmbeddingLoss(2) [code: detectron2/modeling/roi_heads/fast_rcnn.py#L481], so the effective margin is 2, not 10. The margin ablation in app. C therefore varied the norm order, not the margin.
• Loss reduction and weight. The paper sums the loss over prototypes eq. 1 and adds it with unit weight §4.1. The code uses PyTorch's hinge loss, which averages over all (feature, prototype) pairs [code: detectron2/modeling/roi_heads/fast_rcnn.py#L613], and multiplies the clustering loss by 0.1 fast_rcnn.py fast_rcnn.py. Expect different gradient scales.
• Momentum. The main-text momentum table tests eta from 0.4 to 0.9 and marks 0.9 as best app. B. The implementation section says 0.99 app. F. The code's base config uses 0.99 Base-RCNN-C4-OWOD.yaml, but the code default is 0.9 defaults.py.
• Zero prototypes for empty classes. At the first clustering step the code fills classes with empty queues with zero tensors [code: detectron2/modeling/roi_heads/fast_rcnn.py#L590-L598]. Because the list is shared with self.means, those classes then act as real prototypes at the origin in later steps. The paper does not discuss this case.
• Unknown class index. The paper writes the unknown prototype as index 0 eq. 1 §4.1. The code uses the last index, num_classes - 1, for unknown roi_heads.py [code: detectron2/evaluation/pascal_voc_evaluation.py#L56]. Index 0 is a known class in the code.
• Pseudo-label candidate set. The paper selects the top-k background proposals by objectness §4.2. The code selects the top-k from the RoI samples that survived the per-image subsampling, not from all RPN proposals roi_heads.py.
• Energy temperature. The paper uses T = 1 app. D. The fitting code computes T * logsumexp(logits / T) with T = cfg.OWOD.TEMPERATURE, whose default is 1.5 train_loop.py [code: detectron2/config/defaults.py#L621], and the energy-analysis config also sets 1.5 t1_val.yaml. The evaluation code computes logsumexp(logits) with no temperature [code: detectron2/evaluation/pascal_voc_evaluation.py#L99]. The densities are therefore fitted on T = 1.5 values and applied to T = 1 values.
• Data for the Weibull fit. The paper fits the two Weibulls on a held-out validation split containing knowns and unknowns §4.3. The code dumps per-proposal logits during a short run over the validation set and labels as unknown the proposals whose class is the pseudo-unknown class [code: detectron2/engine/train_loop.py#L186-L204] [code: detectron2/modeling/roi_heads/roi_heads.py#L450-L455]. The "unknowns" in the fit are therefore the pseudo-labelled RPN boxes, not annotated unknown objects.
• Decision rule. The paper writes the unknown test as "xi_kn(f) < xi_unk(f)" §4.3. The code marks a detection unknown when the unknown density is larger than the known density at its energy [code: detectron2/evaluation/pascal_voc_evaluation.py#L101-L110]. Detections that the classifier calls unknown but that the energy test rejects are dropped, not moved to a known class pascal_voc_evaluation.py. The density is approximated by summing over a window of plus or minus 0.5 with 100 points pascal_voc_evaluation.py.
• Clustering feature dimension. The config option Z_DIMENSION (128) is passed to the box predictor but not used; the earlier autoencoder that used it is commented out fast_rcnn.py. The clustering runs on the 2048-dim feature, as the paper says app. E.
• Training schedule. The paper says batch 8, learning rate 0.01, 8 epochs (about 50k iterations) per task app. F. The base config uses 16 images per batch and learning rate 0.02 Base-RCNN-C4-OWOD.yaml. The task-1 config trains for 18k iterations t1_train.yaml.
• Unseen-class logits. The paper says unseen classes get a large negative logit v and does not give v §5.2. The code uses -10e10 [code: detectron2/modeling/roi_heads/fast_rcnn.py#L245].
• Exemplar selection. The paper says N_ex = 50 §5.2. The code's fine-tuning splits are prebuilt ImageSets lists [code: detectron2/data/datasets/builtin.py#L227]. I did not trace how those lists choose exemplars, so this is not checked.

Checked and consistent: the feature queue size (20), the burn-in and update intervals (1k and 3k), the Faster R-CNN R-50 C4 backbone, and one pseudo unknown per image app. A app. F Base-RCNN-C4-OWOD.yaml.

Pitfalls

• Read the code, not the symbols, for the clustering loss. The paper's equations are written per feature and summed over prototypes. The code averages a PyTorch hinge loss with a fixed margin of 2 and a p = 10 norm (see Paper vs code). An implementation that follows eq. 1 literally will not reproduce the reported numbers.
• Keep the temperature the same in fitting and in evaluation. Fit and evaluation must use the same T. The code fits on T = 1.5 values and evaluates on T = 1 values. Paper results use T = 1 app. D, and the authors report that T = 2 is slightly better than T = 1 for unknown identification app. D.
• The unknown pseudo-labels are not true unknowns. They come from high objectness background boxes that do not overlap ground truth §4.2. They can include real objects that are unlabelled. The authors describe the scheme as simple and say it works in practice §4.2.
• Momentum value is inconsistent in the paper. Use 0.9 for the table result and 0.99 for the code config. The text in app. F and the table in app. B disagree.
• Margin and norm order are confounded in the code. Changing MARGIN in the config changes the Lp norm order, not the hinge margin. If the margin ablation app. C used this key, it varied the norm order, not Delta.
• Exemplar memory costs. The exemplar set grows with classes and images. The paper reports about 34 MB for N_ex = 50 app. H. Going from 50 to 100 exemplars per class raised WI from 0.0154 to 0.0410 and A-OSE from 7772 to 11065, while the mAP over both known sets rose only from 38.98 to 39.24 tab. 5.
• Replay matters. Without replay, mAP on previously known classes falls from 56.16 to 4.076 in the Faster R-CNN baseline tab. 2. Compare against replay, not only against plain finetuning.
• Unseen classes are sometimes missed. The paper's qualitative examples show a giraffe classified as horse fig. 4 and small unknowns next to large known objects missed app. K.
• Terminology. The paper calls the open-set error "Absolute Open-Set Error" in the text §5.1 but "Average Open Set Error" in the caption of tab. 2. The code computes the absolute count [code: detectron2/evaluation/pascal_voc_evaluation.py#L257-L259].
• Hardware. The paper trains on 8 V100 GPUs app. F. The clustering code calls .cuda() on the prototype tensor [code: detectron2/modeling/roi_heads/fast_rcnn.py#L600], so CPU-only runs need changes.

Lineage

• Builds on: Faster R-CNN §4 for the detector; energy-based scores from Liu et al. §4.3; prototype and contrastive clustering in general §4.1.
• Compared against: Shmelkov et al. ILOD, Peng et al. Faster ILOD (incremental detection) §5.4 tab. 3; Miller et al. dropout sampling (open-set detection) §5.1 tab. 6; Faster R-CNN with finetuning tab. 2.