Open-set recognition

Overview

A classifier trained on a fixed set of labels should answer "unknown" for an input from a class it never saw, instead of guessing a known label. This is open-set recognition. The output is a known-class prediction plus a score for "belongs to a known class", and detection quality is reported as AUROC between known and unknown inputs 2110.06207 §3.

The standard baseline is the maximum softmax probability (MSP) of a cross-entropy classifier 2110.06207 §3.1. Its low reported scores come partly from short training, not only from the scoring rule 2110.06207 app. C 2110.06207 tab. 1.

The paper's main claim is that open-set AUROC tracks closed-set accuracy. On the standard small benchmarks the Pearson correlation is 0.95, so improving the ordinary classifier also improves the unknown decision 2110.06207 §3.1. The correlation is weaker across architecture families and fails for ViT 2110.06207 §3.2. This page rests on one paper card. It is closely related to out-of-distribution detection, which asks the same question about inputs outside the training distribution in a wider setting.

Variants

Maximum softmax probability (MSP)

Score each input by the largest softmax probability of a cross-entropy classifier. This is the common reference point. The paper's MSP rows for standard benchmarks are taken from an earlier paper, trained for 30 epochs according to that paper's code, so they are not a like-for-like comparison with the 600-epoch rows 2110.06207 tab. 1 2110.06207 app. C.

Maximum logit score (MLS) with a stronger closed-set recipe

Score each input by the largest raw logit, before softmax. The paper's closed-set recipe is applied first: 600 epochs at batch size 128, warmup and cosine annealing with two warm restarts, RandAugment with per-dataset magnitude and count, and label smoothing on TinyImageNet only [arxiv:2110.06207 §4] 2110.06207 app. D. The same recipe is applied to the earlier methods as well, shown as "+" rows in the paper's table 2110.06207 tab. 1.

On standard benchmarks with a VGG32 backbone, MLS has an AUROC of 99.3 (MNIST), 97.1 (SVHN), 93.6 (CIFAR10), 97.9 (CIFAR+10), 96.5 (CIFAR+50) and 83.0 (TinyImageNet) 2110.06207 tab. 1. Against the published rows that do not use the "+" recipe, it is the highest on SVHN, CIFAR+10, CIFAR+50 and TinyImageNet. The "+" row for ARPL+CS is higher on CIFAR+10 (98.1) and CIFAR+50 (96.7) 2110.06207 tab. 1 2110.06207 §4. Card: Open-Set Recognition: a Good Closed-Set Classifier is All You Need?.

Fine-grained semantic-shift evaluation

The paper also proposes a harder test, the Semantic Shift Benchmark (SSB). It uses fine-grained datasets (CUB birds, Stanford Cars, FGVC-Aircraft) and ImageNet. Unknown classes are split into Easy and Hard by attribute similarity to the known classes, or by tree distance for ImageNet 2110.06207 §5.1 2110.06207 tab. 2.

On this benchmark MLS is higher than ARPL+ on CUB (AUROC 88.3 / 79.3 against 83.5 / 75.5) and on FGVC-Aircraft (90.7 / 82.3 against 87.0 / 77.7). ARPL+ is higher on both Stanford Cars splits and on ImageNet, by 0.6 to 1.4 points 2110.06207 tab. 3. Use these numbers to compare methods on this benchmark, not as a claim that the two are equal.

When to use which

Implementation sketch

The only method the paper proposes is the score. The training recipe is in 2110.06207 app. D. Score and decision rule:

# Maximum logit score for open-set detection  [arxiv:2110.06207 §4]
# Input:  logits (B, C) from the closed-set classifier
# Output: score (B,); higher means "known"

def mls_score(logits):
    # Largest raw logit per input; softmax is not applied
    return logits.max(dim=1)

# Decision: predict the argmax class if score >= t, else "unknown".
# The threshold t is not stated in the paper; AUROC is threshold-free [§5.2].

Contradictions

None known across papers: only one card covers this concept, so cross-paper conflicts cannot yet be checked. Within the paper, the card's summary describes MLS and ARPL+ as "about the same" on the SSB 2110.06207 §5.2. Table 3 shows MLS 3.7 to 4.8 AUROC points higher on CUB and FGVC-Aircraft, and ARPL+ 0.6 to 1.4 points higher on Stanford Cars and ImageNet 2110.06207 tab. 3. This page uses the table figures.

Pitfalls

Taken from the paper's text and appendix; the released code differs in several of these points, see the paper card.

• Warmup length. The paper states 20 epochs of linear warmup 2110.06207 app. D. The released code uses 10 epochs, so the two do not match.
• Optimizer. The paper does not name one for the standard benchmarks. Its TinyImageNet learning rate of 0.01 pairs with an optimizer the paper does not name 2110.06207 app. D.
• Label smoothing convention. The paper's smoothing value 0.9 for TinyImageNet should be read with the convention used in the code: the true class keeps only 1 − s of the target mass. Reading s as the mass on the true class gives a very different target 2110.06207 app. D.
• Evaluation split. The paper describes tuning on a 20% held-out split 2110.06207 app. D. The released code evaluates on the known-class test split by default.
• Balanced AUROC. The released code subsamples the larger of the known and unknown test sets to equal size before computing AUROC. The paper does not state this step 2110.06207 app. D.
• SSB split direction. The paper does not define "Hard" for ImageNet beyond the name, and tree distance is not defined precisely 2110.06207 §5.1. For CUB and FGVC-Aircraft, the Hard column merges Medium and Hard 2110.06207 tab. 2.
• ARPL+CS on the SSB. It was not run because of its cost; ARPL+ was used instead 2110.06207 §5.2 2110.06207 app. D.

Papers

• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?: shows that closed-set accuracy tracks open-set AUROC, proposes the maximum logit score and the Semantic Shift Benchmark.