---
kind: concept
slug: open-set-recognition
title: Open-set recognition
summary: >-
  Classify known classes and reject inputs from classes never seen in training. One card: a stronger closed-set classifier scored by its maximum logit matches or beats prior open-set methods on standard benchmarks, and is within a few points of ARPL+ on a fine-grained semantic-shift benchmark. All numbers are author-reported.
tags: [open-set, classification, uncertainty, evaluation]
papers: ["2110.06207"]
related: [out-of-distribution-detection, anomaly-detection, calibration, data-augmentation, learning-rate-schedule, uncertainty-estimation]
status: draft
updated_at: 2026-10-24
---

## Overview

A classifier trained on a fixed set of labels should answer "unknown" for an
input from a class it never saw, instead of guessing a known label. This is
open-set recognition. The output is a known-class prediction plus a score for
"belongs to a known class", and detection quality is reported as AUROC between
known and unknown inputs [arxiv:2110.06207 §3].

The standard baseline is the maximum softmax probability (MSP) of a
cross-entropy classifier [arxiv:2110.06207 §3.1]. Its low reported scores come
partly from short training, not only from the scoring rule
[arxiv:2110.06207 app. C] [arxiv:2110.06207 tab. 1].

The paper's main claim is that open-set AUROC tracks closed-set accuracy. On
the standard small benchmarks the Pearson correlation is 0.95, so improving the
ordinary classifier also improves the unknown decision [arxiv:2110.06207 §3.1].
The correlation is weaker across architecture families and fails for ViT
[arxiv:2110.06207 §3.2]. This page rests on one paper card. It is closely
related to [out-of-distribution detection](/concepts/out-of-distribution-detection),
which asks the same question about inputs outside the training distribution
in a wider setting.

## Variants

### Maximum softmax probability (MSP)

Score each input by the largest softmax probability of a cross-entropy
classifier. This is the common reference point. The paper's MSP rows for
standard benchmarks are taken from an earlier paper, trained for 30 epochs
according to that paper's code, so they are not a like-for-like comparison with
the 600-epoch rows [arxiv:2110.06207 tab. 1] [arxiv:2110.06207 app. C].

### Maximum logit score (MLS) with a stronger closed-set recipe

Score each input by the largest raw logit, before softmax. The paper's
closed-set recipe is applied first: 600 epochs at batch size 128, warmup and
cosine annealing with two warm restarts, RandAugment with per-dataset
magnitude and count, and label smoothing on TinyImageNet only [arxiv:2110.06207
§4] [arxiv:2110.06207 app. D]. The same recipe is applied to the earlier
methods as well, shown as "+" rows in the paper's table [arxiv:2110.06207 tab. 1].

On standard benchmarks with a VGG32 backbone, MLS has an AUROC of 99.3 (MNIST),
97.1 (SVHN), 93.6 (CIFAR10), 97.9 (CIFAR+10), 96.5 (CIFAR+50) and 83.0
(TinyImageNet) [arxiv:2110.06207 tab. 1]. Against the published rows that do
not use the "+" recipe, it is the highest on SVHN, CIFAR+10, CIFAR+50 and
TinyImageNet. The "+" row for ARPL+CS is higher on CIFAR+10 (98.1) and CIFAR+50
(96.7) [arxiv:2110.06207 tab. 1] [arxiv:2110.06207 §4]. Card:
[Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](/papers/2110.06207).

### Fine-grained semantic-shift evaluation

The paper also proposes a harder test, the Semantic Shift Benchmark (SSB). It
uses fine-grained datasets (CUB birds, Stanford Cars, FGVC-Aircraft) and
ImageNet. Unknown classes are split into Easy and Hard by attribute similarity
to the known classes, or by tree distance for ImageNet [arxiv:2110.06207 §5.1]
[arxiv:2110.06207 tab. 2].

On this benchmark MLS is higher than ARPL+ on CUB (AUROC 88.3 / 79.3 against
83.5 / 75.5) and on FGVC-Aircraft (90.7 / 82.3 against 87.0 / 77.7). ARPL+ is
higher on both Stanford Cars splits and on ImageNet, by 0.6 to 1.4 points
[arxiv:2110.06207 tab. 3]. Use these numbers to compare methods on this
benchmark, not as a claim that the two are equal.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Need a strong open-set baseline and can retrain the classifier | MLS with the "+" recipe | Best or near-best on the standard benchmarks in the paper's table [arxiv:2110.06207 tab. 1] |
| Cannot retrain; have a pretrained ImageNet classifier | MLS on the pretrained model | The paper evaluates a pretrained timm ResNet50 with MLS and no training for its ImageNet row [arxiv:2110.06207 app. D] [arxiv:2110.06207 tab. 3] |
| Training budget is small | MSP or MLS on a shorter schedule, with the gap stated | The "+" recipe runs 600 epochs; the paper reports 2 to 6 hours per split on a 12 GB GPU for the small benchmarks and one to two days per fine-grained dataset with ResNet50 at 448 px [arxiv:2110.06207 app. D] |
| Fine-grained classes, where unknowns are semantically close | Evaluate on the SSB Easy and Hard splits | Standard splits lack a precise notion of "semantic" class; the SSB ranks unknowns by similarity [arxiv:2110.06207 §5.1] |
| Using closed-set accuracy to choose a model | Only within one architecture family | Across families the correlation weakens (ImageNet Easy ρ = 0.63) and ViT deviates [arxiv:2110.06207 §3.2] |
| Need a non-threshold metric | AUROC or OSCR | The paper reports both; OSCR trades closed-set accuracy against detection as the threshold varies [arxiv:2110.06207 §5.2] |

## Implementation sketch

The only method the paper proposes is the score. The training recipe is in
[arxiv:2110.06207 app. D]. Score and decision rule:

```pseudocode
# Maximum logit score for open-set detection  [arxiv:2110.06207 §4]
# Input:  logits (B, C) from the closed-set classifier
# Output: score (B,); higher means "known"

def mls_score(logits):
    # Largest raw logit per input; softmax is not applied
    return logits.max(dim=1)

# Decision: predict the argmax class if score >= t, else "unknown".
# The threshold t is not stated in the paper; AUROC is threshold-free [§5.2].
```

## Contradictions

None known across papers: only one card covers this concept, so cross-paper
conflicts cannot yet be checked. Within the paper, the card's summary describes
MLS and ARPL+ as "about the same" on the SSB [arxiv:2110.06207 §5.2]. Table 3
shows MLS 3.7 to 4.8 AUROC points higher on CUB and FGVC-Aircraft, and ARPL+
0.6 to 1.4 points higher on Stanford Cars and ImageNet [arxiv:2110.06207 tab. 3].
This page uses the table figures.

## Pitfalls

Taken from the paper's text and appendix; the released code differs in several
of these points, see [the paper card](/papers/2110.06207).

- **Warmup length.** The paper states 20 epochs of linear warmup [arxiv:2110.06207 app. D]. The released code uses 10 epochs, so the two do not match.
- **Optimizer.** The paper does not name one for the standard benchmarks. Its TinyImageNet learning rate of 0.01 pairs with an optimizer the paper does not name [arxiv:2110.06207 app. D].
- **Label smoothing convention.** The paper's smoothing value 0.9 for TinyImageNet should be read with the convention used in the code: the true class keeps only 1 − s of the target mass. Reading s as the mass on the true class gives a very different target [arxiv:2110.06207 app. D].
- **Evaluation split.** The paper describes tuning on a 20% held-out split [arxiv:2110.06207 app. D]. The released code evaluates on the known-class test split by default.
- **Balanced AUROC.** The released code subsamples the larger of the known and unknown test sets to equal size before computing AUROC. The paper does not state this step [arxiv:2110.06207 app. D].
- **SSB split direction.** The paper does not define "Hard" for ImageNet beyond the name, and tree distance is not defined precisely [arxiv:2110.06207 §5.1]. For CUB and FGVC-Aircraft, the Hard column merges Medium and Hard [arxiv:2110.06207 tab. 2].
- **ARPL+CS on the SSB.** It was not run because of its cost; ARPL+ was used instead [arxiv:2110.06207 §5.2] [arxiv:2110.06207 app. D].

## Papers

- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](/papers/2110.06207): shows that closed-set accuracy tracks open-set AUROC, proposes the maximum logit score and the Semantic Shift Benchmark.