Open-Set Recognition: a Good Closed-Set Classifier is All You Need?

In plain words

A normal image classifier is trained to pick one of a fixed set of labels. In the real world it also meets things it was never trained on, and it should say "none of these" instead of guessing a known label. This task is called open-set recognition. The usual baseline reads the classifier's confidence and calls an image unknown when that confidence is low.

The paper's first finding is that a classifier that is better at the known labels is also better at spotting unknowns. The two abilities move together, so improving the ordinary classifier helps the "none of these" decision too. The authors use standard training tricks (longer training, a better learning-rate schedule, stronger data augmentation, and a small label-smoothing change). They also use the raw score before the softmax step, which is the largest raw output value. With these changes, the plain baseline matches or beats several specialised open-set methods.

The paper also argues that the usual test sets are too small and vague about what a "new class" is. It proposes a benchmark built from fine-grained sets (birds, cars, aircraft) and from ImageNet, where unseen classes are grouped by how similar they are to the known ones.

TL;DR

• Closed-set accuracy and open-set AUROC are strongly correlated for a VGG32 classifier across standard benchmarks (Pearson 0.95) and weaker but still positive on ImageNet §3.1 §3.2.
• Training a softmax baseline longer with cosine restarts, RandAugment and warmup, and scoring with the maximum logit, beats the best prior published number on 4 of 6 standard benchmarks §4 tab. 1.
• On the proposed Semantic Shift Benchmark the maximum-logit baseline performs about the same as ARPL+ §5.2 tab. 3.
• Relevant if you need a strong, simple unknown-detection baseline. Most numbers are author-reported and use small VGG32 models or a single ResNet50 setup.

Problem

Open-set recognition asks a model to classify known classes and, for each input, decide whether it belongs to any known class. Evaluation is binary (known/unknown) and reported as AUROC §3. Standard benchmarks re-purpose small datasets (MNIST, SVHN, CIFAR-10, CIFAR+10/+50, TinyImageNet) with a fixed split of known and unknown classes; the paper argues these splits lack a precise notion of a "semantic" class and are small §1 §5.

The common baseline is the maximum softmax probability (MSP) of a cross-entropy classifier §3.1. Earlier work reported this baseline at low AUROC; the paper attributes part of the gap to short training, saying the source's models were trained for 30 epochs according to its public code app. C.

Method

1. Score functions. Closed-set classification returns p(y | x) over the known classes. For open-set detection the MSP baseline uses the maximum softmax probability as the known-class score §3.1. The paper's proposed score is the maximum logit (MLS): the largest raw output of the last linear layer, before softmax §4.

2. Correlation study. Several methods (MSP, ARPL, ARPL+CS) are trained on the standard benchmarks with the same hyperparameters as ARPL, on five known/unknown splits per dataset. Open-set AUROC is plotted against closed-set top-1 accuracy §3.1 fig. 2. The paper motivates the correlation with calibration: a model that is well calibrated with a proper scoring rule should show low confidence on inputs it gets wrong, and the authors treat unknown inputs as such cases §3.1 eq. 1.

3. Stronger closed-set training (the "+" recipe). Applied to the VGG32 backbone on each standard benchmark §4 app. D:

• batch size 128, 600 epochs, single model (no ensemble) in the main table §4;
• learning-rate warmup, then cosine annealing with two warm restarts (at epochs 200 and 400) app. D;
• RandAugment with per-dataset magnitude and number of ops, tuned on a validation split of one known/unknown split app. D;
• label smoothing, applied only to TinyImageNet (value 0.9) app. D;
• the MLS score from step 1.

The same training recipe is applied to OSRCI and to ARPL+CS (OSRCI+ and (ARPL+CS)+), and for OSRCI the learning rates are divided by 10 relative to the baselines app. D. An ablation on TinyImageNet adds the steps one at a time and reports the effect on closed-set accuracy, AUROC and OSCR app. C fig. 4.

4. Semantic Shift Benchmark (SSB). Three fine-grained datasets are used: CUB-200 birds (312 binary attributes per image), Stanford Cars and FGVC-Aircraft §5.1. Given a chosen set of known classes, the remaining classes are binned by attribute similarity to the known set into Easy (least similar), Medium and Hard (most similar); the bin sizes are not stated §5.1 fig. 5 tab. 2. Each attribute vector is a row of a class-by-attribute frequency matrix, and similarity is computed between rows §5.1.

For ImageNet, the known classes are ImageNet-1K, and the open classes come from the disjoint classes of ImageNet-21K-P. Each open class gets a total semantic distance to all known classes, summed over path distances in the ImageNet semantic tree. Sorting these totals gives two sets of 1000 open classes. The paper names the sets Easy and Hard; by our reading of the names, the set with the smallest totals is Hard and the one with the largest is Easy §5.1.

5. Evaluation. AUROC for known/unknown, closed-set accuracy, and OSCR (Dhamija et al.), which trades off closed-set accuracy against open-set detection as the threshold varies §5.2. For the SSB, Medium and Hard are merged into one bin wherever Medium exists §5.2.

# Strengthened closed-set training step  [§4] [app. D]
# Input:  images (B, 3, H, W); labels in the C known classes
# Output: updated model parameters

def train_step(model, optimizer, images, labels, smoothing):
    # Augment with RandAugment, then a 4-pixel-padded random crop and a
    # horizontal flip  [app. D] [code: data/augmentations/__init__.py#L99-L119]
    images = augment(images)
    # Feature (B, 128) from the VGG32 backbone, then a bias-free linear head
    # [app. D] [code: models/classifier32.py#L38-L86]
    logits = model(images)
    # Cross-entropy against a smoothed target: the true class gets 1 - s
    # and the other classes share s equally  [app. D]
    # [code: methods/ARPL/loss/LabelSmoothing.py#L36-L44]
    loss = smoothed_cross_entropy(logits, labels, smoothing)
    # Backward pass and an optimizer step (SGD for all but TinyImageNet)
    # [code: methods/ARPL/osr.py#L92-L95]
    loss.backward()
    optimizer.step()

# Learning rate at a given epoch  [app. D] [code: utils/schedulers.py#L28-L35]
# Input:  epoch index e, base learning rate lr0 (0.1; 0.01 on TinyImageNet)
# Output: learning rate for epoch e

def learning_rate(e, lr0):
    # Linear warmup from a small value; the paper says 20 epochs, the code 10
    # [code: utils/schedulers.py#L33]
    if e < warmup_epochs:
        return linear_warmup(e, lr0)
    # Cosine annealing with restarts every 200 epochs, floor at lr0 / 1000
    # [app. D] [code: utils/schedulers.py#L99-L103]
    return cosine_restart(e, lr0, period=200, floor=lr0 * 1e-3)

# Maximum logit score (MLS) for open-set detection  [§4]
# Input:  logits (B, C) from the classifier on test images
# Output: score (B,); higher means "known"

def mls_score(logits):
    # Largest raw logit per image; softmax is not applied at test time
    # [code: methods/ARPL/osr.py#L84-L85] [code: methods/ARPL/core/test.py#L65]
    return logits.max(dim=1)

# ImageNet Easy/Hard open-set split  [§5.1]
# Input:  candidate unknown classes U, known classes C, tree distance d
# Output: Easy and Hard sets of 1000 unknown classes each

def semantic_split(candidates, known):
    # Sum of tree-path distances from each candidate to all known classes
    # [§5.1]
    total = {u: sum(d(u, c) for c in known) for u in candidates}
    # Sort ascending: the closest candidates are Hard, the farthest Easy
    # [§5.1] (ends inferred from the names)
    ranked = sorted(candidates, key=lambda u: total[u])
    return ranked[-1000:], ranked[:1000]

Hyperparameters

Standard benchmarks (VGG32 backbone, MSP+ and MLS). The paper gives most values in its appendix; the per-dataset RandAugment and label-smoothing values are in the code, not in the paper's text.

Per-dataset values for the Softmax (MSP+/MLS) setting, from the code's table. Image size is the input side length; RandAugment N and M are the number of ops and magnitude paper_hyperparameters.csv:

Fine-grained (SSB) and ImageNet settings app. D:

• FGVC: ResNet50, input 448x448, initialised from MoCo v2 weights trained on Places §5.2, learning rate 0.001, batch 32, 600 epochs, cosine with warmup, RandAugment and label smoothing re-tuned app. D osr_finegrained_train.sh.
• ImageNet MLS: a pre-trained timm ResNet50 is evaluated directly, with no training app. D.
• ImageNet ARPL+: fine-tuned from that checkpoint for 10 epochs with the ARPL loss app. D.
• Compute: 2 to 6 hours per split on a 12 GB GPU for the standard benchmarks; 7.5 hours per split for ARPL+CS on TinyImageNet; one to two days per FGVC dataset app. D.

Results

Author-reported throughout. Setups: VGG32 on standard benchmarks, five known/unknown splits, AUROC averaged over the splits tab. 1. SSB: ResNet50 on fine-grained sets and ImageNet-1K tab. 3.

Headline correlation. Pearson ρ = 0.95 between closed-set accuracy and AUROC on the standard benchmarks §3.1 fig. 2. On ImageNet-scale evaluation ρ = 0.88 (Hard) and 0.63 (Easy) across model families; within the ResNet family ρ = 1.00 (Easy) and 0.99 (Hard) §3.2 fig. 3. ViT deviates from the trend in both splits §3.2.

Standard benchmarks, AUROC (%) tab. 1:

The "+" rows add the closed-set recipe. Gains are in the table's own deltas. The paper reports MLS with an average AUROC increase of 15.6 points over the reported MSP baseline and states it is 0.7 points better on average than the rest of the suite §4. The 0.7 figure matches our recomputation against the best prior result per dataset; the paper does not name its comparator. MLS is above the best prior published value on SVHN, CIFAR+10, CIFAR+50 and TinyImageNet §4 tab. 1; on CIFAR+10 the (ARPL + CS)+ row is higher (98.1 against 97.9). (ARPL+CS)+ averages 0.03 points below MLS §4.

TinyImageNet ablation, VGG32, five splits app. C fig. 4:

The ablation's correlation between closed-set accuracy and AUROC is ρ = 0.93 app. C.

Semantic Shift Benchmark, closed-set accuracy and AUROC / OSCR on Easy / Hard splits (Medium merged into Hard where present) tab. 3:

Split statistics: the paper gives class counts (test images in parentheses) for known and open classes tab. 2:

On the SSB, the paper says the harder ImageNet split lowers AUROC by 5 to 6 points for both methods; randomly adding open classes (1K, then 10K) lowers it by about 0.6 points §5.2. MLS and ARPL+ are close: MLS is higher on CUB and Aircraft, while ARPL+ is higher on ImageNet and on both Cars splits tab. 3.

Paper vs code

Repository: sgvaze/osr_closed_set_all_you_need (MIT, copyright Sagar Vaze, a paper author; linked from the project page at robots.ox.ac.uk/~vgg/research/osr/ as "[Code]"). The commit is the one cloned at compile time. The paper contains no code link of its own.

• Warmup length. The paper says the warmup rises linearly to the initial rate at epoch 20 app. D. The code uses warmup_epochs=10 and sets a target from the cosine curve at epoch 10, so it reaches about 99.4% of the base rate, not the base rate itself schedulers.py schedulers.py.
• First epoch. After the scheduler is constructed, the code sets the learning rate to its floor (base × 1e-3), and the rate is only updated at the end of each epoch schedulers.py osr.py. The first epoch therefore trains at the floor rate. The paper does not describe this.
• Optimizer. The paper does not name one. The code uses SGD with momentum 0.9 and the weight decay argument for all datasets except TinyImageNet, which uses Adam and ignores weight decay osr.py.
• Weight decay. Not stated in the paper for the reported runs; the code default is 1e-4 osr.py. The TinyImageNet script passes 1e-4 osr_train_tinyimagenet.sh, but that run uses Adam, which is built without weight decay osr.py.
• Evaluation split. With split_train_val set to False (the default and the value in the shell scripts), the per-epoch "validation" loader is the known-class test split, and the final-epoch result is returned without early stopping osr.py osr.py cifar.py. The paper describes tuning on a 20% held-out validation split app. D, so the released run does not show that tuning step.
• Balanced evaluation. The code subsamples the larger of the known and unknown test sets so the two are equal in size before computing AUROC osr.py tinyimagenet.py. The paper does not state this.
• ImageNet evaluation transform. The ImageNet test script defines a standard transform helper with bicubic interpolation, a crop percentage of 0.875 and ImageNet mean and std openset_test_imagenet.py. The paper does not state the transform.
• Score. The ImageNet test wrapper returns the negative of the maximum logit as the unknown score when its use_softmax flag is off, which is its default; that ranks the same way as MLS openset_test_imagenet.py.
• Batch and epochs. Batch sizes, epochs, base learning rates and label smoothing match the paper's values paper_hyperparameters.csv app. D.

Pitfalls

• Warmup and first epoch. Following the paper's 20-epoch warmup gives a slower start than the code. Following the code gives a 10-epoch warmup and a one-epoch floor-rate start. An implementation has to pick one, and the paper does not say which it meant.
• Optimizer for TinyImageNet. The paper's learning rate of 0.01 is paired with Adam in the code, not SGD. An implementation that uses SGD for all datasets may not match the TinyImageNet numbers.
• Label smoothing convention. The code's s is the mass removed from the true class: the target puts 1 - s on the label and s / (C - 1) on each other class. With s = 0.9 on TinyImageNet, the true class gets only 0.1 of the target mass LabelSmoothing.py. Reading s as the mass on the true class would give a very different target.
• Missing dropout rate and bias. The paper names the dropout positions but not the rate; the code uses 0.2 classifier32.py. The linear head has no bias classifier32.py. The appendix's bias-free note refers to the feature-space visualisation app. B.
• Per-dataset RandAugment values are only in code. The magnitude M and count N for each dataset are in the repository's table paper_hyperparameters.csv, not in the paper text. The default N=1, M=9 in the augmentation code is overwritten by these values when they are passed [code: data/augmentations/init.py#L112] osr.py.
• Augmentation pipeline. RandAugment is applied first, then a resize, a 4-pixel-padded random crop, a horizontal flip and normalisation [code: data/augmentations/init.py#L99-L119]. None of these steps are described in the paper's text.
• Baselines are not reproduced. The MSP baseline numbers in Table 1 are taken from the earlier OSRCI paper; the authors say that source's models trained for 30 epochs according to its public code, while the MSP+ and MLS rows use 600 epochs app. C tab. 1. Comparisons therefore mix published and re-run numbers.
• Comparator for "0.7 points". The paper's average-advantage claim does not name a comparator. Against the best prior result for each dataset, MLS is about 0.7 points higher on average. Against (ARPL+CS) alone, the gap is about 1.6 points (our arithmetic on tab. 1).
• Balanced AUROC. AUROC is computed on equal-size known and unknown sets in the code, so the numbers depend on the random subsample osr.py. The subsampling uses np.random.choice tinyimagenet.py and the code does not show whether the sample is with or without replacement.
• SSB split direction. The paper does not state which end of the ImageNet ranking is "Hard"; the name implies the closest classes §5.1. Path distance in the tree is also not defined precisely (edge count or weighted), and the paper approximates the total with a sum over all known classes §5.1.
• SSB "Hard" bins. For CUB and Aircraft the Hard column merges Medium and Hard, so it is not a single difficulty level §5.2 tab. 2. Cars has no Medium split.
• ARPL+CS is not run on the SSB. It needs batch size 2 on a 12 GB GPU and was judged too expensive; ARPL+ was used instead app. D §5.2.
• Compute is large for the "+" recipe. 600 epochs with a ResNet50 at 448 px takes one to two days per FGVC dataset app. D.
• Correlation is weaker across families. On ImageNet the correlation is 0.63 for Easy, and ViT does not follow the trend §3.2. The strong ρ values are mostly within the ResNet family or on the small benchmarks §3.2.
• Calibration argument is heuristic. The link between calibration and open-set performance is motivated by an assumption that generalisation error correlates with infinite-data loss §3.1. The paper does not test it directly.

Lineage

• Builds on: the maximum-softmax-probability baseline of Hendrycks and Gimpel (2017) §3.1; the reciprocal-point work RPL of Chen et al. (2020) and its extension ARPL of Chen et al. (2021) with confusing samples (ARPL + CS) §3.1 §4.
• Compared against: OSRCI of Neal et al. (2018), OpenHybrid of Zhang et al. (2020) §4, and OoD results with Outlier Exposure in the appendix app. F.
• Superseded by: not stated in the paper.