MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space

In plain words

Out-of-distribution detection asks a model to notice when an input is something it was never trained to handle, and to flag it instead of guessing. Most published methods were tested on small, low-resolution datasets. This paper asks what happens with ImageNet's 1,000 classes, and finds that the common confidence-based flag gets much worse as the number of classes grows.

The fix is to sort the classes into a few groups of similar things, such as animals, plants or man-made objects. Each group gets one extra "others" slot, so the model can say "not in this group". An image that fits none of the groups gets a high "others" score in every group, and that pattern is the detection signal. The method uses only the normal labelled training data and no extra outside images.

The authors report better detection than several earlier scores on four curated test sets, and a scoring step that is much faster than the strongest earlier method. The network is a pretrained ImageNet model, and only its last layer is retrained.

TL;DR

• Standard OOD scores degrade as the in-distribution label space grows. The MSP baseline's FPR95 rises from 17.34% to 76.94% when the number of ImageNet classes goes from 50 to 1,000 §2 fig. 2.
• The method groups the classes (8 ImageNet super-classes by default), trains each group with its own softmax that includes an "others" class, and scores inputs with MOS, the negated minimum "others" probability over groups §3.1 §3.2 eq. 3.
• Author-reported: average FPR95 39.97 vs 54.30 for KL matching across iNaturalist, SUN, Places and Textures, with ImageNet-1k in-distribution and a BiT-S-R101x1 backbone tab. 1.
• Relevant if you need OOD detection on a large label space with a frozen-backbone fine-tune. Numbers are author-reported, come from one setup, and have no reported variance.

Problem

Models trained on in-distribution data are tested on inputs from another distribution. The task is to choose a score S(x) and a threshold γ so that a high fraction (95%) of in-distribution data is kept, then measure how many out-of-distribution inputs pass §2.

The common baseline is the maximum softmax probability (MSP). The paper measures it with a ResNetv2-101 trained on random subsets of ImageNet with C classes (700 images per class). FPR95 degrades from 17.34% to 76.94% as C goes from 50 to 1,000 §2 fig. 2. Earlier OOD work used small datasets such as CIFAR and MNIST §1. Methods that rely on auxiliary outlier data are excluded from the comparison because the authors assume such data is hard to build for a large label space §5.

Method

Overview. The classes are split into K groups. A softmax is taken within each group, and each group has an extra "others" class that means "not in this group". The network is trained with a sum of per-group cross-entropy losses. At test time, the OOD score is taken from the "others" probabilities, not from the flat softmax over all classes §3 fig. 1.

1. Group softmax. Split the C classes into groups G_1 to G_K. For each group, compute a softmax over the group's classes §3.1 eq. 1. The paper introduces the "others" category into every group §3.1. The released code places "others" at index 0 of each group's logit slice finetune.py.

2. Group-wise labels. For each training image, in each group, the target is the image's class if the class belongs to that group. Otherwise the target is "others" §3.1. The code encodes this as 0 for "others" and c + 1 for class c in group k dataset.py.

3. Training loss. The loss is the sum over groups of the cross-entropy, averaged over training samples §3.1 eq. 2. The code computes the same sum with the group slices finetune.py. The paper notes that this gives each image a training signal for the groups it does not belong to, without any external data §3.1.

4. Inference for classification. For each group, take the largest softmax probability over its non-"others" classes. The final class is the best class of the group with the highest such value §3.1. The paper reports that in-distribution accuracy stays close to the flat-softmax counterpart §4.3.1.

5. MOS score. The score is the negated minimum over groups of the "others" probability §3.2 eq. 3. Higher scores mean in-distribution §3.2. The released test code computes the same value as the maximum over groups of the negated "others" probability test_mos.py.

6. Grouping strategies. The default is taxonomy: the 8 ImageNet super-classes (animal, artifact, geological formation, fungus, misc, natural object, person, plant) §3.3. The paper also tests feature clustering (k-means over class-mean features from the pretrained extractor) and random grouping §3.3.

7. Fine-tuning scope. The pretrained backbone is frozen and only the last fully connected layer is trained in the main results §4.2. Unfreezing more residual blocks is tested in an ablation §4.3.5 fig. 8.

# Group-softmax loss over K groups  [§3.1] [eq. 1] [eq. 2]
# Input:  logits (B, sum over k of (num_classes_k + 1)) from the FC head
#         labels (B, K): per group, 0 = "others", c + 1 = class c of group k
#         slices: per group k, the [start, end) range of its logits
# Output: scalar loss

def group_softmax_loss(logits, labels, slices):
    # Start from zero and add one cross-entropy term per group  [eq. 2]
    loss = 0
    for k in range(K):
        # Take only this group's logits; index 0 of the slice is "others"
        # [§3.1] [code: finetune.py#L128-L135]
        group_logits = logits[:, slices[k][0]:slices[k][1]]
        # Target for every image in group k: its class, or "others"
        # [§3.1] [code: dataset.py#L90-L94]
        loss = loss + cross_entropy(group_logits, labels[:, k])
    # Cross-entropy averages over the batch  [eq. 2]
    # [code: finetune.py#L179-L186]
    return loss

# MOS out-of-distribution score  [§3.2] [eq. 3]
# Input:  logits (B, sum over k of (num_classes_k + 1)) for a test image
# Output: score (B,); higher means in-distribution

def mos_score(logits, slices):
    # For each group, softmax over that group's logits only  [§3.2]
    # [code: test_mos.py#L19-L21]
    others_scores = []
    for k in range(K):
        group_logits = logits[:, slices[k][0]:slices[k][1]]
        group_probs = softmax(group_logits)
        # The "others" probability is entry 0 of the group  [§3.2]
        # [code: test_mos.py#L22]
        others_scores.append(-group_probs[:, 0])
    # Max over groups of the negated "others" probability, which equals
    # minus the minimum "others" probability  [eq. 3]
    # [code: test_mos.py#L24-L28]
    return max_over_groups(others_scores)

# Learning rate at a training step, ImageNet-1k fine-tuning  [§4.2]
# Input:  step index, base learning rate 0.003
# Output: learning rate, or None when training stops

def learning_rate(step, base_lr):
    # Linear warm-up over the first 500 steps  [§4.2]
    # [code: utils/finetune_utils.py#L65-L66]
    if step < 500:
        return base_lr * step / 500
    # Stop after 20,000 steps  [§4.2] [code: utils/finetune_utils.py#L68-L69]
    if step >= 20000:
        return None
    # Divide by 10 after steps 6,000, 12,000 and 18,000 (30, 60, 90%)
    # [§4.2] [code: utils/finetune_utils.py#L72-L75]
    return base_lr / 10 ** count_milestones_passed(step)

Hyperparameters

Main setting, ImageNet-1k in-distribution, BiT-S-R101x1 backbone §4.2.

Results

Author-reported. In-distribution ImageNet-1k, BiT-S-R101x1 backbone, four OOD test sets, all values in percent tab. 1. Test time is the paper's wall-clock time on the same 60k-image test set (see Pitfalls).

Headline claims, from the text and table:

• MOS beats the best flat-softmax baseline (Energy) by 31.06 FPR95 points on average §4.3.1 tab. 1.
• MOS beats KL matching by 14.33 FPR95 points on average §4.3.1 tab. 1. The abstract's "14.33%" is this difference in points, not a relative reduction.
• Test time is 3.2 min for MOS and 20.6 min for KL matching. The paper rounds the ratio to "6x" §4.3.1 tab. 1. Our arithmetic from the table: 20.6 / 3.2 = 6.4.
• In-distribution classification accuracy is 75.16% with group softmax and 75.20% with the flat softmax §4.3.1.
• Average AUPR: MSP 95.37, ODIN 96.37, Mahalanobis 87.67, Energy 96.42, KL matching 95.90, MOS 97.95 tab. 3.

Class-count ablation §4.3.2 fig. 5 app. B.1:

• With a fixed number of images per class, MSP's iNaturalist FPR95 goes from 21.02% (50 classes) to 63.36% (1,000 classes). The paper says MOS degrades by only 4.76 on the same set §4.3.2.
• With a fixed total of 35,000 training images, MSP's Places FPR95 goes from 14.45% to 88.02%, while MOS degrades by 38.73 app. B.1.

Grouping strategies at K = 8 §4.3.3 fig. 6:

• Feature clustering outperforms MSP by 18.2 FPR95 points on average, with about 0.16% lower in-distribution accuracy than taxonomy §4.3.3.
• Random grouping drops in-distribution accuracy by 0.98% compared with taxonomy §4.3.3.

Average over the four OOD sets for each grouping and number of groups tab. 2 (per-dataset values are in the source table):

Source: tab. 2 app. B.2. Level 0 is the main setting; negative levels merge super-classes and positive levels split them.

Feature extractors §4.3.4 fig. 7: MOS beats MSP at every extractor in a family of five BiT-S models, with the head only retrained. Better extractors help both classification and OOD detection, according to the paper.

Fine-tuning depth §4.3.5 fig. 8: MOS beats MSP at every fine-tuning depth in BiT-S-R101x1. Training more blocks raises accuracy but can hurt OOD detection in some cases, which the authors attribute to label overfitting.

Paper vs code

Repository: deeplearning-wisc/large_scale_ood, Apache-2.0, linked from the paper's footnote on the contributions list §1. The owner is the authors' lab (the deeplearning-wisc account, the group of co-author Yixuan Li). The commit is the one cloned at compile time. The README says the code is modified from BiT, ODIN, Outlier Exposure, deep Mahalanobis and Robust OOD Detection [README.md#L5-L9].

• Mixup on ImageNet training. The paper does not mention mixup. The code applies mixup with Beta(0.1, 0.1) whenever the training set has at least 20,000 images, which is the case for ImageNet, and mixes the group-softmax losses of the two targets finetune_utils.py finetune.py finetune.py.
• Horizontal flip and normalisation. The paper describes a crop to 480 after resizing to 512. The code also applies a random horizontal flip and normalises each channel with mean 0.5 and std 0.5 for training and test finetune.py finetune.py test_utils.py. The paper states neither.
• Head initialisation. The code builds the head with zero initialisation finetune.py. The paper does not state this.
• Weight decay. The paper does not state a weight decay. The code uses none for the group-softmax run finetune.py.
• Training schedule. The steps (20,000), batch (512), base learning rate (0.003), warm-up (500), and decays at 6k, 12k and 18k match the paper's text finetune_utils.py finetune_utils.py finetune_utils.py. The code decays only when the step is strictly greater than a milestone, so the decay starts one step after each milestone.
• MOS score. Eq. 3 and the released test code agree: the code takes the maximum over groups of the negated "others" probability, which equals minus the minimum "others" probability test_mos.py.
• Feature clustering and random grouping. The paper reports these in §4.3.3 and in the group-number table tab. 2. The repository has no clustering or random-group code, and the only group configuration file is group_config/taxonomy_level0.npy . The level -3 to 6 results cannot be regenerated from the release.
• In-distribution test set. The paper does not name the in-distribution test set. The released test script uses the ImageNet validation folder (50,000 images) test_mos.sh.
• OOD sets. The sampled OOD images are not in the repository. The README links them and the authors' sampling is described only in the paper README.md §4.1.
• Fine-tuned checkpoints. The README links group-softmax and flat-softmax checkpoints on the authors' web server README.md. They were not downloaded for this card.

Pitfalls

• Units in the headline numbers. "14.33%" and "31.06%" are differences in FPR95 percentage points between averages, not relative reductions. Relative to KL matching's 54.30, the 14.33-point gap is about 26.4% (our arithmetic).
• "60k images" in tab. 1. The table caption says the test time was measured on 60k images. The stated OOD sets total 35,640 (10,000 + 10,000 + 10,000 + 5,640) §4.1. Adding the 50,000-image ImageNet validation set would give 85,640. The in-distribution set for timing is not stated, so the test-time setup cannot be reconstructed from the paper.
• Single-run numbers. No seeds, repeats or variance are reported for any row of tab. 1 or tab. 2. Differences of about a point between methods should not be treated as settled.
• Taxonomy vs feature clustering at small K. The text says taxonomy beats feature clustering when the number of groups is small or mildly large app. B.2. At K = 2 (level -3), feature clustering has the higher average AUROC (87.92 vs 86.05) and taxonomy the lower average FPR95 (52.81 vs 53.11) tab. 2.
• Group-softmax implementation detail. Each group's logits include "others" at index 0. Dropping it, or using the flat softmax over all C + K logits, changes the MOS score. The MOS score uses a softmax within each group, not across all logits test_mos.py.
• Mixup changes the loss. Following eq. 2 alone does not reproduce the released training, because the code also mixes two targets per batch finetune.py.
• NumPy version. dataset.py uses np.int, which NumPy removed in version 1.24. On newer NumPy this line raises an error when the group-softmax dataset is built dataset.py.
• Frozen backbone matters. The paper's claims are for the last layer only. Unfreezing more residual blocks changes OOD results in some cases §4.3.5 fig. 8.
• Expensive comparisons. Mahalanobis needs 145.4 min of test time in tab. 1, and KL matching needs 20.6 min. Most speed claims are against KL matching only tab. 1.
• Figure counts. The figure and table numbers above follow the order of the LaTeX source. Where a float could move in the compiled PDF, check the PDF.

Lineage

• Builds on: the maximum softmax probability baseline of Hendrycks and Gimpel §2; BiT-S pretrained models of Kolesnikov et al. as the backbone and fine-tuning recipe §4.2; the WordNet taxonomy of ImageNet for the default groups §3.3.
• Compared against: MSP, ODIN, Mahalanobis, Energy and KL matching (Hendrycks et al. 2019, benchmark on anomaly segmentation) §4.3.1 tab. 1.
• Related: group softmax for long-tail object detection (Li et al.), which the authors contrast in objective §1 §5. Large-scale OOD benchmarking by Roady et al. §5.
• Superseded by: not stated.