SSD: A Unified Framework for Self-Supervised Outlier Detection

In plain words

Spotting inputs that a model was not built for usually needs labelled examples of normal data, and labels are expensive. This paper asks how little is needed, and its answer is that unlabeled normal data can be enough. The method first teaches a network to recognise images without labels: two random crops of the same picture should give similar features, and crops of different pictures should not. Then it describes where the features of normal data sit, as a blob with a size and a shape, and scores a new input by how far it lies from that blob. The distance it uses, Mahalanobis distance, takes the blob's shape into account. Far away means probably unusual. A few labelled outliers, or labels for the normal data, can be added to sharpen the score. The paper reports that this matches or beats several detectors that use labels, though not on every pair of datasets.

TL;DR

SSD is a label-free outlier detector: contrastive self-supervised training on unlabeled in-distribution images, followed by a single-Gaussian Mahalanobis score on normalised features. Its author-reported average AUROC over four CIFAR-10/100 pairs is 88.7 tab. 1, against 80.9 for Rotation-loss, the best unlabeled baseline with a complete row in that table. Scoring with five OOD samples per class reaches 92.5, and adding labels reaches 92.4 tab. 4. Use it when no labels exist for the normal data; the training recipe has several details the paper leaves out (see Pitfalls and Paper vs code).

Problem

Unsupervised detectors that model only unlabeled in-distribution data often perform close to random, while the strongest detectors need fine-grained labels for the training data §1. The paper also considers two extensions: a few OOD samples available at test time (one to five per class) §1 §3.2, and labels for the in-distribution data when they exist §3.3. The core method assumes no OOD data for training or tuning, and avoids extra tuning parameters, using one standard training setup and a single cluster §5.

Method

SSD has two stages §3. Stage one trains a feature extractor on unlabeled in-distribution images with the NT-Xent contrastive loss §3, eq. 1. Stage two needs no further training: it estimates the mean and covariance of the normalised training features and scores each test input by its Mahalanobis distance to them §3.1, eq. 2.

Steps for the label-free detector:

1. Train backbone f and projection head h with eq. 1 on unlabeled in-distribution images, for 500 epochs with batch size 512 §4.1.
2. Extract f(x), the feature before the projection head, for the training images, and L2-normalise each one alg. 1.
3. Estimate the mean and sample covariance of the normalised training features, using a single cluster §3.1 §4.1.
4. For each test input, compute its normalised feature and the score in eq. 2 §3.1.
5. Report AUROC, AUPR and FPR at 95% TPR §4.1 app. A.2.

Figure 1 plots the AUROC along each principal eigenvector of the features. High-eigenvalue directions dominate Euclidean distance but help least with detection, which motivates scaling each direction by its eigenvalue, i.e. Mahalanobis distance fig. 1 §3.1.

Extensions. SSD_k (few-shot): with k OOD samples per class, the score subtracts a Mahalanobis term for the OOD statistics eq. 3. Those statistics come from augmented copies of the OOD samples, with Ledoit-Wolf shrinkage for the covariance eq. 4 §3.2. SSD_sup (labels): NT-Xent is replaced by a supervised contrastive loss in which same-label images in the batch are positives eq. 5 §3.3. alg. 1 summarises the full pipeline. It holds out a calibration split and sets a threshold at a required true-positive rate T; the experiments report threshold-free metrics instead §4.1.

Eq. 1, NT-Xent over a batch of N images with two views each, where j is the other view of image i and tau is the temperature §3:

\mathcal{L}_{batch} = \frac{1}{2N}\sum_{i=1}^{2N} -\log\frac{e^{u_i^\top u_j/\tau}}{\sum_{k=1}^{2N}\mathbb{1}(k\neq i)\,e^{u_i^\top u_k/\tau}}, \qquad u_i = \frac{h(f(x_i))}{\|h(f(x_i))\|_2} eq. 1

Eq. 2, the single-cluster Mahalanobis score; the paper's general form takes the minimum over m clusters §3.1:

s_x = \min_m\,(z_x-\mu_m)^\top \Sigma_m^{-1} (z_x-\mu_m) eq. 2

Eq. 3, the few-shot score, with in-distribution statistics and OOD statistics §3.2:

s_x = (z_x-\mu_{in})^\top \Sigma_{in}^{-1} (z_x-\mu_{in}) - (z_x-\mu_{ood})^\top \Sigma_{ood}^{-1} (z_x-\mu_{ood}) eq. 3

Eq. 4, the same score with the OOD statistics estimated from the augmented set U with shrunk covariance S_U §3.2:

s_x = (z_x-\mu_{in})^\top \Sigma_{in}^{-1} (z_x-\mu_{in}) - (z_x-\mu_{U})^\top S_{U}^{-1} (z_x-\mu_{U}) eq. 4

Eq. 5, the label-aware loss, where N_{y_i} is the number of images with label y_i in the batch §3.3:

\mathcal{L}_{batch} = \frac{1}{2N}\sum_{i=1}^{2N} -\log\frac{\frac{1}{2N_{y_i}-1}\sum_{k=1}^{2N}\mathbb{1}(k\neq i)\,\mathbb{1}(y_k=y_i)\,e^{u_i^\top u_k/\tau}}{\sum_{k=1}^{2N}\mathbb{1}(k\neq i)\,e^{u_i^\top u_k/\tau}} eq. 5

NT-Xent with the positive-pair rule written out, for the label-free stage. Augmentation and the head width are not stated in the paper; see Hyperparameters and Paper vs code:

# NT-Xent loss over 2N augmented views  [§3] [eq. 1]
# Input:  images (N, 3, H, W), unlabeled; tau = 0.5  [§4.1]
# Output: scalar loss
def ntxent_loss(images):
    # Two random augmented views of every image  [§3]
    views = concat(augment(images), augment(images))  # (2N, 3, H, W)
    # Backbone f, then projection head h, then unit length  [eq. 1]
    embeddings = normalise(head(backbone(views)))     # (2N, d_out)
    # Scaled cosine similarities between all views  [eq. 1]
    logits = embeddings @ embeddings.T / tau          # (2N, 2N)
    loss = 0
    for i in range(2N):
        # The positive of view i is the other view of the same image  [eq. 1]
        positive = pair_index(i)
        # Denominator sums over all views except i itself  [eq. 1]
        loss += -log(exp(logits[i, positive]) / sum_except(exp(logits[i]), i))
    return loss / (2N)

Scoring with one cluster, the core detector. The repository also normalises each dimension (see Paper vs code):

# Single-cluster Mahalanobis score  [§3.1] [eq. 2] [alg. 1]
# Input:  train_feats (N, d) from in-distribution training data
#         test_feat (d,) for one input
# Output: score, larger means more likely an outlier
def ssd_score(train_feats, test_feat):
    # Unit L2 norm for training and test features  [alg. 1]
    train_z = normalise_rows(train_feats)             # (N, d)
    test_z = test_feat / norm(test_feat)              # (d,)
    # Sample mean and sample covariance of training features  [§3.1]
    mu = mean(train_z, axis=0)                        # (d,)
    sigma = covariance(train_z)                       # (d, d)
    # Squared Mahalanobis distance to the training distribution  [eq. 2]
    diff = test_z - mu                                # (d,)
    return diff @ inverse(sigma) @ diff

Hyperparameters

Results

Author-reported. Setup: ResNet-50 unless stated, 500 epochs, batch size 512, CIFAR-10 as in-distribution unless the row says otherwise §4.1. The Softmax, ODIN and Mahalanobis baselines were re-implemented by the authors on the same supervised network app. A.1.

The label-free detector beats the reported unlabeled baselines on average. Supervised training gives a better result on some pairs, such as CIFAR-100 with Texture tab. 2. Contrastive self-supervised features beat supervised ones on 56% of the tasks in the paper's comparison §4.2. The paper states no hardware or compute cost for these runs.

Paper vs code

Code: official repository, commit cf2e162ef7ada254ef916d1064c0a19ad571ee71 (MIT, copyright Princeton INSPIRE Research Group) LICENSE. Repository files: train.py, trainers.py, losses.py, models.py, data.py, utils.py, eval_ssd.py, eval_ssdk.py.

1. Score features. The paper's Algorithm 1 normalises features by L2 norm only alg. 1. The evaluation code then standardises every dimension with the training mean and standard deviation eval_ssd.py eval_ssdk.py. The paper does not mention this step.
2. Covariance estimate. The paper says sample covariance and an inverse §3.1. The code uses np.cov(..., bias=True), which divides by N, and a pseudo-inverse utils.py utils.py utils.py.
3. Shrinkage in few-shot. The paper uses Ledoit-Wolf for the OOD covariance §4.3. The code also applies it to the in-distribution covariance eval_ssdk.py utils.py.
4. Few-shot protocol. The paper averages 25 random trials §4.3 and uses a single transformation per sample when k is small app. A.1. The code runs one trial per call and always uses 10 copies by default eval_ssdk.py. It does not seed NumPy eval_ssdk.py, while the sampler uses np.random utils.py.
5. Supervised contrastive loss. Eq. 5 takes the log of the mean similarity over positives eq. 5. The code averages the log-probabilities over positives losses.py, which is a different loss.
6. Loss scale. The code multiplies the loss by temperature divided by base temperature, which is 0.5 / 0.07 here losses.py losses.py train.py. Eq. 1 has no such factor §3. The effective step size therefore depends on the code's scaling, not only on the learning rate of 0.5 §4.1.
7. Optimiser and schedule. Momentum 0.9 and a 1e-4 cosine floor are in the code train.py train.py; the paper gives neither.
8. Warmup. The paper does not mention warmup §4.1. The README's training command enables it [README L41-L44], and it adds 10 epochs before the 500 train.py.
9. Augmentation of the supervised baseline. The SSD path uses the stronger augmentation data.py. The supervised baseline uses crop and flip with cross-entropy data.py train.py train.py. The paper states an identical training budget §4.1. The augmentations differ, which that statement does not cover.
10. Released checkpoints. The README says its models "aren't exactly identical" to the paper [README L52]. Its example checkpoint path names batch size 900 [README L53], while the paper uses 512 §4.1.
11. Anomaly detection. The paper adds an L2 feature regulariser with coefficient 0.01 and uses tau = 0.1 §4.2. The repository has no such term losses.py and no anomaly mode train.py.
12. Experiments without code. ImageNet, ImageNet-O and the STL-10 extra-unlabeled-data experiment have no data loader in the repository data.py app. B.4 tab. 6.
13. Minor code issue. The STL-10 loader reads from the SVHN folder data.py.
14. Metric definitions. FPR counts OOD scores below the 95th percentile of in-distribution scores utils.py. AUPR treats OOD as the positive class utils.py. The paper does not say which class is positive for AUPR app. A.2.
15. Model selection. train.py keeps the best checkpoint by kNN accuracy on the CIFAR-10 test set train.py utils.py. The paper does not describe checkpoint selection.

No discrepancies were found for the backbone choice (ResNet-50), NT-Xent temperature 0.5 train.py, batch size 512 train.py, epochs 500 train.py, learning rate 0.5 train.py or weight decay 1e-4 train.py.

Searches: the paper links the repository in its abstract, so no other search was needed.

Pitfalls

• Temperature matters a lot. At tau = 0.001 the detector drops to 66.7 AUROC, against 89.5 at tau = 0.5, on the ResNet-18 ablation tab. 5. Treat tau as a sensitive value, not a free one.
• Warmup and batch size. Batch 512 gives 1024 views per forward pass, and BatchNorm statistics are computed over both views together trainers.py. The code's own warning recommends warmup for batches above 256 train.py.
• Checkpoint selection uses test data. The kNN accuracy used to pick the best checkpoint is measured on the CIFAR-10 test set train.py. Results from that checkpoint are then not clean held-out numbers.
• Layer and cluster choice used a reported OOD set. The single-cluster choice rests on an analysis with CIFAR-10 as in-distribution and CIFAR-100 as OOD app. B.2, and CIFAR-100 is also one of the reported OOD sets tab. 1.
• Zero-variance dimensions. Each feature dimension is divided by its standard deviation plus 1e-10 eval_ssd.py. A dimension that is constant in training can then produce very large test values.
• Pseudo-inverse of a 2048 by 2048 matrix. The pseudo-inverse hides near-singularity rather than failing. The code recomputes it for each score call, and twice per call for in-distribution and OOD sets utils.py utils.py.
• Few-shot cost. Each OOD sample generates 10 augmented copies, so the forward passes grow with k times the number of classes times 10 utils.py utils.py.
• Algorithm 1 is not implemented. The threshold procedure with a calibration split alg. 1 has no code path. The evaluation scripts report threshold-free metrics only eval_ssd.py.
• Paper tables disagree with each other.
  • ImageNet SSD and supervised AUROC are swapped between Table 2 (SSD 99.1, supervised 99.4) and Table 6 (SSD 99.4, supervised 99.1) tab. 2 tab. 6.
  • CSI on CIFAR-10 in, CIFAR-100 OOD is 89.2 in Table 1 but 92.2 in Table 4 tab. 1 tab. 4.
  • The few-shot gains of 1.6 and 2.1 §4.3 do not match Table 1's four-pair averages, which rise from 88.7 to 92.5 (+3.8) tab. 1. The averaging set is not defined.
• Compute is not reported in the results or appendix. Stage one is the expensive part: ResNet-50 for 500 epochs at batch size 512.
• Architecture. The paper describes ResNet-50 without its stem. The code uses a CIFAR-style stem, and an ImageNet-style stem would differ from the code models.py. The paper does not say which one the reported numbers used.

Lineage

• Builds on: SimCLR's NT-Xent loss §3, eq. 1; the code's docstring names the SimCLR paper as arXiv:2002.05709 losses.py. The label-aware loss is supervised contrastive learning §3.3, which the code's docstring names as arXiv:2004.11362 losses.py.
• Compared against: PixelCNN++, autoencoder, VAE, Deep-SVDD and Rotation-loss tab. 1; CSI tab. 1 tab. 3; Softmax probabilities, ODIN, the Mahalanobis detector of Lee et al. (2018), Residual Flows, Gram matrix, Outlier Exposure, and Contrastive + supervised tab. 4.
• Superseded by: not stated in content already in this wiki.