Learning and Evaluating Representations for Deep One-class Classification

In plain words

Imagine you only have photos of one kind of object and need to flag any new photo that does not belong with them. This paper splits that job in two. First it trains a neural network to describe images, using tasks that need no labels, such as guessing how an image was rotated or telling two differently cropped copies of one photo apart. Then it hands those descriptions to a classic, simple detector that asks how close a new image is to the training images. Learning the description and drawing the boundary are separate steps.

The authors find that the usual contrastive way of learning descriptions works poorly for this job, because it spreads every training image out evenly and leaves no room for an outlier to stand out. Their fix is to also treat rotated copies of each training image as separate kinds of data, which keeps the normal images in a tighter group. On standard image benchmarks this gives the best mean scores among the methods the paper compares, and it does so without extra work at test time.

TL;DR

• Two stages: a self-supervised backbone trained on inlier images only, then a KDE or RBF OC-SVM fitted on its features §3, §3.2.
• The paper adds distribution-augmented contrastive learning (DA): rotated copies of the training set count as separate distributions §3.1.3.
• Author-reported: mean AUC 89.9 with OC-SVM and 89.8 with KDE over five benchmark settings, against 83.1 for the end-to-end rotation classifier of prior work §4.1, tab. 2, tab. 3.
• Settings matter: batch size 32, temperature 0.2, ResNet-18, 2048 epochs. The released code differs from the paper in places; see Paper vs code.

Problem

One-class classification trains only on inlier data and must flag outliers at test time §1. Density models such as KDE scale poorly to high-dimensional images §1. Deep one-class classifiers trained end to end tend to collapse to a trivial solution (hypersphere collapse) §1. Surrogate classifiers trained on simulated outliers, such as rotation prediction, work but are not built for the one-class task, so their classifier can overfit the proxy task §1, §3.1.2. Contrastive representations are nearly uniform on the hypersphere, which makes outliers hard to isolate §1, §3.1.3.

Method

Stage 1: representation learning §3.1. The backbone f is a ResNet-18 whose pooled output is a 512-dimensional feature (the width is set by the released model, not stated in the paper) resnet_util.py. Optionally, a projection head g (an MLP of linear, batch-norm and ReLU blocks) sits on top of f and is used only to compute the proxy loss §4. At test time the representation is the L2-normalised f §3.1. Three proxy objectives are compared:

1. Rotation prediction: classify which of four rotations (0, 90, 180, 270 degrees) was applied to an augmented image §3.1.2 eq. 1.
2. Contrastive learning: two independent augmented views of each image should match, and the other images in the batch serve as negatives §3.1.3 eq. 2.
3. Distribution-augmented contrastive learning: 90, 180 and 270 degree rotations of each training image are treated as separate instances and separate distributions. This enlarges the training set four times and eases the uniformity problem §3.1.3. Adding horizontal flips as a further distribution also helps §5.2.

Stage 2: one-class classifier §3.2. The learned features of the inlier training set are L2-normalised. A kernel density estimator (KDE) or an RBF one-class SVM is fitted on them, and the test score is the KDE log-sum-exp score app. A.1 eq. 6. The gradient of the KDE score with respect to the input gives a visual explanation of the decision §3.2.1.

Batch size and uniformity §3.1.3, §5.1. The paper argues that large contrastive batches push features toward uniformity, which hurts one-class detection. It measures uniformity with an MMD distance to samples from a uniform distribution on the sphere §5.1. It finds batch size 32 best on the CIFAR-10 validation set among the sizes it tried §5.1. A correlation between MMD and AUC is reported in the appendix app. A.2.2.

Key equations as printed in the source (numbering counted from the LaTeX):

\mathcal{L}_{\mathrm{rot}} = \mathbb{E}\Big[\mathrm{CE}\big(y, p_{q\circ g}(y \mid \mathrm{rot90}(\mathcal{A}(x), y))\big)\Big] eq. 1

\mathcal{L}_{\mathrm{clr}} = -\mathbb{E}\Bigg[\log\frac{\exp(\phi(\mathcal{A}(x))^{\top}\phi(\mathcal{A}'(x))/\tau)}{\exp(\phi(\mathcal{A}(x))^{\top}\phi(\mathcal{A}'(x))/\tau) + \sum_{i=1}^{M-1}\exp(\phi(\mathcal{A}(x))^{\top}\phi(\mathcal{A}(x_i))/\tau)}\Bigg] eq. 2

Pseudocode for the contrastive step with distribution augmentation. Cited points follow the paper, and the code reference shows how the released code implements the same step. The denominator difference is discussed under Paper vs code.

# Distribution augmentation: stack rotated copies of the inlier set so each
# rotation is its own distribution, four times the data  [§3.1.3]
def build_train_set(images):
    rotated = [rot90(images, k) for k in range(4)]  # k = 0, 1, 2, 3
    return concatenate(rotated)  # (4N, H, W, C)

# One contrastive step on a batch of B images  [eq. 2]
def contrastive_step(images):
    # Two independent views; both use the same augmentation pipeline
    # [§3.1.3] [code: data/augment.py#L216-L222]
    view_a = random_augment(images)  # crop, flip, jitter, gray, blur
    view_b = random_augment(images)
    # Embed, apply the projection head, L2-normalise  [§3.1.1]
    feats = projection_head(backbone(concatenate(view_a, view_b)))
    feats = l2_normalize(feats)  # (2B, 128)
    feats_a, feats_b = split(feats, 2)
    # Similarity of each view-A sample with each view-B sample  [eq. 2]
    logits = matmul(feats_a, transpose(feats_b)) / temperature  # (B, B)
    # Positive pair is the diagonal; the released code uses only these
    # cross-view terms in the softmax  [code: contrastive.py#L96-L98]
    loss = mean(cross_entropy(logits, labels=arange(B)))
    # L2 penalty on the weights  [code: contrastive.py#L101]
    return loss + weight_decay * l2_loss(weights)

Rotation prediction, for comparison eq. 1:

# Rotation prediction: classify the rotation applied to each augmented image
# [§3.1.2] [eq. 1] [code: unsup_embed.py#L69-L90]
def rotation_step(images):
    # Augment, then rotate by 0, 90, 180, 270 degrees; label = rotation index
    inputs = concatenate([rot90(augment(images), k) for k in range(4)])
    labels = one_hot(concatenate([full(B, k) for k in range(4)]), 4)
    # Projection head g, then linear classifier q  [§3.1.2]
    logits = classifier(projection_head(backbone(inputs)))
    # Cross-entropy against the rotation index  [eq. 1]
    loss = mean(cross_entropy(labels, logits))
    return loss + weight_decay * l2_loss(weights)

Detector scores, fitted on inlier training features only:

# Fit the kernel width from the training features  [§3.2] [app. A.4]
# [code: util/train.py#L557] [code: util/train.py#L564]
def fit_kde(train_feats):
    feats = l2_normalize(train_feats)  # (N, d) unit vectors
    # The paper writes 10 / (|f| Var(f)); variance over all entries
    gamma = 10 / (variance(feats) * feats.shape[1])
    return feats, gamma

# KDE score, higher = more anomalous (sign flipped from the paper's normality
# score)  [app. A.1] [eq. 6] [code: util/train.py#L566-L573]
def kde_anomaly_score(x, feats, gamma):
    sims = matmul(l2_normalize(x), transpose(feats))  # cosine similarities
    return -logsumexp(2 * gamma * sims, axis=1) / gamma

Hyperparameters

Values from the paper's appendix, cross-checked with the released scripts. Dataset resolutions and training settings are the same for every method in the paper's main comparison app. A.4.

Results

Author-reported. Setup: one-class protocol; the inlier class is trained on and the other classes are outliers §4. CIFAR-100 uses 20 super-classes §4. The mean is over the five datasets §4.1. Backbone ResNet-18 (except the ImageNet baseline) §4. Hardware not stated. Baselines are listed in the rows.

Table 2 tab. 2, AUC mean and standard deviation over five runs, no test-time augmentation for the paper's methods:

Comparison to earlier one-class methods tab. 3. Rows marked † use test-time augmentation. The two "ours" rows repeat the KDE and OC-SVM values above. The authors caution that implementations differ §4.1.

MVTec anomaly detection tab. 4 app. B.3. Image-level detection and pixel-level localization AUCs, five seeds. This is a defect dataset, not the semantic protocol above.

Other reported results:

• Distribution augmentation with horizontal flip and rotations reaches a mean AUC of 93.7 on CIFAR-10 without test-time augmentation §5.2 fig. 7.
• Ensembling: averaging detectors over 5 augmented-distribution sets, each trained with 5 seeds (25 models), reaches 94.6 AUC on CIFAR-10 app. A.2.2 tab. 6.
• Data efficiency (CIFAR-10): with 100 training images, the DA representation matches the ImageNet ResNet-50 baseline; rotation prediction needs about 250 and vanilla contrastive about 1000 app. A.2.3.
• Unsupervised anomaly detection (outliers in training data, 0.5% to 10%) shows rotation prediction is more robust at high outlier ratios, while contrastive learning gains when classifiers are fitted on clean inliers §5.3.

Paper vs code

The official repository is linked in the abstract, so no further search was needed. It is Apache-2.0 LICENSE, TensorFlow 2, commit above. Paths are relative to the repository root at that commit. Discrepancies, each with both sources:

1. Contrastive denominator. The paper's eq. 2 sums both cross-view terms and same-view negatives A(x) vs A(x_i) eq. 2. The code's softmax covers only the other view's samples in the batch contrastive.py. The two views are built with the same pipeline, as the paper says §3.1.3 augment.py. Treat the released code as a cross-view-only version of eq. 2. The paper does not explain this choice.
2. Rotation configuration. The appendix says all methods use batch size 32 and the projection head [512×8, 128] app. A.4. The rotation script uses batch size 64 and a head with no 128-dimensional layer [code: script/run_rotation.sh#L48-L50]. The script's comment claims it reproduces the 91.3 entry of Table 2 run_rotation.sh. Which config produced that number is not stated.
3. "KDE" versus "1-NN". The rotation script calls the 91.3 result "KDE (1-NN)" run_rotation.sh. The paper calls it KDE with the γ formula §4.1 app. A.4. The code's "auc" key is a 1-nearest-neighbour score train.py, which is the γ-to-infinity limit of the KDE. Whether Table 2 used finite γ or 1-NN is not clear from the source.
4. Which feature is the representation. The paper uses f, the normalised pooled backbone output §3.1. The code reports two sets of metrics, pool (f) and embed (g∘f) resnet_util.py [code: util/train.py#L522-L545]. The scripts do not say which key produced Table 2. The paper's ablation says f beats g∘f §5.1.
5. Kernel width. §3.2 says the width is reduced ten times relative to the default §3.2. Appendix A.4 says γ is ten times larger than the default app. A.4. The code uses 10 / (d · Var) train.py train.py, which is larger. The §3.2 wording contradicts both the appendix and the code.
6. Variance in γ. The paper writes Var(f) without saying what it is over. The code uses the variance of all entries of the normalised training features train.py train.py.
7. Weight decay on batch norm. The flag regularize_bn does the opposite of its name: when it is True, variables with "bn" in their name are excluded from the L2 term train.py train.py. The default is False train_and_eval_loop.py; the contrastive and rotation scripts leave it unset, so their runs decay the batch-norm parameters too. The L2 term is λ times the sum of squared weights divided by two (TensorFlow's l2_loss) train.py.
8. OC-SVM ν. The paper uses "the default" app. A.4. The code does not set ν train.py, so the scikit-learn default applies. Pin ν in any reimplementation.
9. Mean column. With the weighting described in §4.1, the Mean column does not reproduce exactly from the rows. For example, the rotation-prediction KDE row gives about 87.8 with CIFAR-100 counted as 20 classes, against 88.2 printed. The card reports the printed values.
10. MVTec. The repository has no MVTec loader or patch pipeline (no mention of MVTec in its Python files), so Table 4 cannot be reproduced from the release as far as the source shows app. B.2.
11. Matching points. Temperature 0.2, learning rate 0.01, momentum 0.9, weight decay 0.0003, 2048 epochs and the 32×32 input size agree between the appendix and the contrastive script app. A.4 run_contrastive_da.sh.

No discrepancies found beyond the list above.

Pitfalls

• Model selection uses outliers. Hyperparameters were chosen on CIFAR-10 using 500 inliers and 500 outliers app. A.4. Results on the other datasets reuse those settings. The paper acknowledges this may violate the one-class assumption app. A.4.
• Distribution augmentation costs. The rotation copies quadruple the offline training set cifar.py, so one epoch processes four times as many images. The contrastive batch of 32 is 32 pairs, which is 64 forward passes per step.
• Scoring cost. KDE and OC-SVM scoring compare each test sample with the whole training set. The code batches the KDE sum in blocks of 100 test samples train.py. The paper notes that a single Gaussian is cheaper at test time and competitive app. A.2.1 tab. 5.
• Normalisation before the kernel. Features are L2-normalised before the RBF kernel, the KDE and the γ calculation train.py. Skip this step and γ changes meaning.
• Rotation baseline and test time. For the rotation models the code builds test-time rotations for its logit score. Its embedding path keeps only the unrotated view train.py train.py. KDE numbers therefore use no test-time augmentation.
• Sensitivity. The main text and appendix report one γ rule and no sweep over it. The effect of changing the 10× factor is not reported as far as the source shows.
• Uniformity is measured on g∘f. The batch-size and MMD analysis measures uniformity on the projection output, not on f app. A.2.2. Keep that in mind when interpreting the batch-size curve.
• Single-class scope. The main results train on one class at a time. The unsupervised and semi-supervised results are a different setting and use other training sets app. A.3.
• Augmentation order. The rotation loss applies the augmentation before the rotation, rot90(A(x), y) eq. 1, and the released pipeline does the same augment.py.
• Numbering. Equation, figure and table numbers here were counted from the LaTeX source, and some source blocks are commented out. Check them against the published PDF before quoting a number.

Lineage

• Builds on: rotation prediction [gidaris2018unsupervised]; end-to-end geometric-transform one-class classifiers [golan2018deep] and [hendrycks2019using], which this paper replaces with a second-stage detector §3.2; SimCLR-style contrastive learning [chen2020simple], with the one-class fixes described above; uniformity and class-collision analyses [wang2020understanding] [saunshi2019theoretical].
• Concurrent: distribution-augmented contrastive learning in CSI, arXiv:2007.08176 §2 §5.2.
• Compared against: Deep SVDD [ruff2018deep], [bergman2020classification], [huang2019inverse], and CAVGA for MVTec app. B.3.
• Superseded by: not stated.