ViM: Out-Of-Distribution with Virtual-logit Matching

In plain words

An image classifier is trained on a fixed set of categories. Shown something from outside those categories, it often still answers with confidence. Out-of- distribution (OOD) detection asks whether an input belongs to the world the model was trained on, so the system can refuse or flag the rest.

Most detectors read one signal from the model: the internal feature vector, the raw class scores (logits), or the normalised probabilities. The authors argue that these signals miss different kinds of outliers. Their method, ViM, uses both. It measures how far the feature sticks out from the main directions seen in training, turns that into an extra fake class score, and uses that score's probability as the detector. The method needs no extra training and no outlier data.

TL;DR

• Post-hoc OOD score for a trained classifier: no retraining and no OOD data §1.
• It combines a feature-space residual (outside a principal subspace) with the usual class logits, and returns the softmax probability of the residual term §4.
• Author-reported: average AUROC 90.91% over four OOD sets with BiT-S-R101x1, against 86.62% for the best baseline tab. 2.
• Best average AUROC and FPR95 on both BiT and ViT in the paper's table; on iNaturalist with BiT it ranks third, which the authors link to small residual norms on that set §6.1 tab. 2.
• Code is public (Apache-2.0) and is short; the score itself is a few lines of numpy.

Problem

Logit- and probability-based scores, such as maximum softmax probability, are limited by the single signal they read. Feature-space scores capture variation that the classifier's weights throw away, but they ignore class-specific evidence §1 §3. The authors show that the same OOD set can be easy for one family and hard for the other fig. 2. Large-scale OOD benchmarks also suffer from small or tag-selected coverage, which the authors say makes results easy to game §5.

Assumptions: a single trained network with a linear classification head, and a set of ID training features that can be stored §4. The method requires an offline pass over training data to fit the subspace and the constant alpha.

Method

The score is computed in three steps for each input feature §4:

1. Move the origin of feature space so that the logits are bias-free §3 eq. 1.
2. Project the feature onto the residual space, outside the top-D principal directions of the training features, and take its norm §4 eq. 4, eq. 5.
3. Multiply that norm by a constant alpha and treat it as one extra "virtual" logit. Compute the softmax over the original logits plus this logit. The probability assigned to the virtual logit is the ViM score §4 eq. 7.

Alpha is set so that, on average over training samples, the virtual logit has the same magnitude as the mean maximum logit §4 eq. 6. Higher ViM means more OOD.

Equations (as written in the paper):

Origin offset, with weight matrix W (N by C) and bias b:

\mathbf{l} = \mathbf{W}^T \mathbf{x}' = \mathbf{W}^T (\mathbf{x} - \mathbf{o}), \quad \mathbf{o} := -(\mathbf{W}^T)^{+}\mathbf{b} §3 eq. 1

Residual score (a baseline and step 2 of ViM):

\mathrm{Residual}(\mathbf{x}) = \|\mathbf{x}^{P^\perp}\| §3 eq. 3

Principal space from the training features X (rows are features relative to o):

\mathbf{X}^T \mathbf{X} = \mathbf{Q} \boldsymbol{\Lambda} \mathbf{Q}^{-1} §4 eq. 4

Virtual logit, where R holds the columns of Q after the first D:

l_0 := \alpha \|\mathbf{x}^{P^\perp}\| = \alpha \sqrt{\mathbf{x}^T \mathbf{R}\mathbf{R}^T \mathbf{x}} §4 eq. 5

Matching constant from K training samples:

\alpha := \frac{\sum_{i=1}^{K} \max_j \{ l^i_j \}} {\sum_{i=1}^{K} \|\mathbf{x}_i^{P^\perp}\|} §4 eq. 6

ViM score:

\mathrm{ViM}(\mathbf{x}) = \frac{e^{\alpha\sqrt{\mathbf{x}^T\mathbf{R}\mathbf{R}^T\mathbf{x}}}} {\sum_{i=1}^{C} e^{l_i} + e^{\alpha\sqrt{\mathbf{x}^T\mathbf{R}\mathbf{R}^T\mathbf{x}}}} §4 eq. 7

Monotone equivalent used in practice (the log-odds of ViM). Ranking, AUROC and FPR95 are unchanged by it §4 eq. 8:

\alpha \|\mathbf{x}^{P^\perp}\| - \ln \sum_{i=1}^{C} e^{l_i} §4 eq. 8

Pseudocode, in our own words. Fitting is offline; scoring runs per input.

# Fit the residual basis and alpha (offline)  [§4] [eq. 4] [eq. 6]
# Input:  train_feats (K, N), train_logits (K, C), origin (N,), dim D
# Output: residual_basis (N, N - D), alpha (scalar)

def fit_vim(train_feats, train_logits, origin, dim):
    # Move the origin so the logits are bias-free  [§3] [eq. 1]
    centred = train_feats - origin                     # (K, N)
    # Eigen-decompose X^T X; eigh returns ascending eigenvalues  [§4] [eq. 4]
    gram = centred.T @ centred                         # (N, N)
    eig_values, eig_vectors = eigh(gram)
    # The N - D smallest-eigenvalue directions span the residual space  [§4]
    feat_dim = train_feats.shape[-1]
    residual_basis = eig_vectors[:, : feat_dim - dim]  # (N, N - D)
    # Scale so the residual norm matches the mean max logit  [§4] [eq. 6]
    residual_norms = norm(centred @ residual_basis, axis=-1)  # (K,)
    alpha = train_logits.max(axis=-1).mean() / residual_norms.mean()
    return residual_basis, alpha

# Score one input (online)  [§4] [eq. 5] [eq. 8]
# Output: scalar, higher = more OOD; compare with a threshold set on ID data

def vim_score(feature, logits, origin, residual_basis, alpha):
    # Shift the feature to the bias-free origin  [§3] [eq. 1]
    centred = feature - origin                         # (N,)
    # Virtual logit: scaled norm of the residual  [§4] [eq. 5]
    virtual_logit = alpha * norm(centred @ residual_basis)
    # Log-odds of the ViM probability; same ranking  [§4] [eq. 8]
    return virtual_logit - logsumexp(logits)

Notes on the pseudocode: the score returns the log-odds form of ViM, which the paper says gives the same ranking as the probability §4. The paper does not say which eigen-solver to use; the official code uses a general (non-symmetric) solver, see Paper vs code.

Pipeline, as a figure: fig. 3 shows the three steps (feature, logit, probability).

Hyperparameters

Training is not involved. Every value below is either fixed by the paper or derived from data.

Results

Author-reported. ID dataset: ImageNet-1K. Models: BiT-S-R101x1 (CNN) and ViT-B/16 (pre-trained on ImageNet-21K, fine-tuned on ImageNet-1K) §6 app. A. OOD sets: OpenImage-O (17,632 images, introduced here), Texture (with four ImageNet-overlapping classes removed), iNaturalist (10,000-image subset from §6), and ImageNet-O (2,000 images) tab. 1. Baselines do not need re-training §6. All numbers in percent, from tab. 2.

Source column: the paper's grouping of the information each method uses (prob, logit, feat, label, grad) tab. 2. ViM is best on the average columns for both models; on BiT it is third on iNaturalist §6.1.

Other architectures, average over the four OOD sets, AUROC (A) and FPR95 (F), from tab. 3:

Grouping (BiT only, from tab. 4). MOS needs fine-tuning, the others do not:

Score computation time, in seconds, excluding the network forward pass, numpy on a Skylake CPU tab. 8 app. G:

Sensitivity: varying D over a wide range changes performance little fig. 4. Scaling alpha by a factor reduces performance on average; the paper's definition of alpha is the best of the tested factors fig. 5 §6.4.

Paper vs code

Official code: https://github.com/haoqiwang/vim, at commit dabf9e5b242dbd31c15e09ff12af3d11f009f79c (Apache-2.0). The paper links this repository in its abstract (abstract). The score is implemented in benchmark.py; feature extraction is in extract_feature_bit.py and extract_feature_timm.py/extract_feature_vit.py.

• Principal-space dimension: paper says D = 1000 when N > 1500 and 512 otherwise §6. Code uses 1000 when N >= 2048, 512 when N >= 768, and N // 2 below that benchmark.py. Models in the paper have N = 2048 (BiT) and N = 768 (ViT), so the reported numbers are unaffected, but the two rules differ for 1500 < N < 2048 and for N < 768.
• Eigen-decomposition: paper writes the eigendecomposition of X^T X §4 eq. 4. Code fits an empirical covariance with assume_centered=True (that is X^T X divided by K) and calls np.linalg.eig, a general solver, then sorts the eigenvectors by eigenvalue benchmark.py. Dividing by K does not change the eigenvectors. Using a general solver on a symmetric matrix is not what the paper describes; the paper does not name a solver.
• Origin: paper uses o = -(W^T)^+ b §3 eq. 1. Code computes u = -pinv(w) @ b with w stored as (C, N), so pinv(w) = (W^T)^+ and u = o benchmark.py. Consistent.
• Virtual logit and alpha: paper's alpha uses the sum of max logits over the sum of residual norms on K sampled images §4 eq. 6. Code takes the ratio of means over all training features it is given benchmark.py. Equal in form; the code's K is the 200,000-line list file.
• Score sign and form: code's ID score is -vlogit + logsumexp(logits), and OOD score is -vlogit_ood + energy_ood benchmark.py. This is the negative of the log-odds form §4 eq. 8 for ID, so the code's scores are ID-oriented. Same ranking.
• ReAct clipping: code takes np.quantile(feature_id_train, 0.99) over the whole training feature matrix, flattened, not per dimension [code: benchmark.py#L208]. The paper says only that the percentile is estimated on ID data app. E.
• ViT input: paper names ViT-B/16 and gives no resolution §6. Code uses the 384-input checkpoint vit-base-p16_in21k-pre-3rdparty_ft-64xb64_in1k-384 extract_feature_vit.py with resize to 384 [code: extract_feature_vit.py#L52].
• Extra baseline: the code also computes GradNorm benchmark.py. It does not appear in the paper's tables. No discrepancy found for: the residual definition (projection onto the N - D smallest-eigenvalue directions) §4 benchmark.py, and the ViM score as the log-odds form in §4 eq. 8 benchmark.py. I did not check the Mahalanobis and KL matching settings line by line against app. E.

Pitfalls

• Dimension rule is ambiguous. The paper gives D = 1000 or 512 with a threshold of N > 1500 §6; the code uses N >= 2048 and adds an N // 2 rule for small N benchmark.py. Implementers should state which one they follow. For ViT-like models with N < C, the paper suggests D in [N/3, 2N/3] §6.4.
• Eigen-solver choice. The paper specifies the decomposition only as eigenvectors of X^T X §4 eq. 4. The code uses a general solver and then sorts by eigenvalue benchmark.py. A symmetric solver is the natural choice for a covariance matrix. I did not test whether the two give the same subspace.
• Residual is sensitive to the origin. The residual is computed after shifting the feature by o = -(W^T)^+ b §3 eq. 1. Using the raw feature without this shift changes the score. Only the pseudo-inverse is specified, so implementers must use the same pseudo-inverse.
• Scale of alpha. Alpha is a ratio of two averages over training data §4 eq. 6. The paper links its weaker iNaturalist result to that set having the smallest average residual norm §6.1; the behaviour of alpha in that regime is not analysed further.
• Avoid tuning on the evaluation ID set. The paper says OOD hyperparameter selection should not use the ID set used for FPR95 and AUROC; overfitting inflates results app. E. ViM has almost no free settings (D, and the 200,000 sample count), but the threshold must come from a held-out ID split.
• Results are author-reported, on one ID dataset. Every number is ImageNet-1K. Generalisation to other ID datasets is not tested in the paper.
• Limit of the method. ViM is limited by the quality of the original network's features, because it trains nothing §6.6. It gains least on OOD sets with small residuals §6.6 §6.1.
• OpenImage-O numbers need the dataset. The file list is in the repository README.md, but the images must be obtained from OpenImages. The licence of those images is not checked in this card.
• Inconsistent dataset-size ratio. The abstract says the new dataset is 8.8 times the size of existing ones [abstract]; the introduction says it is 7.8 times larger than ImageNet-O §1. Use the figure from the body with care.
• Runtime excludes the forward pass. The timings in tab. 8 exclude the network forward pass and assume features are already extracted app. G. The paper does not report end-to-end overhead.

Lineage

• Builds on the residual score, which is related to reconstruction error of a nonlinear manifold method §3; the paper cites it as ndiour2020out.
• Builds on the energy score (log-sum-exp of logits). The paper rewrites ViM as the virtual-logit term minus the log-sum-exp term §4 eq. 8 and says it "completes" the energy method with residual information §4.
• Compared against: MSP §6, Energy, ODIN, MaxLogit, KL Matching, Residual, ReAct and Mahalanobis, plus MOS for grouping §6.5.
• Null-space score NuSA is discussed as a related feature-space score §3 eq. 2.
• Superseded by: not stated in the paper or the wiki.