Mahalanobis distance

Overview

The task is to measure how far apart two points are, given their feature vectors. The distance is a squared quadratic form. For two points x_i and x_j, with a matrix Γ, it is 2605.04664 p. 3:

r^2(\mathbf{x}_i, \mathbf{x}_j) = (\mathbf{x}_i - \mathbf{x}_j)\, \Gamma^{-1}\, (\mathbf{x}_i - \mathbf{x}_j)^T

The shared core is this quadratic form. Setting Γ^{-1} to the identity gives the Euclidean case. Setting Γ to the population covariance Σ gives the Mahalanobis case 2605.04664 p. 3. The three papers below use the same form for different jobs.

• Neighbour selection. Rank past cases by distance to a target and fit a local model on the closest ones 2605.04664 p. 2 2605.04664 p. 3.
• Out-of-distribution (OOD) scoring on labelled classes. Use the distance from a feature to the nearest class mean as an OOD score. The Mahalanobis baseline in the ViM paper is this use 2203.10807 §1 2203.10807 app. E.
• OOD scoring on unlabelled data. Fit one Gaussian to the normalised features of unlabelled in-distribution images and score by distance from its mean. No class labels are used 2103.12051 §1 2103.12051 §3.1.

Variants

Euclidean form (Γ^{-1} = I)

The paper writes this as the special case of its general form 2605.04664 p. 3. It is not among the reported experiments, which compare only the other two selections 2605.04664 tab. 2. Card: Evidence-based anomaly detection in clinical domains.

Mahalanobis neighbour selection (Γ = Σ)

The population covariance Σ replaces Γ. The paper does not state which sample Σ is estimated from 2605.04664 p. 3. The k = 40 nearest cases are kept 2605.04664 p. 4. Reported with a Naive Bayes model: AUC PR 0.49 and AUC ROC 0.77, against 0.42 and 0.74 with all cases. Reported with a Bayesian network: AUC PR 0.52 and AUC ROC 0.79, against 0.56 and 0.81 with all cases 2605.04664 tab. 2. Card: Evidence-based anomaly detection in clinical domains.

Weighted Mahalanobis (w scales each difference)

Each attribute difference is multiplied by an importance weight w before the quadratic form. The weights come from the univariate dependence between each attribute in C(x) and the target. The experiments measure that dependence with the Wilcoxon rank-sum statistic 2605.04664 p. 3 2605.04664 p. 4. The paper does not state how the statistic becomes a weight: its scaling, sign or normalisation 2605.04664 p. 3. Reported with a Naive Bayes model: AUC PR 0.60 and AUC ROC 0.80. Reported with a Bayesian network: AUC PR 0.56 and AUC ROC 0.80 2605.04664 tab. 2. Card: Evidence-based anomaly detection in clinical domains.

Class-centroid Mahalanobis OOD score

The score is the minimum Mahalanobis distance from a feature to the class centroids 2203.10807 §1. It uses the final feature only; the paper reports that adding lower-level features degrades the BiT result, to 56% average AUROC 2203.10807 app. E. The class-wise mean vectors and the precision matrix are estimated from 200,000 random training samples, and the ground-truth class labels are used 2203.10807 app. E. Results are author-reported on ImageNet-1K ID data with four OOD sets: average AUROC 86.62% and FPR95 53.34% with BiT-S-R101x1, and 96.02% and 19.45% with ViT-B/16 2203.10807 tab. 2. Scoring time, excluding the forward pass, is 2135.13 s on OpenImage-O against 1.49 s for ViM, on a Skylake CPU with numpy 2203.10807 tab. 8. Card: ViM: Out-Of-Distribution with Virtual-logit Matching.

Self-supervised feature Mahalanobis score (SSD)

No labels are needed for the normal data 2103.12051 §1. A backbone is first trained with a contrastive loss on unlabelled in-distribution images 2103.12051 §3, eq. 1. The score then uses one cluster: the mean and sample covariance of the L2-normalised training features, and the squared distance of a normalised test feature from them 2103.12051 §3.1, eq. 2. The paper's few-shot extension subtracts a second Mahalanobis term built from OOD samples, with Ledoit-Wolf shrinkage for that covariance [arxiv:2103.12051 §3.2, eq. 3, eq. 4]. Author-reported, averaged over four CIFAR-10/100 pairs: AUROC 88.7 against 80.9 for the best unlabelled baseline with a full row, and 92.5 with five OOD samples per class 2103.12051 tab. 1. The backbone is ResNet-50 trained for 500 epochs at batch size 512 2103.12051 §4.1. The paper states no compute cost. Temperature matters: on a ResNet-18 ablation, AUROC is 66.7 at tau = 0.001 and 89.5 at tau = 0.5 2103.12051 tab. 5. Card: SSD: A Unified Framework for Self-Supervised Outlier Detection.

When to use which

Open choices for any implementation: the sample used for Σ, the weight formula, and whether to regularise Σ. The paper leaves all three unstated 2605.04664 p. 3. With 18 binary attributes and 40 neighbours, a covariance estimate can be unstable. This is an implementer's concern, not a result of the paper. For the SSD variant, the paper's Algorithm 1 normalises features by L2 norm only 2103.12051 alg. 1. Whether to also standardise each dimension is an implementer's choice; the SSD card lists what the released code does SSD card.

Implementation sketch

Adapted from the papers' descriptions; neither gives pseudocode. The first sketch is neighbour selection, the second the OOD score. The SSD variant differs from the second only in its inputs: features are L2-normalised, the training set is unlabelled, and there is a single cluster with no minimum over classes 2103.12051 §3.1 2103.12051 alg. 1.

# Weighted Mahalanobis neighbour selection  [arxiv:2605.04664 p. 3]
# Input:  target: vector C(x) for the scored case, shape (d,)
#         past: past cases, one row each, shape (n, d), target excluded
#         weights: w, shape (d,); all ones gives plain Mahalanobis
#         k: neighbourhood size (40 in the paper)
# Output: indices of the k nearest past cases

def select_neighbours(target, past, weights, k):
    # Population covariance. The paper does not say which sample; using
    # past is an implementer's choice  [arxiv:2605.04664 p. 3]
    cov_inv = inverse(covariance(past))
    # Scale each attribute difference by its importance weight  [p. 3]
    diff = weights * (past - target)            # element-wise, (n, d)
    # Squared distance per row; same ranking as the unsquared form  [p. 3]
    dist = sum((diff @ cov_inv) * diff, axis=1)
    # Keep the k nearest cases as the training set E  [p. 4]
    return argsort(dist)[:k]

# Class-centroid Mahalanobis OOD score  [arxiv:2203.10807 §1] [app. E]
# Input:  train_feats (K, N), train_labels (K,): ground-truth labels
#         feature (N,): the input to score
# Output: scalar, higher = more OOD

def fit_centroids(train_feats, train_labels, classes):
    # Class-wise mean vectors, from ground-truth labels  [app. E]
    means = stack([train_feats[train_labels == c].mean(axis=0)
                   for c in classes])                   # (C, N)
    # One precision matrix from the class-centred features  [app. E]
    centred = train_feats - means[train_labels]         # (K, N)
    precision = inverse(centred.T @ centred / len(train_feats))
    return means, precision

def mahalanobis_ood_score(feature, means, precision):
    # Squared distance to every class centroid  [§1]
    diffs = feature - means                             # (C, N)
    dists = sum((diffs @ precision) * diffs, axis=1)    # (C,)
    # Distance to the nearest centroid; the paper's score is this minimum  [§1]
    return min(dists)

The paper names the precision matrix but not its estimator beyond the sample size, so the centred empirical covariance above is an implementer's choice 2203.10807 app. E.

Contradictions

Papers that use the distance for different tasks, neighbour selection and OOD scoring, measure different things, so their numbers should not be compared directly. The two OOD papers also use different ID data and networks. The SSD paper's supervised Mahalanobis rows 2103.12051 tab. 2 are on CIFAR-100 and are not the ViM baseline on ImageNet-1K 2203.10807 tab. 2.

Within the clinical paper, the neighbourhood step does not help every model. It raised the Naive Bayes AUC PR but lowered the Bayesian network's AUC ROC from 0.81 to 0.79 2605.04664 tab. 2. The authors attribute the Bayesian network's drop to its complexity and small samples 2605.04664 p. 5.

Within the SSD paper, the reported gains depend on the OOD set. Against the supervised Mahalanobis baseline, SSD is ahead on CIFAR-10 OOD from CIFAR-100 (69.6 against 55.3) and behind on Texture (82.9 against 98.8) 2103.12051 tab. 2.

Papers

• SSD: A Unified Framework for Self-Supervised Outlier Detection: uses the Mahalanobis distance of normalised self-supervised features, fitted on unlabelled in-distribution images, as a label-free OOD score.
• ViM: Out-Of-Distribution with Virtual-logit Matching: uses a class-centroid Mahalanobis score as one of its baselines for OOD detection; the paper's own method, ViM, scores with a virtual logit instead.
• Evidence-based anomaly detection in clinical domains: uses a weighted Mahalanobis distance to choose the k most similar past patients, then scores a decision with a Bayesian network or Naive Bayes model fitted on them.