Metric learning

Overview

Metric learning trains an encoder so that a chosen distance (or similarity) ranks related inputs above unrelated ones. The review describes the field as spanning metric learning, NLP, vision, audio and reinforcement learning, without a common framework, and places the older metric-learning losses at the root of the later contrastive family 2010.05113 §I 2010.05113 §IV. The shared core is a score S(\mathbf q,\mathbf k) (distance, dot product, cosine or bilinear) and a loss that pulls positive pairs together and pushes negatives apart 2010.05113 §III.A 2010.05113 §III.E.1.

The choices that separate the variants are (1) which comparisons enter the loss, (2) how many per query, (3) whether classes are represented by samples or by learned proxies, and (4) how the trained embedding is used at test time, for example by nearest class mean 2004.00440 §3.1.

Two constraints from the review hold across variants. First, the score matters: the dot product is unbounded and scales with vector norms, so cosine or an explicit norm penalty is the usual remedy 2010.05113 §III.E.1. Second, using only positive pairs is not enough: minimising the positive distance alone can be solved by a constant encoder, so negatives or an architectural constraint must be present 2010.05113 §III.E.

Variants

Pair loss (energy-based, Siamese)

Positive pairs are pulled to zero distance; negative pairs contribute only while they lie inside a margin m 2010.05113 eq. 2. The review credits Chopra, Hadsell and LeCun for the pair loss and Bromley et al. for the Siamese setting in metric learning 2010.05113 §IV 2010.05113 tab. 1. Note that eq. 2 places the margin on the squared distance, while the text calls m a radius; check which form an implementation uses 2010.05113 §III.E.2. Card: Contrastive Representation Learning: A Framework and Review.

Triplet loss

An anchor must be closer to its positive than to its negative by a margin 2010.05113 eq. 3. It is the most common energy-based loss in the review's tables: image retrieval, sentence embeddings, knowledge-graph embeddings (TransE) and speech-to-text matching (CSTNet) all use it 2010.05113 tab. 2 2010.05113 tab. 3 2010.05113 tab. 5 2010.05113 tab. 6. Each query sees one positive and one negative per step, so training converges slowly and hard-negative mining is essential 2010.05113 §III.E.2. The review cites Hermans et al. for the importance of mining hard samples 2010.05113 tab. 1.

Citation-defined triplets with hard negatives (SPECTER)

Here relatedness comes from a citation graph, not from class labels: a paper is a positive for a query when the query cites it 2004.07180 §2.3. Each query gets up to five triples: two hard negatives, which are papers cited by a paper the query cites but not cited by the query, and three random easy negatives 2004.07180 §2.4 2004.07180 §4. The loss is a triplet margin loss on L2 distance with margin 1 2004.07180 eq. 2. The paper reports that cosine distance did worse for this loss 2004.07180 §2.3. Only title and abstract are needed at inference, so papers without citations can still be embedded 2004.07180 §2.5. Dropping the hard negatives lowers the author-reported 12-metric average from 80.0 to 78.4 2004.07180 tab. 2. In the paper's comparison, fine-tuning SciBERT on each user-activity signal scored between 76.0 and 78.0 against 80.0 for the fixed embedding 2004.07180 tab. 3. Card: SPECTER: Document-level Representation Learning using Citation-informed Transformers.

Siamese pooled encoder with cosine score (Sentence-BERT)

One pretrained encoder (BERT or RoBERTa) is applied to each sentence separately. Its token outputs are mean-pooled into one vector, and two vectors are compared by cosine similarity, so each sentence is embedded once and the vectors can be stored and searched 1908.10084 §3 1908.10084 §6. Three training objectives are given: a three-way NLI softmax over the concatenation of the two embeddings and their absolute difference, a mean-squared-error regression of the cosine against a gold score, and a triplet hinge on Euclidean distance with margin 1 1908.10084 §3. The softmax head is used only for training and is dropped at inference 1908.10084 §6. On the STS benchmark dev set, mean pooling beats CLS and MAX pooling (87.44 against 86.62 and 69.92) 1908.10084 tab. 6. The authors report that the closest pair among 10,000 sentences takes about 65 hours with a cross-encoder and about 5 seconds with this setup, on their hardware 1908.10084 §1. The paper reports SBERT-NLI-large at 76.55 average Spearman over seven STS tasks with no STS training data 1908.10084 tab. 1. Card: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.

Single-center loss with softmax (face forgery)

Real faces are pulled toward one learnable center, and fake faces must sit at least a margin farther from it than the real faces do 2103.09096 eq. 1 2103.09096 §3.3. The paper's reason for a one-sided rule is that triplet or center losses would squeeze fakes from many manipulation methods into one cluster, though these fakes differ from each other [arxiv:2103.09096 §1] 2103.09096 §3.3. The loss is added to softmax cross-entropy with weight \lambda 2103.09096 eq. 6. The margin is m\sqrt{D}, where D is the embedding dimension, so the final model's m = 0.3 gives about 9.49 at D = 1000; this value is derived on the card and not stated in the paper 2103.09096 §4.2.1. On FF++ c40, softmax alone reaches AUC 0.860 and pAUC(0.1) 0.652, centre plus softmax 0.868 and 0.666, triplet plus softmax 0.863 and 0.655, and single-center plus softmax 0.916 and 0.790 2103.09096 tab. 1. These are author-reported. The center's learning rate and the batch-level handling are not stated 2103.09096 §3.3 2103.09096 Pitfalls. Card: Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection.

Reconstruction error as the class score (few-shot)

Each class is scored by how well its support feature maps can rebuild the query's feature map. The support features of one class, across all shots and spatial positions, form one pool; a ridge regression rebuilds the query from that pool, and the class score is the negative mean squared error over locations 2012.01506 §3.1 2012.01506 eq. 4. The solve is closed form, so no gradient step runs at test time 2012.01506 §3.4 2012.01506 §4.2. Beyond the backbone, the learned parameters are a penalty scale, a recalibration factor and the softmax temperature 2012.01506 §3.2. The paper reports 83.16% (1-shot) and 92.59% (5-shot) on cropped CUB with ResNet-12, against 80.80% and 91.19% for its best listed baseline; these are author-reported 2012.01506 tab. 3. On mini-ImageNet the 1-shot result is slightly below FEAT (66.45% against 66.78%) [arxiv:2012.01506 tab. 7]. Pre-training needs a separate head with one learnable map per base class, which is discarded afterwards 2012.01506 §3.6. Card: Few-Shot Classification with Feature Map Reconstruction Networks.

Softmax plus triplet embedding for tracking (POI)

The embedding is a 128-d appearance feature for re-identifying people across frames. A GoogLeNet-like network is trained with a softmax identity loss, which makes features discriminative, plus a triplet loss, which keeps same-identity cosine distances small 1610.06136 §2. Features are compared by cosine similarity, and the cosine is one factor in a product with motion and shape affinities 1610.06136 §3, eq. 1 1610.06136 §3, eq. 4. The triplet margin, the triplet sampling and the weight between the two losses are not stated 1610.06136 §2. On MOT16 train, with the same detector and online tracker, the paper's feature reaches MOTA 65.2 against 63.6 for a baseline labelled "GoogLeNet feature" 1610.06136 tab. 2; the card notes that this label is not confirmed by the text. These are author-reported. Card: POI: Multiple Object Tracking with High Performance Detection and Appearance Feature.

Multi-pair losses (lifted structure, N-pair)

Lifted-structure and its generalisation use many positives and negatives per query within a batch, which gives more comparisons per gradient step than a triplet 2010.05113 §III.E.2. The N-pair loss is a non-parametric classification over one positive and several negatives; the review writes Sohn's multi-class N-pair loss in the same form as the NCE-global loss, with in-batch samples as negatives 2010.05113 §III.E.3 2010.05113 eq. 10. The review's discussion says negative quality matters, hard negatives included, as well as the number of negatives 2010.05113 §VI.B.

Circle loss on dense correspondences (PREDATOR)

Here the pairs are points of two 3D scans, and the positives are the ground-truth correspondences under a known transform. The loss pulls each anchor's positives within a radius r_p of it and pushes its negatives beyond a radius r_s, with a log-sum-exp over the pairs and a per-pair weight that grows with the distance from its margin 2011.13005 eq. 8 2011.13005 §3.5. The margins are \Delta_p = 0.1 and \Delta_n = 1.4, and the scale \gamma is 24 for 3DMatch, 64 for ModelNet40 and 48 for odometryKITTI 2011.13005 §3.5 [arxiv:2011.13005 tab. 6]. The circle loss is one of three terms trained jointly with an overlap and a matchability term 2011.13005 §3.5. The paper gives its own loss weights as equal, and it does not report a circle-loss ablation: its ablation varies the overlap-attention parts instead 2011.13005 tab. 3. The paper's footnote says the loss was added to the released code after publication and is not described in the paper body, so the code is the reference for it 2011.13005 §3.5 2011.13005 Paper vs code. Card: PREDATOR: Registration of 3D Point Clouds with Low Overlap.

Proxy-based classifier (LSC in PODNet)

Each class is represented by K learned proxies instead of by sample pairs. The score of a class is a soft-attention-weighted sum of cosine similarities to its proxies 2004.13513 eq. 9, trained with a modified NCA loss that has a margin \delta on the target logit and a hinge 2004.13513 eq. 10 2004.13513 §3.2. The paper reports that NCA converged faster than plain cross-entropy 2004.13513 §3.2. New classes get their proxies from k-means centroids of their features under the current model, which the paper reports adds about 1 point on CIFAR100 2004.13513 §3.2. In its class-incremental experiments, going from one to ten proxies per class added 1.18 points on CIFAR100 and 1.51 on ImageNet100 with ten steps; the paper reports K as stable between 5 and 15 2004.13513 §6.3. Card: PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning.

Triplet loss with nearest-class-mean classification

An embedding network trained with the triplet loss classifies a test image by the nearest class prototype, the mean embedding of that class's training images 2004.00440 §3.1 2004.00440 eq. 2 2004.00440 eq. 3. In class-incremental learning on six-task CUB-200-2011 with ResNet-18, the paper reports average incremental accuracy of 37.4 for embedding fine-tuning against 11.4 for softmax fine-tuning 2004.00440 tab. 1. Stored prototypes go stale as the embedding moves. Semantic Drift Compensation shifts each stored prototype by a Gaussian-weighted average of the embedding drift measured on current-task images, without any old data 2004.00440 §4.1 2004.00440 eq. 10 2004.00440 eq. 11. The gains are author-reported and differ by dataset; see Contradictions. Card: Semantic Drift Compensation for Class-Incremental Learning.

Cross-view triplet with region branches (FSRA)

FSRA encodes each view with a ViT-S and adds region branches next to a global one. It splits the patch tokens into regions by their mean activation and pools each region 2201.09206 §3.1 2201.09206 §3.2 2201.09206 eq. 3 2201.09206 eq. 4. Each region vector and the global feature go through a classifier head trained with cross-entropy, and the triplet loss is applied to every branch with margin 0.3, using only pairs from different views 2201.09206 §3.4 2201.09206 eq. 10. Each satellite image is repeated k times with augmentation; the paper reports k=3 as best at batch size 8 2201.09206 §3.3 2201.09206 fig. 11. On University-1652, drone-to-satellite R@1 is 84.51 with k=3, against 75.93 for LPN with ResNet-50 2201.09206 tab. 2. These are author-reported. Card: FSRA, a transformer with region alignment for UAV-view geo-localization.

Symmetric InfoNCE with GPS and similarity-based hard negatives (cross-view geo-localisation)

One ConvNeXt-B encodes both street-view and satellite images, with mean-pooled features and no attention pooling or polar transform 2303.11851 §3.2 2303.11851 fig. 2. Each batch is scored in both directions: every street image is a query over the satellite images, and the reverse, with the matching pair as the positive and all other batch items as negatives 2303.11851 §3.1 2303.11851 eq. 1. Negatives are chosen in two stages. Early in training, nearby GPS locations are placed in the same batch 2303.11851 §3.3. Later, Dynamic Similarity Sampling takes the top K most similar samples per query under cosine similarity, keeps the first k/2 of them and draws k/2 at random from the rest of the pool; the pool is recomputed every e epochs, with k = 64, K = 128 and e = 4 2303.11851 §3.4. On CVUSA, GPS plus DSS reaches R@1 98.68 against 97.83 for random batches 2303.11851 supp. "Sampling Strategies", tab. 10. The symmetric loss beats either single direction on VIGOR SAME (R@1 77.86 against 77.49 and 74.69) 2303.11851 supp. "Symmetric InfoNCE Loss", tab. 8. All numbers are author-reported. The triplet and InfoNCE comparison is under Contradictions. Card: Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation.

Triplet ranking with domain and semantic losses (zero-shot sketch retrieval)

Two VGG-16 encoders with attention embed sketches and photos into one space, ranked by L2 distance 1904.03451 §4.2. Training sums three terms: a triplet ranking loss on L2 distances with margin \mu>0 1904.03451 eq. 1 1904.03451 §4.3; a domain loss that labels sketches and photos as two domains 1904.03451 eq. 2; and a semantic loss that scores each embedding against the word vector of its category, using a half cosine distance 1904.03451 eq. 3. The paper does not state the margin or the loss weights, and does not state how negatives are sampled. The released code fixes the margin at 1.0 and samples negatives by distance between category vectors loss.py quickdraw_extended.py. In the ablation, the triplet-only baseline scores 0.3020 mAP on Sketchy, 0.0590 on TU-Berlin and 0.0354 on QuickDraw. Adding attention, domain and semantic losses raises these to 0.3691, 0.1094 and 0.0752 1904.03451 tab. 3. These are author-reported full-gallery mAP values on unseen test categories. Card: Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval.

Bi-encoder with additive-margin softmax (speech-text retrieval)

Speech and text are embedded separately, and each batch is scored as a softmax over its in-batch pairs, with the matching pair on the diagonal and an additive margin. The loss runs in both directions, speech to text and text to speech 2205.12446 §4.3. Only the 600M mSLAM model is fine-tuned for this retrieval, so the paper offers no second model to compare with 2205.12446 tab. 5. The margin, scale, similarity function and pooling are not stated [arxiv:2205.12446 §4.3]. Author-reported P@1 over all 102 languages is 76.9 for speech to text and 74.4 for text to speech, but CJK falls to 4.7 in both directions 2205.12446 tab. 5. Card: FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech.

When to use which

No card compares proxy-based losses with triplet or pair losses head to head. The table above follows from each method's constraints, not from a measured ranking. For face embeddings trained through a classification head, see angular-margin-loss.

Implementation sketch

The common core is one batch, one score and one margin loss. The variants differ in which pairs enter and how the loss normalises.

# Pair and triplet losses on a batch  [arxiv:2010.05113 eq. 2] [eq. 3]
# Inputs: embeddings z (B, d), margin m; the review's margin is on D^2

def pair_loss(z_q, z_k, is_pos, m):
    # Squared distance on the embeddings  [eq. 2]
    d2 = sum((z_q - z_k) ** 2)
    # Positives are pulled in; negatives count only inside the margin  [eq. 2]
    return where(is_pos, d2, max(0, m - d2))

def triplet_loss(z_a, z_p, z_n, m):
    # Anchor nearer to positive than to negative by at least m  [eq. 3]
    d_ap = sum((z_a - z_p) ** 2)
    d_an = sum((z_a - z_n) ** 2)
    return max(0, d_ap - d_an + m)

# Proxy-based logits, K proxies per class  [arxiv:2004.13513 eq. 9]
# Inputs: embedding z (d,), proxies (C, K, d)

def proxy_logits(z, proxies):
    # Cosine similarity of z to every proxy of every class  [eq. 9]
    sims = cosine(z, proxies)                  # (C, K)
    # Soft attention over the K proxies of each class  [eq. 9]
    attn = softmax(sims, axis=-1)
    return sum(attn * sims, axis=-1)           # (C,) one logit per class

# Nearest-class-mean prediction  [arxiv:2004.00440 eq. 2] [eq. 3]
# Inputs: query embedding z (d,), prototypes (C, d), one per class

def nearest_class_mean(z, prototypes):
    # Squared distance to each class mean  [eq. 2]
    d2 = sum((z - prototypes) ** 2, axis=-1)   # (C,)
    # Predicted class is the closest prototype  [eq. 2]
    return argmin(d2)

Notes that follow from the cards. The margin in eq. 2 is on the squared distance, not on the distance 2010.05113 §III.E.2. The PODNet paper states cosine similarity, but the official code uses the negative squared Euclidean distance of L2-normalised vectors, which on unit vectors equals 2\cos - 2; the code's logits are therefore twice the cosine logits at the same scale 2004.13513 Paper vs code. The code also builds a per-target margin that is never used, so the margin \delta in eq. 10 has no effect there 2004.13513 Paper vs code. Anyone reproducing LSC has to decide which of these to follow and should say so.

For the nearest-class-mean sketch, the paper writes the distance only as a generic dist(·,·), while the official code uses Euclidean distance 2004.00440 Paper vs code. The triplet margin is not stated in the SDC paper, and the code default is 0 2004.00440 Pitfalls. FSRA uses margin 0.3 on L2-normalised features, whereas its paper writes the raw distance; with normalisation, the margin acts on the unit sphere 2201.09206 Paper vs code 2201.09206 Pitfalls.

The sketch's triplet loss uses squared distances. SPECTER's eq. 2 uses L2 distance with margin 1 2004.07180 eq. 2, so the same margin value means different things on the two scales. Pick one, state it, and keep it fixed across training and evaluation. Sentence-BERT's triplet hinge also uses Euclidean distance with margin 1 in the paper 1908.10084 §3, but the official library's default triplet margin is 5, so a run that takes the default is not the paper's setting 1908.10084 Paper vs code.

For the circle loss, the PREDATOR code clips the pair weights at zero and detaches them from the graph, and it keeps only correspondences within the positive radius minus 1 mm, then randomly keeps at most n_p of them 2011.13005 Paper vs code. A reimplementation that follows eq. 8 literally will weight pairs differently from the released code.

For the single-center loss, the margin is m\sqrt{D}, so changing the backbone changes the margin unless m is rescaled 2103.09096 eq. 1. The loss needs at least one real and one fake sample in each batch, which the paper does not address, and the batch size is not stated 2103.09096 Pitfalls. The card records no official code, so the margin and loss form here come from the paper alone 2103.09096 Paper vs code.

For FRN, the penalty scales with kr/d so that repeating a support pool does not change the reconstruction, and the Woodbury form inverts a d \times d matrix once per class per episode 2012.01506 §3.2 2012.01506 §3.4. The released code differs from the paper in the auxiliary loss and adds a small constant to the penalty 2012.01506 Paper vs code.

Contradictions

The review's slow-convergence remark concerns pair and triplet losses 2010.05113 §III.E.2, while PODNet's concerns NCA against cross-entropy on proxies 2004.13513 §3.2; these do not conflict. The PODNet paper and code disagree on the similarity function and on the margin, as described in the sketch notes 2004.13513 Paper vs code. Further disagreements:

• Hard-negative mining. The review calls hard-negative mining essential for triplet losses 2010.05113 §III.E.2. The SDC demo selects a triplet loss that pairs each anchor by batch position, not by hardest negative 2004.00440 Paper vs code. These are different settings, so neither settles the other. FSRA's code uses the hardest cross-view positive and negative, which the paper does not describe 2201.09206 Paper vs code. SPECTER measures a smaller effect of hard negatives than "essential" suggests: removing them lowers its author-reported average by 1.6 points 2004.07180 tab. 2. That is one setting, and the review does not quantify its claim.
• Triplet loss collapse. The review calls hard-negative mining essential for triplet losses 2010.05113 §III.E.2. Sample4Geo states that hard negatives with a triplet loss can make the model collapse 2303.11851 §1 2303.11851 §3.1. In its CVUSA run, a triplet loss with one negative per anchor reached R@1 0.00, while a soft-margin triplet reached 91.83 and InfoNCE 98.68 2303.11851 supp. "Loss comparison", tab. 9. The paper does not state the triplet run's margin, mining or batch composition, so this result does not show that triplet losses fail in general.
• Cosine versus L2. The review recommends normalising or using cosine when the dot product depends on norms 2010.05113 §III.E.1. SPECTER reports that L2 distance beat cosine distance for its triplet loss 2004.07180 §2.3. Sentence-BERT scores with cosine at inference and reports that negative Manhattan and Euclidean distances gave roughly the same results, without numbers 1908.10084 §4. The papers compare different things, so none of them decides the choice for another dataset.
• SDC gains differ by dataset. Within one paper, E-LwF+SDC over E-LwF is +1.3 on six-task CUB-200-2011 and +6.2 on six-task Flowers-102 2004.00440 tab. 1, but 0.0 on eleven-task CIFAR100 and −1.0 on eleven-task ImageNet-Subset 2004.00440 tab. 5. The paper does not resolve this difference; a reader should not assume SDC helps on every dataset.
• FSRA table versus text. The k = 1 row of Table 2 matches the KL-plus-triplet row of Table 7, but §3.4 says the KL loss was not used in the final model. The triplet-only row differs slightly 2201.09206 tab. 2 2201.09206 tab. 7 2201.09206 §3.4.

Papers

• Contrastive Representation Learning: A Framework and Review: a framework that places pair, triplet, N-pair and NCE losses in one taxonomy of similarity, encoders, heads and losses.
• PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning: a proxy-based classifier (LSC) with K proxies per class, used with a distillation loss in class-incremental learning.
• Semantic Drift Compensation for Class-Incremental Learning: a triplet-trained embedding network with nearest-class-mean classification, plus a drift-based correction of stored class prototypes.
• SPECTER: Document-level Representation Learning using Citation-informed Transformers: a SciBERT encoder trained with a triplet loss on citation-defined triples, giving one embedding per paper from its title and abstract.
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks: a BERT or RoBERTa siamese encoder with mean pooling, trained with NLI softmax, cosine regression or triplet losses, and compared by cosine similarity.
• FSRA, a transformer with region alignment for UAV-view geo-localization: a ViT-S with region branches and cross-view triplet losses for drone-to-satellite matching.
• Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval: two attention-equipped VGG-16 encoders trained with triplet, domain and semantic losses for sketch-to-photo retrieval of unseen categories.
• Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation: one shared ConvNeXt-B trained with symmetric InfoNCE, using GPS-based and then similarity-based hard negatives.
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech: a 102-language speech benchmark whose speech-text retrieval baseline trains a margin softmax over in-batch pairs in both directions.
• PREDATOR: Registration of 3D Point Clouds with Low Overlap: a point-cloud registration network whose per-point descriptors are trained with a circle loss over ground-truth correspondences, alongside overlap and matchability terms.
• Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection: an Xception with a learned DCT-frequency branch, trained with softmax plus a single-center loss that pulls real faces to a center and pushes fakes beyond a margin.
• Few-Shot Classification with Feature Map Reconstruction Networks: scores each class by the ridge-regression reconstruction error of the query's spatial feature map from that class's support maps.
• POI: Multiple Object Tracking with High Performance Detection and Appearance Feature: a Faster R-CNN tracker whose 128-d appearance embedding is trained with softmax and triplet losses and combined with motion and shape affinities.