Contrastive learning

Overview

Contrastive learning trains an encoder without per-sample labels by deciding which inputs count as a match. A query is pulled towards its positive and pushed away from negatives. A review of the field splits every method into four parts: a similarity distribution that supplies positive and negative keys, an encoder, an optional transform head, and a loss made of a scoring function and a form 2010.05113 §III.A 2010.05113 §III.B–E. The parts can be swapped independently, so a new method is mostly a choice of positives, negatives and head.

The most common loss is InfoNCE, a softmax over one positive and K negatives: \mathcal L_i = -\log \frac{\exp(z_i^\top z_i'/\tau)}{\sum_{j=0}^{K}\exp(z_i \cdot z_j'/\tau)}, with temperature \tau and the positive included in the denominator 2010.05113 eq. 1 2010.05113 §II.C. The same loss in the survey's notation is eq. 9 of 2006.08218 §4. Minimising InfoNCE maximises a lower bound on the mutual information I(x,y) \ge \log N - \mathcal L 2006.08218 §6.2.2, but the same survey stresses that MI and downstream accuracy correlate only loosely 2006.08218 §6.2.2, and Tschannen et al. argue that MI maximisation alone does not explain why contrastive losses work 2010.05113 §III.E.4.

A second view of the same loss splits it into an alignment term, which pulls positive pairs together, and a uniformity term, which spreads the embeddings over the hypersphere 2006.08218 §6.3.2. The authors of that survey report that both terms are needed, and that an over-weighted uniformity term causes collapse 2006.08218 §6.3.2.

A survey of semi-supervised and unsupervised visual learning files instance discrimination under local smoothness across views, and dense or graph contrast under local smoothness across neighbours 2208.11296 §4.1.1 tab. 3. It is a taxonomy with no experiments of its own 2208.11296 §1, so its claims are conceptual.

A survey of contrastive self-supervised learning for vision and NLP collects linear-probe numbers from the original papers, for example SwAV at 75.3% top-1 on ImageNet against 76.5% for a supervised ResNet-50 2011.00362 tab. 1. It re-runs no method, and its tables mix protocols, so its numbers are not a like-for-like ranking 2011.00362 §7 tab. 1–4.

What counts as a positive decides what the model learns to ignore. Positives can come from augmentations of one image, from several sensors or modalities, from neighbouring frames or sentences, or from patches of the same image 2010.05113 §III.B.1–4. They can also be defined at a coarser level, such as all segments of one patient 2005.13249 §4.1, or a class mean 2109.12273 eq. 4. Augmentations set the invariances: many contrastive methods do poorly without crop and colour distortion even with the same architecture and loss 2010.05113 §II.C. The reviews also flag that invariances learned from object-centred ImageNet crops may be wrong for scene-rich data 2010.05113 §VI.A.

Variants

End-to-end with in-batch negatives (SimCLR-style)

Both query and key encoders receive gradients. From a batch of N images, 2N views are made; each positive pair has 2(N−1) negatives taken from the batch, scored with cosine similarity 2006.08218 §4.2.2 eq. 13–14. The authors credit a learnable non-linear projection before the loss, longer training and deeper networks 2006.08218 §4.2.2. The cost is memory: all activations live on the GPU, so the batch size caps the number of negatives 2010.05113 §III.C.1. The survey lists the SimCLR head as a two-layer MLP 2010.05113 tab. 3.

The same in-batch recipe, at much larger scale, is used for text and code. The Transformer encoder of 2212.03533 §4.1 is trained on query and passage pairs with a temperature of 0.01 and a batch of 32,768 2212.03533 tab. 11. The authors report that in-batch negatives beat MoCo when the batch is large 2212.03533 §4.1 2212.03533 tab. 8. Text and Code Embeddings by Contrastive Pre-Training uses the same idea with web text neighbours and docstring–function pairs, and the authors show that batch size is the main lever: for a 300M text model, NQ dev MRR@10 rises from 71.4 at batch 1536 to 84.7 at batch 12288 2201.10005 tab. 9. Its temperature is trainable and its initial value is not stated 2201.10005 §2.2.

Memory bank

Each sample's detached representation is stored in a bank, and keys are read from it. Samples are refreshed only once per epoch, so the keys can be up to one epoch old 2010.05113 §III.C.2 2010.05113 Pitfalls. This removes the batch-size limit at the price of stale keys.

Momentum encoder with a queue (MoCo-style)

The key encoder's parameters are an exponentially weighted average of the query encoder's, and the newest keys are pushed into a queue while the oldest are dropped 2010.05113 §III.C.2. The queue held up to 65536 keys in the original method 2006.08218 §4.2.2. The momentum coefficient is not stated in either survey 2006.08218 Hyperparameters 2010.05113 §III.C.2. The queue keys are more consistent than memory-bank entries, which are refreshed once per epoch 2010.05113 §III.C.2. MoCo v2 shows that SimCLR's tricks help the queue method too 2006.08218 §4.2.2.

Momentum encoder for ViT backbones with a frozen patch projection (MoCo v3)

MoCo v3 drops the queue and uses in-batch keys from a momentum encoder with a ViT backbone and a prediction head on the query side only 2104.02057 §3. Each query is scored against every other image's key in the batch, and the loss is symmetrised over the two views 2104.02057 alg. 1. The authors report that training is unstable in a way that can hide: the run may lose a few points of accuracy without diverging, and dips in the training curve are small 2104.02057 §4.1. Their main stability change is to freeze the random patch projection with a stop-gradient, so only the rest of the network learns 2104.02057 §4.2. In the authors' ablation with ViT-B/16 at 100 epochs, learned and random projections reach 72.2 and 72.8 linear top-1 at base lr 1.0e-4, and 71.7 and 73.4 at 1.5e-4 2104.02057 fig. 5. The gain is therefore author-reported and depends on the learning rate. The paper's own discussion says the trick reduces instability but does not remove it when the learning rate is too large 2104.02057 §4.2.

Negative-free (BYOL, SimSiam)

An online network with an extra predictor regresses the output of an exponential-moving-average target network, using the normalised squared distance \|\bar q_\theta(z_\theta)-\bar z'_\xi\|_2^2 2006.08218 §4.2.2 eq. 15. No negatives are used. The survey reports that SimSiam identifies the stop-gradient as the critical component 2006.08218 §4.2.2. Without explicit negatives, a constant encoder also minimises the positive-pair distance, so collapse must be prevented by architecture, such as the predictor and the EMA target 2010.05113 Pitfalls. In a survey table, BYOL loses 0.3% when its batch goes from 512 to 4096, against 1.4% for SimCLR 2006.08218 Results. These are second-hand figures from one ImageNet setting.

A dynamics analysis of this family gives the mechanism. In a linear two-layer model, the predictor, the stop-gradient, the EMA target and weight decay together keep the online weights from collapsing 2102.06810 §2–§3. The same paper proposes DirectPred, which sets the predictor from the eigendecomposition of a moving average of feature correlations, with p_j = \sqrt{s_j} + \epsilon \max_j s_j and no gradient steps 2102.06810 §4 eq. 18–19. On ImageNet at 300 epochs the author-reported top-1 is 72.4 for DirectPred against 69.9 for a gradient-trained linear predictor, with 72.5 as the reference for a two-layer BYOL predictor 2102.06810 tab. 9. Removing EMA from a symmetric predictor on STL-10 drops top-1 from 72.85 to 36.62 2102.06810 tab. 3. Each DirectPred update needs a d×d eigendecomposition, which the authors reduce by updating only every few minibatches 2102.06810 §4.

Cluster-based (SwAV, DeepCluster)

Views of one image must receive the same cluster assignment, so the loss is a swapped prediction over clusters and no instance negatives are used 2006.08218 §4.2.1 2010.05113 tab. 3. The reported gain over instance discrimination is better results at small model size and better compute efficiency 2006.08218 Results.

Cluster-based positives from geometric matches (FeudalNav-style SMoG)

FeudalNav learns a latent memory for image-goal navigation without labels or odometry. It uses a modified SMoG model. A new view joins the cluster whose SuperGlue match confidence is highest, if that confidence is above 0.7, and otherwise starts a new cluster; positives are then sampled from each cluster 2402.12498 §4.1 §5.2. Standard augmentations are deliberately not used 2402.12498 §4.1. At inference a view joins the nearest cluster centre if its feature MSE is below 300, which depends on a feature scale the paper does not state 2402.12498 §4.1 §5.2. The card does not give the contrastive loss, so this entry covers only how positives are chosen. The feature comparison is a single Gibson trajectory, where SMoG keeps ground-truth distances better than MoCo, SwAV and ResNet-18 2402.12498 fig. 7 §7.1.

Patient-level positives (CLOCS)

The positive set is widened from "the same instance" to "any representation of the same patient". Given a mini-batch, two transformation operators produce views, and every pair of instances from one patient is a positive 2005.13249 §4.1. Three transformations are compared: adjacent non-overlapping segments (CMSC), different leads (CMLC), and both (CMSMLC) 2005.13249 §4.2. The loss is an InfoNCE-style cosine objective at temperature 0.1, with off-diagonal same-patient pairs as extra positives 2005.13249 eq. 1–4 2005.13249 §5.5. The authors report, on 12-lead ECG at 50% labels and E=128, a linear AUC of 0.896 on Chapman for CMSC against 0.738 for SimCLR 2005.13249 tab. 1. On PhysioNet 2020, CMLC falls below SimCLR (0.596 against 0.615) 2005.13249 tab. 1. The assumptions behind the method, that nearby segments share a class and that simultaneous leads share cardiac function, can fail 2005.13249 §4.1 §4.2.

Supervised plus unsupervised instance contrast (open-world discovery)

For an unlabelled set that mixes seen and unseen classes, a DINO-pretrained ViT-B/16 is fine-tuned with two terms: an unsupervised InfoNCE over all images and a supervised contrastive loss over the labelled ones, mixed with λ = 0.35 2201.02609 eq. 1–3 2201.02609 §4.1. Labels enter only through the contrast; no classification head is trained 2201.02609 §3.1.1. In the ablation on CIFAR100 (All accuracy), adding the unsupervised term gives +2.6 points (52.0 to 54.6), and adding the supervised term on top gives +16.5 (54.6 to 71.1). Semi-supervised k-means then gives 73.0 2201.02609 tab. 5. The supervised term therefore accounts for most of the gain in the authors' ablation. The paper's own splits and clustering protocol are not the same as the NCD literature (see Contradictions).

Pixel-level contrast for dense prediction (U2PL)

For semi-supervised segmentation, the contrastive term works on pixels. For each class, the anchors are confident labelled or pseudo-labelled pixels with teacher probability above 0.3, and the positive is the mean teacher feature of those anchors 2203.03884 §3.3. Negatives are unreliable pixels whose class rank falls in [3, 20), held in a FIFO queue per class 2203.03884 §3.3. The loss is InfoNCE with cosine similarity and τ = 0.5 2203.03884 §3.3. On PASCAL VOC blender split at 1/4 labels, the full loss reaches 79.30 mIoU, against 73.02 without the contrastive term 2203.03884 tab. 6.

A label-free alternative for dense features is STEGO. It freezes a self-supervised DINO ViT-B (patch size 8) and trains a small head that maps each pixel to a 70-dimensional code 2203.08414 §3.1–§3.3. It does not use a softmax over negatives. Instead, the code correlation between two pixels is clamped at zero and multiplied by the centred feature correlation minus a shift b, and the loss is the negative of that sum 2203.08414 eq. 4. Pairs come from the image itself, its top-7 nearest neighbour and a shuffled batch partner 2203.08414 §3.2 2203.08414 eq. 5. The author-reported unsupervised mIoU on CocoStuff-27 is 28.2, against 14.4 for PiCIE + H 2203.08414 §4.2. Removing the clamp drops ViT-S unsupervised mIoU from 24.5 to 10.3 2203.08414 app. Additional Ablation Study. The shift values are dataset-specific and were set by a heuristic the authors describe as open 2203.08414 §3.3 2203.08414 app. A Heuristic for Setting Hyper-parameters.

Pixel-to-prototype contrast with learned sub-cluster prototypes (ProtoSeg-style)

Rethinking Semantic Segmentation replaces the class weights of a segmentation head with non-learnable prototypes. Each class has K prototypes, each the centre of one sub-cluster of that class's training-pixel embeddings, found by online Sinkhorn clustering and updated by momentum μ = 0.999 2203.15102 §3 §5.1 eq. 14. Each pixel's hard prototype is its assigned one. The positive is that prototype, and the negatives are the other C·K − 1 prototypes, under a softmax with temperature τ = 0.1 2203.15102 §3 eq. 11. The contrast term is added to cross-entropy and to a pixel-to-prototype distance term, each extra term weighted 0.01 2203.15102 eq. 13 §5.1. On ADE20K val with MiT-B2, the contrast term alone gives 45.9 mIoU against 45.0 for cross-entropy only 2203.15102 tab. 5. Cosine distance beats Euclidean in the authors' ablation, 46.4 against 45.7 2203.15102 tab. 5. The parameter count of the prototypes does not depend on the number of classes 2203.15102 §5.3 tab. 4.

Pairwise sigmoid (SigLIP)

For image-text pairs, the batch softmax is replaced by an independent binary sigmoid on each of the |B|² pairs, with logit t\,x_i\cdot y_j + b, learnable t = \exp(t') and bias b 2303.15343 §3.2 2303.15343 alg. 1. Positives lie only on the diagonal, and every cross-device block is all negatives 2303.15343 §3.3. The loss is a sum, so it can be computed device by device with text embeddings passed between devices; per-device memory drops from |B|² to b² 2303.15343 §3.3. The authors report that the sigmoid loss beats the softmax below about 16k batch and that both saturate near 32k 2303.15343 §4.1–4.2 2303.15343 tab. 8. At 3B examples and batch 2k, the sigmoid scores 81.3 ImageNet zero-shot against 80.1 for softmax 2303.15343 tab. 8.

Graph contrast

Two augmented views of a node, subgraph or graph form the positive pair. Augmentations are attribute-level (feature masking, shuffling) or topology-level (edge modification, graph diffusion, subgraph sampling) 2103.00111 §6.1. Cross-scale contrast, such as node against a graph summary, tends to beat same-scale contrast in the survey's compiled tables 2103.00111 tab. 7 2103.00111 app. D. Contrast-based methods are reported to beat generation- and auxiliary-based methods on node classification 2103.00111 tab. 5. The same survey notes that the numbers come from different papers and protocols (see Contradictions).

Centrality-adaptive edge and feature dropping (GCA-style)

GCA builds two graph views by dropping edges and masking feature dimensions. Each probability is set from a node-centrality score (degree, eigenvector or PageRank), so that structure with high centrality is kept more often 2010.14945 §3.3.1 §3.3.2 eq. 4 eq. 9. Each view draws its own strengths. The loss is an InfoNCE-style cosine objective: a node in one view is pulled to itself in the other view, and pushed from every other node in both views 2010.14945 §3.2 eq. 1–2. Author-reported node classification accuracy on Wiki-CS is 78.30% for GCA-DE, against 77.19% for a supervised GCN 2010.14945 tab. 3. On Amazon-Photo, the supervised GAT (92.56) is above all three GCA variants 2010.14945 tab. 3. The adaptive parts help by different amounts: the full model is 0.1 above uniform dropping on Wiki-CS and 1.6 above it on Amazon-Computers 2010.14945 tab. 4. Because the loss compares every node with every other, memory grows with the square of the node count 2010.14945 eq. 1.

Node-versus-subgraph contrast for anomaly detection (CoLA)

CoLA scores each node against a subgraph sampled around it. The positive pair is the node and a random-walk-with-restart subgraph seeded at that node, with the seed's attributes zeroed. The negative pair uses the same node with a subgraph seeded at another node 2103.00113 §4.1. A one-layer GCN with mean readout and a bilinear discriminator scores each pair, trained with binary cross-entropy 2103.00113 §4.2 eq. 10. A node's anomaly score is the mean, over R sampled rounds, of its negative score minus its positive score 2103.00113 §4.3 eq. 12; the authors use R = 256 2103.00113 §5.2.3. Author-reported ROC-AUC is the best of the compared methods on all seven benchmarks, reaching 0.9512 on Pubmed 2103.00113 tab. 3. The anomalies are injected, so these numbers measure recovery of injected cliques and copied attributes, not real anomaly rates 2103.00113 §5.1. Each score needs R sampled passes per node, which is the main inference cost on large graphs 2103.00113 §4.5.

Learned edge-dropping augmenter, graph level (AD-GCL)

AD-GCL contrasts a whole graph with a copy in which an adversarially trained network drops edges. Positives are the graph and its learned augmented copy, and negatives are the other graphs in the minibatch 2106.05819 §3. The loss is cosine InfoNCE 2106.05819 eq. 9 with no temperature in the paper. A penalty λ on the mean drop probability keeps the augmenter from deleting nearly every edge 2106.05819 eq. 8. The projection head has two layers, with widths not stated, and there is no momentum encoder or queue. Author-reported, the RDT-M5K score is 54.93 for AD-GCL-OPT against 53.05 for GraphCL 2106.05819 tab. 1. The mean rank is 2.4 on transfer 2106.05819 §5.3 and 1.5 on semi-supervised tasks 2106.05819 §5.4. Pre-training uses unlabelled graphs with GIN, batch 32 and Adam at 0.001 2106.05819 app. G. The released code uses GINE, temperature 0.2 and a symmetrised loss, and ZINC defaults λ to 0 2106.05819 Paper vs code. The paper's seed counts disagree between sections 2106.05819 Pitfalls, and the transfer and semi-supervised baselines are copied from other papers 2106.05819 §5.3.

Contrastive critic for goal-conditioned RL (NCE)

Here the contrast is over state-action pairs and future states. The positive pair is a state-action pair with a future state drawn from the occupancy measure, and the negative pair uses a future state drawn from the marginal 2206.07568 eq. 6 2206.07568 §4.2. The critic is an inner product of two learned encoders 2206.07568 §4.4. Contrastive RL uses a binary cross-entropy over the full batch matrix, with batch size 256 and no target networks or auxiliary losses 2206.07568 alg. 1 2206.07568 tab. 2 2206.07568 §4.5. On image tasks built on TD3+HER, the author-reported result is that it beats DrQ, an autoencoder auxiliary loss and CURL on all tasks 2206.07568 §5.2 2206.07568 fig. 4. On offline AntMaze, the variant with a behavioural-cloning term wins on 5 of 6 tasks 2206.07568 tab. 1. The paper's proof draws negatives from the marginal, but the released code takes them from the other batch rows 2206.07568 Paper vs code.

Video-text pairs as a reward (MineCLIP)

MineCLIP pre-trains a video and text pair of encoders on 640K clips with time-aligned transcripts, with an InfoNCE loss that pulls each clip towards its own transcript 2206.08853 §4.1. The paper does not state the exact form of the loss or its temperature 2206.08853 §4.1. The resulting similarity is used as a dense reward for a PPO agent 2206.08853 §4.2. The author-reported results are competitive with hand-made rewards on 8 programmatic tasks and well above a CLIP-guided baseline on 4 creative tasks, where no programmatic success test exists 2206.08853 §5 2206.08853 tab. 1.

Pair-labelled cosine loss for few-shot classification (SetFit)

Positive pairs join two examples of one class, and negative pairs join examples of different classes. The body of a sentence encoder is fine-tuned so that the cosine similarity of each pair matches its 1 or 0 target, and a logistic-regression head is then fit on the embeddings 2209.11055 §3.1. The number of pairs per class is a hyperparameter R, set to 20 unless stated otherwise 2209.11055 §3.1. No prompts or verbalizers are used 2209.11055 §1. With 8 examples per class, the author-reported average over five datasets is 62.3 ± 4.9 for the 110M-parameter model, against 63.4 ± 1.9 for T-Few 3B, which uses prompts 2209.11055 tab. 2. Standard deviations are large at this size, so the comparison should use them 2209.11055 Pitfalls.

Distribution-augmented instance contrast for one-class detection

For one-class classification, where only inlier images are seen in training, a contrastive encoder is trained on inliers and its features are scored by a kernel density estimator or a one-class SVM. The authors argue that plain instance contrast spreads features almost uniformly over the hypersphere, which leaves outliers little to stand out against. They therefore treat the 90, 180 and 270 degree rotations of each training image as separate distributions, which enlarges the training set four times 2011.02578 §3.1.3. The loss stays an InfoNCE over two views with in-batch negatives, with batch size 32 and temperature 0.2 2011.02578 app. A.4. Author-reported, the mean AUC over five settings is 89.9 with an OC-SVM, against 86.8 for plain contrastive features with KDE 2011.02578 tab. 2. The paper's rotation-prediction representation reaches 88.2 with KDE on the same table, so the margin over that proxy task is smaller than the margin over plain contrast 2011.02578 tab. 2. Hyperparameters were tuned on CIFAR-10 with 500 inlier and 500 outlier images, which the authors acknowledge may break the one-class assumption 2011.02578 app. A.4.

Fused text-and-graph encoder with in-batch link contrast

GraphFormers nests a graph-attention step inside each layer of a pre-trained language model, so every node's text is encoded with its neighbours' context 2105.02605 §3.1. Training is link prediction with in-batch negatives: the positive partner of one pair is a negative for every other pair in the batch 2105.02605 §3.3 eq. 6. A warm-up on span-masked inputs comes before training on clean text 2105.02605 §3.3. The reported batch sizes are 240, 240 and 160 2105.02605 tab. 6, so the negatives depend on which pairs share a batch. On DBLP the author-reported P@1 is 0.7267, against 0.6934 for the best cascaded baseline, PLM+Max 2105.02605 tab. 2. The paper reports about 3.5% extra time per batch at 200 neighbours 2105.02605 §4.5.

Video-text contrast with entity pseudo-labels

ALPRO pre-trains a video encoder and a text encoder with a symmetric video-text contrastive loss, which averages the two directions 2112.09583 §3.2.1. The batch is 256 and the temperature is learnable, with its initial value not stated 2112.09583 §3.4 §3.2.1. A separate term predicts soft entity labels for random crops, and those labels come from a frozen prompter that was pre-trained with the same contrast 2112.09583 §3.2.2. Author-reported ablation: adding the contrastive term to masked language modelling and video-text matching lifts MSRVTT R@1 from 28.5 to 32.8, and the full model reaches 33.9 2112.09583 tab. 1.

Symmetric contrast over unlabelled images for a pseudo-word token

Pic2Word trains only a three-layer mapping network. It takes the frozen CLIP image embedding of a picture and produces one token that is inserted into a text prompt. The loss is the sum of a text-to-image and an image-to-text contrastive term 2302.03084 §3.2 eq. 4. The model is trained on unlabelled images only, with batch 1024 2302.03084 §3.2 §4.1. The temperature is not stated 2302.03084 §3.1. Author-reported zero-shot CIRR test R@10 is 65.3, against 49.1 for image plus text features without training 2302.03084 tab. 3. The prompt template matters, and the authors used simple templates without tuning them 2302.03084 §3.3.

Tri-modal contrast with frozen encoders and generated captions

ULIP-2 trains a 3D point-cloud encoder against two frozen OpenCLIP ViT-G/14 encoders, one for images and one for text. It uses two symmetric contrastive terms, point-to-image and point-to-text, with a learnable temperature 2305.08275 §3.3 eq. 1-3. The text side is generated: BLIP-2 writes ten descriptions per rendered view, and CLIP ranking keeps the best one 2305.08275 §3.2 §4.1. On ModelNet40, top-1 holistic captions reach 69.7, against 60.4 for manual captions 2305.08275 tab. 5. Author-reported zero-shot top-1 on Objaverse-LVIS is 50.6 2305.08275 tab. 1. The paper gives no optimiser, learning rate or batch size for pre-training 2305.08275 §3.3.

Contrastive fine-tuning of a decoder-only retriever

Llama2Vec first adapts LLaMA-2-7B without labels, then fine-tunes it on query-answer pairs with a contrastive loss. Each query is scored against its positive and against the other passages in the batch and against hard negatives 2312.15503 §3.2 eq. 5. The temperature is not stated in the paper 2312.15503 §3.2. Fine-tuning uses LoRA 2312.15503 §4.1. Author-reported, MS MARCO dev MRR@10 is 43.1, against 41.2 for RepLLaMA, the strongest baseline 2312.15503 tab. 1 §4.2.

Reasoning trace before embedding (Think-Then-Embed)

Think-Then-Embed runs a reasoner that writes a trace for a query, then an embedder encodes the input together with that trace. The embedder is trained with one-directional query-to-target InfoNCE, cosine similarity, temperature 0.02 and a global batch of 8192, with gradient caching 2510.05014 §5.1 eq. 1. Negatives are the other targets in the batch 2510.05014 §5.1. A small reasoner fine-tuned on the task adds 5.0 points over zero-shot at 7B 2510.05014 tab. 3. Author-reported MMEB-V2 overall is 71.5 for the 7B model 2510.05014 tab. 2. The reasoner runs at query time, and the card records no latency measurement 2510.05014 Pitfalls. Pooling position and optimiser are not stated 2510.05014 §5.1.

Trend and seasonal contrast for forecasting (CoST)

CoST pre-trains a dilated convolution encoder with two contrastive branches, then fits a ridge regressor on the frozen features 2202.01575 §3 §4.1. The trend branch is MoCo-style: one random time step per iteration, a momentum key encoder, a queue of 256 negatives, temperature 0.07 and momentum 0.999 2202.01575 §3.1 §4.1. The seasonal branch contrasts the amplitude and phase of Fourier components across augmented views, with the other batch items as negatives and no temperature 2202.01575 §3.2. The total loss is the time-domain term plus α/2 times the amplitude and phase terms, with α = 5e-4 2202.01575 §3 §4.3. Author-reported, the multivariate average MSE is 0.590 against 0.750 for TS2Vec and 0.972 for TCN 2202.01575 §4.2 tab. 1. On univariate series the gap is smaller, 0.193 against 0.203 2202.01575 tab. 7. Several details differ between paper and released code, including the projection head, the number of trend experts, batch size and augmentations 2202.01575 Paper vs code. Two baselines, Informer and LogTrans, are copied from other papers 2202.01575 app. E.

Contrastive prior estimator for joint audio-video generation

JavisDiT uses contrast to train a module that turns a prompt into timing and place hints for a video and audio generator, not to train the encoders themselves. The anchor is the prompt and its estimated priors, the positive is the real, synchronised video-audio pair, and the negatives are asynchronous pairs. Easy negatives use audio generated from the caption alone; hard negatives corrupt either modality 2503.23377 app. C.2.1 app. C.2.4. The total objective sums two contrastive terms and two KL terms 2503.23377 eq. A1. The author-reported JavisScore is 0.154, against 0.151 for the cascaded baseline FoleyCrafter. The main table has single values and no error bars 2503.23377 tab. 1. The three training stages sum to 328 H100 GPU-days 2503.23377 tab. A1, and the released code's loss differs from the paper's in several terms 2503.23377 app. C.2.3.

Cross-modal EEG alignment (survey)

In EEG-to-image, EEG-to-text and EEG-to-speech work, a common step is to align the EEG embedding with the target space, often with a contrastive loss, before a generator decodes it 2502.12048 §4.6 §7. The survey gives no hyperparameters for this step 2502.12048 §7.

Siamese margin pair loss on discriminator features (SD-GAN)

Not a softmax over negatives. Semantics Disentangling for Text-to-Image Generation feeds two captions of one image through a text-to-image GAN as a Siamese pair. A pairwise loss acts on the discriminator features of the two outputs, at each of three stages 1904.01480 §3.1. Same-image pairs are pulled together until a floor α = 0.1, and different-image pairs are pushed apart to a margin ε = 1.0 1904.01480 eq. 2. The loss weight is not stated 1904.01480 §4.1. In the authors' ablation on AttnGAN, the Siamese term alone lifts CUB Inception score from 4.36 to 4.47 and MS-COCO from 25.89 to 29.77 1904.01480 tab. 3. The purpose is consistency across paraphrased captions, not representation learning for downstream tasks 1904.01480 §1.

Dense per-pixel InfoNCE with a momentum queue (VADeR)

Unsupervised Learning of Dense Visual Representations applies InfoNCE per pixel instead of per image. Two augmented views of one image are matched pixel to pixel through the known crop and resize geometry, and each matched pair is a positive 2011.05499 §3.1 §3.2. Negatives are random pixels from other images, held in a MoCo queue of 65,536 entries with momentum 0.999, and the temperature is 0.07 2011.05499 §3.2. Views must share at least 32 pixels, and 32 pairs are drawn per image 2011.05499 §3.2. Author-reported, frozen features reach 56.7 VOC mIoU, against 43.0 for MoCo and 54.4 for ImageNet-supervised ResNet-50 2011.05499 tab. 1. On COCO Mask R-CNN the box AP is 39.2, against 38.9 for MoCo 2011.05499 tab. 4, so the detection gain is small. The encoder starts from MoCo v2, and the baselines are adapted to the same output resolution, so the comparison depends on that setup 2011.05499 §3.2 §4.1. Same-image negatives did not work in the authors' preliminary runs 2011.05499 §3.2.

Symmetric in-batch InfoNCE with hard-negative sampling (cross-view matching)

Sample4Geo matches street or drone images to satellite tiles with one shared encoder. Each street image is a query over the satellite images in its batch, and each satellite image is a query over the street images. The two cross-entropies use label smoothing 0.1, and the temperature is learnable with its start value not stated 2303.11851 §3.1 §4.2. Hard negatives come from GPS-near locations in early epochs, then from Dynamic Similarity Sampling: every four epochs the top-K cosine neighbours are recomputed, and half of k = 64 are the nearest while the other half are drawn at random from the rest 2303.11851 §3.3 §3.4. On VIGOR SAME, GPS plus DSS gives R@1 77.86, against 65.23 with random batches 2303.11851 tab. 5. The authors report that a triplet loss with hard negatives collapses to R@1 0.00 on CVUSA, while InfoNCE reaches 98.68 2303.11851 Loss comparison. Those are the authors' runs; the triplet setup is not stated in enough detail to reproduce.

False-negative filter with top-k hard negatives (UniME)

Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs turns a multimodal LLM into an image-text embedder in two stages. Stage 1 is text only: the student's batch similarity distribution is matched to a teacher's with a KL term, training LoRA on the LLM 2504.17432 §3.2 eq. 2. Stage 2 is InfoNCE on image-text pairs with a per-query filter. Any in-batch candidate whose cosine similarity exceeds the positive's similarity plus β = 0.1 is removed as a likely false negative. The k = 8 best remaining candidates become the hard negatives 2504.17432 §3.3 eq. 3 eq. 4 §4.1. On Phi-3.5-V, stage 1 alone raises the MMEB average from 25.3 to 40.3, and stage 2 alone reaches 63.8, against 64.2 for both 2504.17432 tab. 3. The β and k values come from ablations on that one backbone 2504.17432 tab. 4 tab. 5. The paper gives no temperature for either stage 2504.17432 §3.3; see Implementation sketch for the code's default.

Image-text contrast with a momentum teacher and queue (ALBEF-style)

ALBEF aligns an image encoder and a text encoder with a symmetric image-text loss before cross-modal fusion. Each image's scores are computed against the batch plus a queue of 65,536 recent momentum-encoder features, with an EMA momentum of 0.995 2107.07651 §3.2 §3.5. The temperature is learnable, and its initial value is not stated 2107.07651 §3.2. Momentum distillation replaces part of the one-hot target with the teacher's similarity distribution, weighted by α 2107.07651 §3.3 eq. 6. Author-reported, on 4M images, adding the contrastive term to masked language modelling and matching raises Flickr30K text-retrieval average recall from 93.96 to 96.55. Adding hard negatives and momentum distillation on that term reaches 97.33 2107.07651 tab. 1.

BLIP keeps the momentum-encoder image-text contrast with soft targets and adds a matching head whose hard negatives are drawn by similarity 2201.12086 §3.2. The paper does not state the momentum value or the temperature 2201.12086 §3.2. Author-reported, COCO 5K fine-tuned text-to-image R@1 rises from 77.6 for ALBEF to 80.6 for BLIP at 14M images 2201.12086 tab. 5.

Image-text contrast on frozen towers via a query transformer (BLIP-2-style)

BLIP-2 trains a small query transformer between a frozen image encoder and a frozen language model. Stage 1 has three losses, one of them image-text contrast. Query outputs are compared with the text feature, under an attention mask that stops queries and text from seeing each other, with in-batch negatives 2301.12597 §3.2. The hard negatives for the matching loss come from the same similarity scores 2301.12597 §3.2. Stage 1 batch sizes are 2320 with ViT-L and 1680 with ViT-g 2301.12597 §3.4. The projection width and the temperature are not stated 2301.12597 §3.2. The paper's retrieval ablation varies the generation loss, not the contrastive term alone 2301.12597 tab. 6.

Masked contrastive prediction over quantised targets (wav2vec 2.0, XLS-R)

XLS-R masks spans of 10 time steps. For each masked step, the model must pick the true quantised latent out of K = 100 distractors drawn from other masked steps, scored by cosine similarity, and a codebook diversity penalty is added 2111.09296 §2. The weight of the penalty is not stated 2111.09296 §2. The target is the quantised latent of the masked step itself, so this is prediction rather than matching two augmented views. Batches mix languages, sampled in proportion to unlabelled hours raised to α = 0.5 2111.09296 §2 §4.1. A review of self-supervised speech representation learning files CPC and wav2vec 2.0 under contrastive methods, and writes the wav2vec 2.0 objective as InfoNCE with a codebook term 2205.10643 §3.3. The review reports no new method, and its numbers are copied from the original papers and are author-reported 2205.10643 §3 tab. 1.

Contrast between two views of one time series (TS-TCC)

TS-TCC builds two augmented views of a raw series and adds two terms. Temporal contrasting asks the context of one view to identify the future latent steps of the other 2106.14112 §3.2. Contextual contrasting is a SimCLR-style cosine InfoNCE over 2N contexts, with temperature τ 2106.14112 §3.3 eq. 5. The paper uses batch 128 and τ = 0.2 2106.14112 §4.2. Author-reported linear-probe accuracy on HAR is 90.37, against 80.97 for SimCLR under the same protocol 2106.14112 tab. 2.

Retriever for in-context examples with scorer-labelled hard negatives (EPR-style)

Learning To Retrieve Prompts for In-Context Learning trains a dense retriever in the DPR style. Each training example is labelled by a scoring language model: candidates are ranked by how likely the model gives the gold output with that candidate in the prompt. The top 5 are positives and the bottom 5 are hard negatives 2112.08633 §3.1 §4.2. Each step uses in-batch negatives plus one hard negative per instance 2112.08633 §3.2 eq. 1. The paper's batch size is 120 and its learning rate is 1e-4 2112.08633 app. A. Author-reported, MTOP exact match is 64.2 against 57.0 for the best baseline 2112.08633 tab. 2.

Pairwise softmax over two images (PickScore-style)

Pick-a-Pic trains a scorer on pairs of images for one prompt, not on in-batch negatives. Each image gets an inner product with the text, scaled by a learned temperature, and a two-way softmax over the pair gives the probability that the preferred image wins 2305.01569 §3 Model §3 Objective. Ties enter as a soft target 2305.01569 §3 Objective. The main model has in-batch negatives off. The appendix variant that adds them reaches 65.2 test accuracy, against 70.5 for the main model 2305.01569 app. Training a Scoring Function tab. 1.

Web-scale pairs filtered by CLIP, then InfoNCE reproduction (LAION-style)

LAION-5B keeps alt-text image-text pairs only when OpenAI's CLIP ViT-B/32 scores them above 0.28 in English or 0.26 in other languages 2210.08402 §3.1. Contrastive training appears only in the authors' reproduction runs, which use InfoNCE with the global batch gathered across workers. The authors report this lowers the memory of the loss matrix from order N² to order nN 2210.08402 app. E.1 app. E.2.

Large global batch and scaling study (OpenCLIP-style)

Reproducible scaling laws trains OpenCLIP models with InfoNCE, following the original CLIP recipe. Negatives span a global batch of 86 to 88 thousand 2212.07143 §3.2. The direction of the loss and the temperature are not stated 2212.07143 §3.2. Embeddings gathered across workers must pass gradients 2212.07143 app. "Sharding contrastive loss". Author-reported, ViT-H/14 on LAION-2B reaches 78.0% zero-shot ImageNet top-1, and error follows power-law fits over model and data size 2212.07143 §4.

Continual CLIP pretraining with captions written from boxes and masks (RemoteCLIP)

RemoteCLIP continues CLIP pretraining on aerial and satellite image-text pairs. Captions are generated from existing labels: boxes become five captions per image (B2C), and each class mask is first turned into boxes (M2B) 2306.11029 §3.2.1 §3.2.2. The training set has 165,745 images with five captions each, 828,725 pairs in all 2306.11029 §4.1.2. The loss is a symmetric InfoNCE with a learnable temperature, whose initial value is not stated 2306.11029 §3.1 eq. 1. The batch is 256 for ResNet-50 and ViT-B-32 but 28 for ViT-L-14, so the largest model has far fewer in-batch negatives, and the paper does not discuss this 2306.11029 §4.1.3 [Pitfalls]. In the authors' loss ablation, InfoNCE scores 36.32 retrieval average against 28.93 for margin ranking and 26.68 for a SigLIP-style loss; the zero-shot averages are 48.57, 48.47 and 45.66 2306.11029 tab. 5. The text does not say which dataset these rows use 2306.11029 §4.3. Author-reported, ViT-B-32 zero-shot average over 12 datasets is 62.41, against 56.02 for CLIP 2306.11029 tab. 3.

Fixed CLIP recipe for comparing image-text datasets (DataComp)

DataComp is a benchmark that fixes the contrastive recipe and the compute, and lets participants change only the image-text pairs. Training uses a symmetric contrastive loss with a learnable temperature, in the OpenCLIP code base 2304.14108 §3.4. The global batch is 4096 at the small, medium and large scales and 90,112 at xlarge, which uses ViT-L/14 2304.14108 tab. 14. The filtering baselines score pairs with CLIP similarity. At xlarge, keeping the top 30% by CLIP L/14 score, intersected with an image-cluster filter, reaches 0.792 ImageNet zero-shot, against 0.723 with no filtering 2304.14108 tab. 3. Random subsets of the same size do worse than CLIP-score filtering, and the CLIP-score filter peaks near the top 30% 2304.14108 §5.1 fig. 2. The paper reports single-seed runs with no error bars 2304.14108 app. Checklist, so gaps of about a point are not established.

Table rows with CutMix and mixup views (SAINT-style)

SAINT is a transformer for tabular rows. Each column has its own embedding, the columns of one row attend to each other, and the rows of a batch attend to each other 2106.01342 §3 §3.2. Pre-training builds two views of each row, one with CutMix in raw input space and one with mixup in embedding space. An InfoNCE loss over the batch, at temperature 0.7, pulls the two views of one row together, and a per-column denoising loss is added 2106.01342 §4 §5 eq. 3–5. Author-reported, the mean AUROC over 14 binary datasets is 93.13 for SAINT, against 91.06 for XGBoost 2106.01342 tab. 2. With 50 labelled rows, pre-training adds about 1.1 points, and the gap closes with all labels 2106.01342 tab. 3. A row's output depends on the other rows in its batch, so the batch composition is part of the method 2106.01342 Pitfalls.

Projector-free InfoNCE on a fixed sub-vector (DirectCLR)

DirectCLR shows that contrastive learning also suffers dimensional collapse, where embeddings span fewer dimensions than available 2110.09348 §3. The authors trace it to strong augmentation in a linear model and to implicit low-rank bias when augmentation is weak 2110.09348 §4.2 §5.3. Their method drops the trainable projector and applies InfoNCE to the first d0 channels of the encoder output, after normalising them, with in-batch negatives 2110.09348 §6.2. Author-reported ImageNet linear-probe top-1 at 100 epochs is 62.7% for DirectCLR, against 61.1% for SimCLR with a one-layer linear projector and 66.5% for SimCLR with a two-layer projector 2110.09348 tab. 1. The collapse theory is proven only for linear networks 2110.09348 §4.2. Random channels in place of a fixed sub-vector reach 43.0%, so the fixed choice matters 2110.09348 app. F.

Image-text contrast plus an image-only SimCLR term (SLIP-style)

SLIP trains one Vision Transformer on two losses: CLIP's symmetric image-text InfoNCE, and a SimCLR loss on two augmented views of each image. The two are summed with weight 1.0 2112.12750 §3.3 alg. 1. The SimCLR term uses a three-layer projection head, with all other views in the batch as negatives 2112.12750 §3.2 §4.1. The paper says the weight can be rescaled and that other self-supervised objectives can be swapped in 2112.12750 §3.3 §5.5. It does not state the SimCLR temperature 2112.12750 §3.2. Author-reported, on YFCC15M with ViT-B/16 for 25 epochs, zero-shot ImageNet accuracy is 42.8% against 37.6% for the authors' CLIP run, and linear accuracy is 72.1% against 66.5% 2112.12750 tab. 2. The extra branch costs about three times the image activations per step 2112.12750 §3.3.

Video-text matching with cross-frame message tokens (X-CLIP-style)

X-CLIP adapts a pretrained CLIP image-text model to video. Each frame gets a message token that is exchanged across frames inside every transformer block, and a small transformer merges the frame features into one video vector 2208.02816 §3.2 §3.2.1. Class names are turned into video-specific text embeddings by a prompt generator that attends to each video's patch tokens, and the score is the cosine similarity between the video and each class text 2208.02816 §3.3 eq. 3. Author-reported Kinetics-400 top-1 is 87.1% with ViT-L/14 at 8 frames 2208.02816 tab. 1. The compiled card does not record the training loss, so this entry covers the matching score only. Check the paper's training section before treating it as InfoNCE. Because each video has its own prompts, class text embeddings cannot be cached across videos 2208.02816 Pitfalls.

Fusion-level prediction of each modality, plus input-level bounds (MMIM-style)

MMIM predicts a sentiment score from text, visual and acoustic inputs. Its contrastive term asks the fusion vector to predict each modality representation through a predictor, with unit-length vectors and the other batch items as negatives 2109.00412 §3.5 eq. 9–11. Its input-level term is a Barber–Agakov lower bound on the pairs (text, visual) and (text, acoustic), with a two-class Gaussian-mixture estimate of the entropy 2109.00412 §3.4 eq. 2 eq. 7. The training loss adds the contrastive term and the bound to the task loss, weighted by α and β 2109.00412 eq. 13. The paper trains the predictors in a separate first stage 2109.00412 §3.6 alg. 1. Author-reported weights are α = 0.3 and β = 0.1 for CMU-MOSI, and α = 0.1 and β = 0.05 for CMU-MOSEI 2109.00412 tab. 3. On CMU-MOSEI, removing all three contrastive terms lowers Acc-7 from 54.24 to 53.49 2109.00412 tab. 4. The reported MOSEI MAE of 0.526 is the lowest in the authors' table, while their non-negative binary accuracy is below the MAG-BERT reproduction 2109.00412 tab. 2.

Global image-text contrast with masked-modality losses in one model (FLAVA-style)

FLAVA has image, text and fusion encoders, trained on image-text pairs, unpaired images and unpaired text. On pairs, a CLIP-style global contrastive loss runs on the un-masked encoders, and its gradients flow through embeddings gathered from all devices, not only the local ones 2112.04482 §3.2 app. C. The same pairs also train masked multimodal modelling and an image-text matching head on the fused encoder 2112.04482 §3.2. Author-reported macro average over 35 tasks is 75.85 for the full model, against 66.78 for released CLIP ViT-B/16 trained on 400M pairs, and 61.52 for CLIP trained on the same 70M public pairs 2112.04482 tab. 3. Part of the gain over the CLIP-70M run comes from architecture changes, which the paper credits with +6.99 macro 2112.04482 app. D tab. D.1. CLIP is still ahead on vision: its released model reaches a vision average of 82.57 against 79.44 for FLAVA 2112.04482 tab. 3, and 80.2 against 75.5 on ImageNet linear evaluation 2112.04482 tab. 5. On COCO retrieval, CLIP leads text retrieval (R@1 52.48 against 42.74) and FLAVA leads image retrieval (38.38 against 33.07) 2112.04482 tab. 4. Batch size is 8192 in the pretraining table 2112.04482 tab. A.1.

Sparse mixture-of-experts one tower with routing losses (LIMoE-style)

LIMoE sends images and captions through one shared sparse transformer, and trains it with a symmetric softmax contrastive loss on pooled per-modality embeddings, averaged over both directions, with a learned temperature 2206.02770 §2.1 §2.2. Mixing modalities in the router can overflow expert capacity and collapse experts, so the authors add per-modality local and global entropy losses and batch priority routing 2206.02770 §2.2.1 §2.2.2 §2.2.3. Author-reported zero-shot ImageNet top-1 for the largest model, LIMoE-H/14, is 84.1, with 675M parameters used per token 2206.02770 §3.2. The authors' comparison table lists CoCa at 86.3 with 2.1B parameters per token 2206.02770 §3.2. Against dense one-tower baselines of equal compute, the authors report average zero-shot gains of 10.1 points across scales 2206.02770 §3.1.

Supervised contrast on region proposals for few-shot detection (FSCE)

FSCE adds a contrastive head on top of detector RoI features, so that proposals of the same class cluster together. The head maps 1024-dimensional RoI features to 128 dimensions and uses cosine similarity at temperature 0.2 2103.05950 §3.2 §4.3. Positives are RoIs with the same label in the mini-batch and negatives are all other RoIs. Anchors are RoIs whose IoU with a ground-truth box is at least φ, 0.7 in the text 2103.05950 §3.3. The term is added with weight λ = 0.5 2103.05950 eq. 5. There is no momentum encoder or queue. Author-reported nAP50 on PASCAL VOC split 3 at 10 shots is 58.5, against 50.2 for TFA with a fully connected head and 49.7 for MPSR 2103.05950 tab. 2. The contrastive term adds 0.9 to 2.1 points on split 1 across shot counts 2103.05950 tab. 5. On COCO at 30 shots, novel AP is 16.4 against 13.7 2103.05950 tab. 3. The released code normalises the loss by the number of RoIs that pass the IoU filter, not by all RoIs in the batch, and its configs use λ of 0.1 or 0.2 and φ between 0.5 and 0.8 2103.05950 Paper vs code. Results with fewer passing RoIs therefore get a larger effective loss.

Contrastive clustering of detector features with pseudo-labelled unknowns (ORE-style)

For object detection, ORE keeps a queue of recent RoI features for each known class and for an unknown class. Each feature is pulled towards its own class prototype and pushed from the others up to a margin 2103.02603 §4.1 eq. 1. Unknown examples are the highest-objectness background proposals that do not overlap a ground-truth box 2103.02603 §4.2. These can include real unlabelled objects, so the unknown class is noisy 2103.02603 §4.2. At test time a detection is called unknown when a Weibull fit to unknown energies is denser than the fit to known energies at that detection's energy 2103.02603 §4.3. On the authors' open-world protocol, task 2 Wilderness Impact is 0.0154 against 0.0371 for Faster R-CNN, and mAP over both known sets is 38.98 against 37.47 for fine-tuning 2103.02603 tab. 2. Plain fine-tuning is therefore a stronger reference than plain Faster R-CNN. The reference code's distance and margin differ from eq. 1 as written, so read the card before reimplementing.

Auxiliary contrast on a robot-centric attention pool (RcP-style)

Here the contrast is an auxiliary term on a visuomotor policy, not the main objective. RcP replaces the final pooling of a ResNet18 policy with a cross-attention whose query comes from joint and end-effector state, and a hard argmax mask picks one image location 2411.04331 §3.1 eq. 8. The contrastive term trains the attention so that context vectors for the same robot pose agree across augmentations and differ across poses 2411.04331 §3.2. Augmentation pastes the cropped arm onto random backgrounds and sometimes adds a copy of the robot as a distractor 2411.04331 §3.2. The loss is MoCo-style InfoNCE against a queue of 4096 projected negatives, with temperature 0.1 and a momentum encoder at m = 0.95 2411.04331 §4.1 eq. 9. The total loss is the L1 policy loss plus 0.05 times the contrastive and reconstruction terms 2411.04331 §3.3 eq. 12 §4.1. Author-reported simulated reaching success under all distractors is 76.4% for RcP, against 31.9% for ResNet18 with Spatial-Softmax and 8.3% for R3M 2411.04331 tab. 1 §4.2.1. The paper reports no ablation of the contrastive term alone, so the gain cannot be attributed to the contrast rather than to the pooling and the augmentation 2411.04331 Pitfalls. No official code was found 2411.04331 Paper vs code.

Symmetric audio-video contrast with frozen encoders and mapping heads (STEVE-Audio-style)

STEVE-Audio aligns audio with video in a shared 512-d space, trained as CLIP is. A frozen MineCLIP video encoder and a frozen Audio Spectrogram Transformer feed two trainable mapping networks, and matching audio-video pairs get high cosine similarity while non-matching pairs in the batch get low similarity 2412.00949 §3.1. Only the two mapping networks are trained, for 100 epochs with no learning-rate schedule 2412.00949 §3.3. The table lists batch size 1024 and learning rate 0.001 2412.00949 tab. 2. A separate conditional VAE then maps the audio embedding into the MineCLIP goal space of the existing STEVE-1 policy, which is not changed. That prior is not contrastive 2412.00949 §3.4. Author-reported, audio goals beat the visual goal on four of six item-collection tasks and the text goal on five of six, and the three place tasks are weaker 2412.00949 §4.1 §4.2.1. The paper reports ratios, not counts, and its conclusion is stronger than §4.1 2412.00949 §5.

Masked-span contrast over raw EEG sequences (BENDR-style)

BENDR adapts the wav2vec 2.0 design to multi-channel EEG. A convolutional encoder turns raw signals into vectors at about 2.67 Hz, and an 8-layer transformer reads them 2101.12037 §2.3. During pre-training, spans of those vectors are replaced by one learned mask vector. At each masked position the transformer output is scored by cosine similarity against the true vector and against 20 distractors from the same sequence, at temperature 0.1 2101.12037 §2.4.1 eq. 2. The loss adds a mean-squared activation term with weight 1 2101.12037 §2.4.1. The paper gives no unit for the span length; counted in encoder positions it is about 3.75 s 2101.12037 §2.3. The authors chose contrast over reconstruction because neighbouring EEG samples are highly correlated 2101.12037 §1. Author-reported, a linear head on pooled features gives the best mean in four of five downstream tasks, with 86.7 balanced accuracy on MMI 2101.12037 §3.2 tab. 2. The official code has no licence file and departs from the paper in several places 2101.12037 Paper vs code.

Label-free contrast for outlier scoring (SSD-style)

SSD trains a ResNet-50 on unlabelled in-distribution images with the NT-Xent loss. Two augmented views of one image are the positive pair, and the other views in the batch are negatives 2103.12051 §3 eq. 1. No further training follows. The method fits one Gaussian to the normalised training features and scores each test input by its Mahalanobis distance 2103.12051 §3.1. The recipe is 500 epochs at batch size 512 and temperature 0.5 2103.12051 §4.1. The temperature matters: on ResNet-18 with CIFAR-10 as inliers and CIFAR-100 as outliers, AUROC is 66.7 at tau = 0.001 and 89.5 at tau = 0.5 2103.12051 tab. 5. Author-reported, the mean AUROC over four CIFAR-10/100 pairs is 88.7, against 80.9 for Rotation-loss 2103.12051 tab. 1. With five OOD samples per class the mean rises to 92.5 2103.12051 tab. 1. The paper states no compute cost for these runs 2103.12051 Results.

Momentum-model keys with pseudo-label-aware negatives (AdaContrast-style)

AdaContrast adapts a source-trained model to unlabelled target data without source images at adaptation time 2204.10377 §3. A momentum copy of the target model supplies the key for one strong view, and the query comes from another strong view of the same image 2204.10377 §3.2 eq. 5. Negatives are queued keys whose pseudo label differs from the current one 2204.10377 §3.2 eq. 6. Pseudo labels are refined at every batch by averaging the predictions of the N nearest entries in a second queue 2204.10377 §3.1 eq. 3 eq. 4. The total loss sums cross-entropy, the contrastive term and a diversity term, each with weight 1 and no tuning 2204.10377 §3.3 eq. 10. No confidence threshold is applied, so wrong pseudo labels reach the loss, and the diversity term is the only guard against collapse 2204.10377 §3.3. Author-reported, the VisDA-C per-class average is 86.8, against 83.0 for SHOT 2204.10377 tab. 1. The official code has no licence file and omits or changes details the paper states 2204.10377 Paper vs code.

Contrast against frozen image-text towers for a new 3D modality (ULIP-style)

ULIP trains a 3D point-cloud encoder to share an embedding space with the frozen image and text encoders of SLIP. Training uses ShapeNet55 triplets of rendered image, category caption and point cloud 2212.05171 §3.1 §3.2. The 3D encoder gets a symmetric contrastive loss for each of the 3D-text and 3D-image pairs, with a learnable temperature 2212.05171 §3.2 eq. 4 eq. 5. The paper gives batch size 64 on eight GPUs 2212.05171 §4.3. The code gathers embeddings across GPUs before computing logits, so each step contrasts against 512 objects losses.py. Author-reported, zero-shot top-1 on ModelNet40 with all classes is 60.4 for PointBERT+ULIP 2212.05171 tab. 4. Standard classification gains are about 3 points on ScanObjectNN 2212.05171 tab. 1. The later ULIP-2 variant above builds on this framework.

Synthetic-data InfoNCE for an LLM text embedder (E5-Mistral-style)

Improving Text Embeddings with Large Language Models has a strong proprietary model write about 500k training examples in 93 languages, using a two-step prompting scheme 2401.00368 §3.1 §4.1. Mistral-7B is then fine-tuned with LoRA at rank 16 for one epoch, using InfoNCE at temperature 0.02 2401.00368 §3.2 §4.2 app. A. Each query gets a one-line task instruction, and documents do not 2401.00368 §3.2. Author-reported, the MTEB average is 66.6 with the full data mix and 63.1 with synthetic data only 2401.00368 tab. 1. The repository holds only evaluation code, so the training recipe cannot be checked against code 2401.00368 Pitfalls.

Strong-view distribution matching over a key queue (CLSA-style)

CLSA keeps a MoCo-style key queue but changes what a strongly augmented query has to reproduce. The weak and strong views of one image each give a similarity profile over the queued keys, and the strong view is trained to match the weak view's profile 2104.07713 §3.1 eq. 1. The loss adds a distillation term to the usual contrastive term on the weak query 2104.07713 §3.1. Strong augmentation draws from 14 operations, five rounds, each at probability 0.5 2104.07713 §3.3 tab. 1. Author-reported, the ResNet-50 linear top-1 at 200 epochs is 69.4 against 67.5 for MoCo v2 2104.07713 tab. 2. The released code sums the contrastive terms over views and differs in augmentation details 2104.07713 Paper vs code.

Cross-modal momentum InfoNCE on weakly paired web pairs (BriVL-style)

BriVL is a two-tower image-text model. Two momentum encoders maintain an image queue and a text queue of recent keys 2110.14378 §Methods: Contrastive Loss. Each image's paired text is its positive, the text queue supplies the negatives, and the reverse direction uses the image queue 2110.14378 eq. L_i2t eq. L_t2i. Pre-training uses 650M weakly correlated Chinese web pairs, with no object detector 2110.14378 §Results: Pre-Training Data Collection. Author-reported, UCM zero-shot accuracy at a 5/16 unseen/seen split is 82.41 against 76.02 for CLIP with ResNet-50x4 2110.14378 fig. 4a. In the authors' ablation, replacing the queue with SimCLR-style in-batch negatives lowers the recall sum from 268.66 to 259.45 2110.14378 fig. S2a. The paper does not say whether embeddings are normalised before the dot product 2110.14378 Pitfalls.

Within-sequence margin contrast on hidden states (SimCTG-style)

SimCTG adds a contrastive term to next-token training of GPT-2. The term runs over the last-layer hidden states of all tokens in one sequence: each pair of distinct positions gets a hinge on cosine similarity with a margin 2202.06417 §3.1 eq. 1 eq. 2. The total is the usual next-token cross-entropy plus this term 2202.06417 §3.1. The released training script uses margin 0.5 2202.06417 §6.2 train.sh. The same paper pairs the trained model with contrastive search at decoding time, which picks among the top-k tokens by model probability minus a similarity penalty 2202.06417 §3.2 eq. 5. The decoder can also be applied to models trained without the term 2202.06417 app. D. On WikiText-103, the 2-gram repeat rate is 3.93% for SimCTG with contrastive search, against 69.21% for MLE with greedy decoding 2202.06417 tab. 1.

Margin ranking over model-scored candidates (BRIO-style)

BRIO starts from a pre-trained summariser and adds a pairwise margin loss over candidate summaries. The candidates are diverse-beam outputs, ordered by their ROUGE against the reference 2203.16804 §3. The margin between ranks i and j grows with the rank gap 2203.16804 eq. 8. The total loss weights this term against the reference cross-entropy 2203.16804 eq. 10. Author-reported, CNN/DailyMail ROUGE-1 is 47.78 for the generator BRIO-Mul, against 44.29 for BART 2203.16804 tab. 2. The contrastive-only variant BRIO-Ctr can rerank candidates but cannot generate 2203.16804 §3. The released XSum setting gives the ranking term a much smaller effective weight than the paper's value 2203.16804 Paper vs code.

Triplet boundary contrast in an anchor-free action localizer (AFSD-style)

AFSD adds boundary contrastive learning on top of its detector. An action is cut into two fragments with a random background clip between them. The end feature of the first fragment should lie near the start feature of the second, and far from a background boundary feature 2103.13137 §3.3. The triplet loss uses squared distances and a margin of 1 2103.13137 eq. 9. Only videos with a long enough action and a background clip qualify, and their consistency pass runs after the main loss on each batch 2103.13137 §3.3 §3.4. Author-reported, the THUMOS14 average mAP is 52.0 2103.13137 tab. 1. Several settings differ between the paper and the code 2103.13137 Paper vs code.

Prompt-versus-augmented-prompt InfoNCE on frozen text features (CLAP-style)

CLAP trains a small residual MLP on frozen CLIP text features, and uses text only 2311.16445 §3. Its loss has two InfoNCE terms: the prompt against its style-augmented copy, and the class name against that copy 2311.16445 eq. 4. The MLP is then applied to CLIP image and text features 2311.16445 §3. Author-reported, the ViT-B/16 zero-shot average over PACS, VLCS, OfficeHome and DomainNet rises from 77.4 to 79.9 2311.16445 tab. 3. The paper's InfoNCE is one-directional over cross-view pairs, while the code's version is two-directional and uses same-view negatives 2311.16445 Paper vs code.

Cards tagged with this slug that use no contrastive loss

Two cards carry this slug, but their methods do not use a contrastive loss. TSDAE trains a sentence encoder to rebuild a sentence from a single vector, with no negatives in its objective 2104.06979 §3.1 §3.2. VASA-1 adds a cross-identity term that maximises the cosine similarity between ArcFace features 2404.10667 §3.1. They are listed so that the page covers every card tagged with this slug. Their results should not be read as contrastive ones.

When to use which

Implementation sketch

The core is the same for every variant: score each query against its positive and the negatives, then take cross-entropy over those scores. The variants change where keys and queue come from.

# InfoNCE over one positive and K negatives per query  [arxiv:2010.05113 eq. 1]
# Input: queries q[B, d], keys k[B, d] (k[i] is the positive of q[i]),
#        temperature tau, optional queue of extra negatives  [arxiv:2006.08218 eq. 12]
# Output: scalar loss

def info_nce(q, k, tau, queue=None):
    # Normalise, so the score is cosine similarity  [arxiv:2010.05113 §III.E.1]
    q = l2_normalize(q)
    k = l2_normalize(k)
    # Score each query against all keys; the diagonal holds the positives
    keys = k if queue is None else concat(k, queue)
    logits = q @ keys.T / tau                       # [B, B + Q]
    # Positive of row i is column i; the rest are negatives  [arxiv:2010.05113 §II.C]
    labels = arange(B)
    return cross_entropy(logits, labels)

Notes on the sketch:

• Keys from a momentum encoder get no gradient. Update them as an EMA of the query encoder, and push them into the queue 2010.05113 §III.C.2. The momentum value is not stated in either survey, so take it from the primary method.
• Temperature. Neither survey gives a value for \tau in its SimCLR or MoCo discussion 2006.08218 Hyperparameters 2010.05113 Hyperparameters. The U2PL pixel loss uses 0.5 2203.03884 §3.3. The E5 recipe uses 0.01 2212.03533 §4.1 and the CLOCS recipe uses 0.1 2005.13249 §5.5. The VADeR pixel loss uses 0.07 2011.05499 §3.2. The UniME paper gives none, while its released stage-2 loss defaults to 0.02 2504.17432 Paper vs code. The TS-TCC time-series recipe uses 0.2 2106.14112 §4.2, and the RcP auxiliary loss uses 0.1 2411.04331 §4.1. ALBEF and BLIP-2 leave the initial value of a learnable temperature unstated 2107.07651 §3.2 2301.12597 §3.2, and so do RemoteCLIP and DataComp 2306.11029 §3.1 2304.14108 Pitfalls. Use the primary paper's value.
• DataComp's printed loss. The footnote in its §3.4 is a ratio of exponentiated scores, with no logarithm or minus sign. Implement the standard form above and cite the paper for the symmetric design 2304.14108 Pitfalls 2304.14108 §3.4.
• Group positives. Replace the diagonal target with a mask over pairs that share a group id, and average the log-probabilities over the masked pairs, as CLOCS does 2005.13249 eq. 3. Decide what to do when a batch has no such pair; the paper does not say 2005.13249 Pitfalls.
• Pairwise sigmoid replaces the softmax lines with logits t * q @ k.T + b, labels 2 * eye(B) - 1, and loss -sum(log_sigmoid(labels * logits)) / B 2303.15343 alg. 1. Initialise t' = \log 10 and b = -10 2303.15343 §3.2 2303.15343 §4.9.
• Never divide by |B|² in the sigmoid loss. The paper divides by |B| 2303.15343 §3.2.
• Negative-free methods drop the queue and the denominator and regress a predictor's output onto an EMA target's output with a normalised squared error 2006.08218 §4.2.2 eq. 15. DirectPred replaces the trained predictor with W_p = \hat U\,\mathrm{diag}(\sqrt{s_j} + \epsilon\max_j s_j)\,\hat U^\top from a running feature correlation 2102.06810 eq. 18–19. Its reported runs use EMA 2102.06810 tab. 6.
• Prototype-based supervised losses use class means as the only keys. FedProc's eq. 4 has no temperature term, so its scores are raw cosine similarities; add a scale only as a deliberate change 2109.12273 eq. 4 2109.12273 Hyperparameters.
• One-class denominator. Eq. 2 of the one-class paper includes same-view negatives in the softmax. The released code, as far as the card records it, keeps only cross-view terms, and the paper does not explain the difference 2011.02578 eq. 2. Decide which form to implement and record the choice.
• False-negative filter (UniME). Before the softmax, mask every candidate whose score exceeds the positive's score plus β, and mask the diagonal 2504.17432 §3.3. The released code takes top-k over the masked matrix and does not refill when too few candidates remain, so masked entries can enter the top-k 2504.17432 Pitfalls. Guard against this in your own code.
• Prototype contrast temperature (ProtoSeg). The paper uses τ = 0.1 in its pixel-to-prototype contrast 2203.15102 §5.1 eq. 11. The card records that the released code takes raw cosine similarity as logits with no temperature 2203.15102 Paper vs code. Decide which to implement and record the choice.
• Margin pair loss (SD-GAN). Replace the softmax with the pair loss of 1904.01480 eq. 2: pull same-pair distances down to the floor α, and push different-pair distances up to the margin ε. Batch construction is not described in the paper 1904.01480 Pitfalls.
• No projector (DirectCLR). Apply the loss to the first d0 channels of the encoder output, after normalising them, with no head 2110.09348 §6.2. The paper states no temperature term, so choose τ explicitly 2110.09348 §4.1.
• Dense use (VADeR). Treat each pixel as one sample. Build positives from geometry-matched pixel pairs across the two views, and keep the same loss 2011.05499 §3.1.
• Graph views (GCA). Keep the same-view negatives in the denominator, as eq. 1 does, not only the cross-view ones 2010.14945 eq. 1. Drop edges and mask feature dimensions with centrality-based probabilities, capped as in eq. 4 and eq. 9 2010.14945 eq. 4 eq. 9. The paper averages the two endpoint scores for an undirected edge 2010.14945 §3.3.1; the card lists where the reference code departs from this.

Contradictions

• Is the MoCo positive augmented? One survey says MoCo's positives come from the same sample "without any transformation or augmentation" 2006.08218 Pitfalls. Another survey lists MoCo's query as an augmented image and its positive as an augmented view of the same image 2010.05113 tab. 3. Check the original method before implementing.
• Binary cross-entropy vs InfoNCE, inside U2PL. The paper's main pixel loss is InfoNCE, but its appendix reports binary cross-entropy at 79.64 against 79.30 for InfoNCE at 1/4 on the blender split 2203.03884 app. tab. A4 2203.03884 tab. 6. The gap is small, and it is the authors' own result.
• U2PL: paper vs released code. The paper gives λ_c = 0.1 and M = 50 anchors and N = 256 negatives per class. The code uses λ_c = 1 and swaps the two counts, and its positive centre omits the δ_p filter 2203.03884 app. tab. A1 2203.03884 §3.3. Use the code's values only after deciding which one to test.
• Negatives: more is better, or not? The InfoNCE bound says more negatives tighten the MI bound 2006.08218 §6.2.2, and the review discusses more negatives as a helpful lever 2010.05113 §VI.B. The same survey, citing Arora et al., reports that beyond some threshold more negatives can hurt 2006.08218 §6.3.3. SigLIP also finds that batches above about 32k do not help and that 307k hurts both losses 2303.15343 §4.2. The text-embedding papers use large batches, and the cited tables show gains up to 12,288 2201.10005 tab. 9 and a batch of 32,768 in E5 2212.03533 tab. 11; neither card reports a batch size where the gain stops.
• Graph comparisons. The graph survey says contrastive methods beat generation- and auxiliary-based ones 2103.00111 tab. 5, but its numbers are taken from the cited works, which used different protocols and datasets. The survey itself flags this mixed provenance 2103.00111 Pitfalls. Treat the ordering as a literature overview, not a controlled comparison.
• Contrastive term in open-world discovery: evaluation protocol. The GCD paper computes one Hungarian match over all unlabelled images and reads Old and New accuracy from it. The NCD literature matches Old and New separately, so the same cluster can be counted twice. Its numbers are therefore not directly comparable to NCD papers 2201.02609 app. E 2201.02609 Pitfalls.
• Sigmoid vs softmax. SigLIP reports that softmax needs about 98k batch to reach the level the sigmoid reaches at 32k 2303.15343 §4.2. The 2020 review does not compare the two 2010.05113 §III.E.3. RemoteCLIP's loss ablation points the other way on a much smaller task: InfoNCE scores 36.32 retrieval average against 26.68 for a SigLIP-style loss, and 48.57 against 45.66 zero-shot 2306.11029 tab. 5. Those runs use a small remote-sensing set rather than web-scale pairs, so the two results do not settle the question.
• Llama2Vec eq. 5 and its text. The equation puts the query in the SELF block and the answer in the NEXT block 2312.15503 eq. 5. The text gives the reverse: queries use the NEXT prompt and answers use the SELF prompt 2312.15503 §3.2. Both readings are cited here, and this page does not choose between them.
• Batch size as the main lever. The text-embedding card reports NQ MRR@10 rising from 71.4 to 84.7 as batch size goes from 1,536 to 12,288 2201.10005 tab. 9. The OpenCLIP scaling study reports that batch size changed zero-shot ImageNet by only 0.2 to 0.5% in its control runs 2212.07143 app. Control experiments. The tasks and scales differ, so this page does not settle which holds for a given setup.
• UniME: text vs its own table. The text says the stage-2 model beats VLM2Vec with LLaVA-1.6 by 10.3% 2504.17432 §4.3. The same paper's MMEB table gives 66.6 against 63.3, a 3.3-point gap 2504.17432 tab. 1. The 10.3 figure may use a different setting or metric; this page quotes the table.

Papers

• Self-supervised Learning: Generative or Contrastive: taxonomy of generative, contrastive and adversarial self-supervision, with the InfoNCE, alignment and uniformity derivations.
• Contrastive Representation Learning: A Framework and Review: the four-part framework (similarity, encoder, head, loss) and classification of about 50 methods by modality.
• Understanding Self-Supervised Learning Dynamics without Contrastive Pairs: linear analysis of why BYOL- and SimSiam-style training avoids collapse, and DirectPred, a gradient-free predictor.
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients: ECG pre-training where positives are all segments of one patient, across time and leads.
• Text and Code Embeddings by Contrastive Pre-Training: in-batch contrastive training of text and code embedders from web and docstring pairs, initialised from GPT and Codex.
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences: label-free pixel codes distilled from a frozen DINO ViT, using a clamped correlation loss.
• Contrastive Learning as Goal-Conditioned Reinforcement Learning: binary contrastive critic over state-action and future-state pairs for goal-reaching agents.
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge: MineCLIP, a video-transcript InfoNCE model used as a dense reward for a Minecraft agent.
• Efficient Few-Shot Learning Without Prompts: SetFit, contrastive fine-tuning of a sentence encoder on labelled pairs, then a logistic-regression head.
• Text Embeddings by Weakly-Supervised Contrastive Pre-training: E5, contrastive pre-training on about 270M cleaned text pairs with batch 32,768.
• Graph Self-Supervised Learning: A Survey: contrast-based graph pretext tasks, augmentations, and compiled node and graph classification numbers.
• FedProc: Prototypical Contrastive Federated Learning on Non-IID data: federated averaging with a supervised prototype-contrast loss and a per-round weight shift into cross-entropy.
• Generalized Category Discovery: contrastive fine-tuning of a DINO ViT with unsupervised and supervised terms, then semi-supervised k-means for open-world labels.
• Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels: pixel-level InfoNCE with unreliable pixels as negatives, inside a teacher-student segmenter.
• Sigmoid Loss for Language Image Pre-Training: per-pair sigmoid loss for image-text contrast, with a chunked multi-device formulation.
• Learning and Evaluating Representations for Deep One-class Classification: distribution-augmented instance contrast on inliers, with KDE or OC-SVM scoring.
• TSDAE: sentence encoder trained by reconstruction from one vector; no contrastive loss.
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph: graph attention nested in each language-model layer, with link-prediction contrast on textual graphs.
• Align and Prompt: Video-and-Language Pre-training with Entity Prompts: video-text contrast plus entity pseudo-labels from a frozen prompter.
• Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval: a mapper to a pseudo-word token, trained with symmetric contrast over unlabelled images.
• ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding: 3D point clouds aligned to frozen image and text encoders, with BLIP-2 captions.
• Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval: unsupervised adaptation of LLaMA-2, then contrastive LoRA fine-tuning for retrieval.
• VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time: talking-face generation; uses a cosine identity term, not a contrastive loss.
• Contrastive Learning with Stronger Augmentations: CLSA, a distribution-matching term that lets strongly augmented views join a MoCo-style queue.
• Towards artificial general intelligence via a multimodal foundation model: BriVL, a two-tower image-text model with momentum encoders and queues, pre-trained on Chinese web pairs.
• A Contrastive Framework for Neural Text Generation: SimCTG, a within-sequence contrastive loss for GPT-2 fine-tuning, plus contrastive search at decoding.
• BRIO: Bringing Order to Abstractive Summarization: margin ranking over candidate summaries, added to cross-entropy for summarisation.
• Learning Salient Boundary Feature for Anchor-free Temporal Action Localization: AFSD, an anchor-free localizer with a boundary triplet loss on start and end features.
• CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts: a residual MLP on frozen CLIP text features, trained with prompt-vs-augmented-prompt InfoNCE.
• A Survey on Bridging EEG Signals and Generative AI: survey of EEG-to-media generation, noting contrastive alignment as a common step.
• JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization: contrastively trained prior estimator for joint audio-video generation.
• Semantics Disentangling for Text-to-Image Generation: Siamese margin pair loss on discriminator features, so captions of one image give consistent pictures.
• Unsupervised Learning of Dense Visual Representations: per-pixel InfoNCE from view geometry with a MoCo queue, for frozen dense features.
• Semi-Supervised and Unsupervised Deep Visual Learning: A Survey: taxonomy of semi- and unsupervised visual methods, linking contrast to smoothness assumptions. No experiments.
• Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation: symmetric InfoNCE with GPS and similarity-based hard negatives for cross-view matching.
• Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs: UniME, batch-similarity distillation then false-negative-filtered InfoNCE for a multimodal LLM embedder.
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation: image-text contrast with a momentum teacher, a 65,536-entry queue and soft targets, before fusion.
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation: momentum-encoder image-text contrast with soft targets, plus similarity-based hard negatives for matching.
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models: query transformer trained with image-text contrast between frozen encoders and a frozen language model.
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale: masked prediction against quantised distractors for multilingual speech, with language sampling.
• Time-Series Representation Learning via Temporal and Contextual Contrasting: two-view contrast for raw time series, with temporal and contextual terms.
• Learning To Retrieve Prompts for In-Context Learning: DPR-style retriever for in-context examples, with positives and hard negatives labelled by a scoring LM.
• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation: pairwise two-way softmax scorer trained on real user picks for text-to-image outputs.
• LAION-5B: An open large-scale dataset for training next generation image-text models: web-scale image-text dataset filtered by a CLIP similarity threshold, with InfoNCE reproduction runs.
• Reproducible scaling laws for contrastive language-image learning: OpenCLIP scaling study with a global-batch InfoNCE recipe and power-law fits.
• DataComp: In search of the next generation of multimodal datasets: benchmark that fixes a CLIP-style contrastive recipe and varies only the image-text pairs, with CLIP-score filtering baselines.
• RemoteCLIP: A Vision Language Foundation Model for Remote Sensing: continual CLIP pretraining on remote-sensing pairs with captions generated from boxes and masks.
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning: node-versus-subgraph contrast with a bilinear scorer, giving unsupervised node anomaly scores.
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training: tabular transformer with intersample attention and CutMix and mixup contrastive pre-training.
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning: shows dimensional collapse in contrastive learning, and proposes DirectCLR, a projector-free InfoNCE on a sub-vector.
• SLIP: Self-supervision meets Language-Image Pre-training: CLIP's image-text loss plus a SimCLR loss on one shared Vision Transformer.
• Expanding Language-Image Pretrained Models for General Video Recognition: X-CLIP, which adapts CLIP to video with cross-frame message tokens and video-specific class prompts.
• An Empirical Study of Training Self-Supervised Vision Transformers: MoCo v3 on ViT backbones, the instability it causes, and a frozen random patch projection that reduces it.
• A Survey on Contrastive Self-supervised Learning: survey of contrastive pretext tasks, negative-storage designs and losses for vision and NLP, with benchmark numbers copied from the original papers.
• Graph Contrastive Learning with Adaptive Augmentation: GCA, two graph views with edges and feature dimensions dropped in proportion to node centrality.
• Towards Open World Object Detection: ORE, contrastive clustering of detector features with pseudo-labelled unknowns and an energy-based unknown test.
• MMIM: hierarchical mutual-information fusion for multimodal sentiment: MMIM, a fusion vector trained to predict each modality by InfoNCE, plus a Barber–Agakov bound on input pairs.
• FLAVA: A Foundational Language And Vision Alignment Model: one transformer family for image, text and image-text tasks, trained with a global contrastive loss and masked objectives.
• FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding: supervised contrastive head on RoI features for few-shot detection, with the contrastive term added to the detector loss.
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning: AD-GCL, a learned adversarial edge dropper for graph-level contrast.
• CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting: momentum-queue trend contrast and Fourier seasonal contrast for forecasting.
• Think Then Embed: Generative Context Improves Multimodal Embedding: a reasoner writes a trace that the embedder encodes alongside the input, trained with batch InfoNCE.
• Feudal Networks for Visual Navigation: image-goal navigator whose latent memory is a SMoG-style clustering, with positives chosen by SuperGlue match confidence.
• Raising Body Ownership in End-to-End Visuomotor Policy Learning via Robot-Centric Pooling: robot-centric attention pooling with an auxiliary MoCo-style contrast for distractor robustness.
• STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft: audio-video CLIP-style mapping for a Minecraft agent, with a conditional VAE prior into the agent's goal space.
• BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data: wav2vec 2.0-style encoder for raw EEG, pre-trained by masked-span contrast against distractors.
• SSD: A Unified Framework for Self-Supervised Outlier Detection: contrastive features with a single-Gaussian Mahalanobis score, for outlier detection without labels.
• Contrastive Test-Time Adaptation: AdaContrast, source-free adaptation with a momentum-model contrastive term and pseudo-label-aware negatives.
• ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding: aligns a 3D point-cloud encoder with frozen image and text encoders using symmetric contrastive losses.
• Improving Text Embeddings with Large Language Models: synthetic LLM-written training pairs and LoRA-tuned InfoNCE for a decoder-only text embedder.
• Rethinking Semantic Segmentation: A Prototype View: non-learnable sub-cluster prototypes per class, with a pixel-to-prototype contrast term.
• Self-Supervised Speech Representation Learning: A Review: survey that files CPC and wav2vec 2.0 under contrastive methods, with the InfoNCE forms written out.
• Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts: sparse mixture-of-experts one tower trained with a symmetric image-text contrastive loss and entropy routing losses.