--- kind: concept slug: text-embedding title: Text embedding summary: >- Fixed-size vectors for text or code, so similar inputs sit close. Variants differ by training signal: weak pairs, denoising, silver labels, teacher distillation, few-shot tuning, n-gram averaging, LLM-written data or an unlabelled LLM. Frozen embeddings can feed a classical classifier when labels are scarce. Sparse concept latents are interpretable. Pick by labels, compute and serving budget. tags: [embeddings, retrieval, few-shot, contrastive-learning, denoising, quantization, siamese-network, multilingual, long-context, sentiment-analysis, knowledge-distillation, unsupervised-learning, synthetic-data, interpretability, small-sample-learning] papers: ["1703.02507", "1904.08067", "1908.10084", "1911.03688", "2004.09813", "2006.03541", "2010.08240", "2103.11790", "2104.06979", "2201.10005", "2209.11055", "2212.03533", "2312.00819", "2312.15503", "2401.00368", "2402.03216", "2512.10092", "2603.18161"] related: [contrastive-learning, self-supervised-learning, few-shot-learning, zero-shot-learning, prompt-engineering, domain-adaptation, data-augmentation, nearest-neighbor-search, retrieval-augmented-generation, dense-retrieval, learned-sparse-retrieval, denoising-autoencoder, quantization-aware-training, low-rank-adaptation, metric-learning, siamese-network, knowledge-distillation, text-classification, sentiment-analysis, word-embedding, multilingual-evaluation, synthetic-query-generation, sparse-autoencoder, mechanistic-interpretability, controllable-text-generation, toxicity-detection] status: draft updated_at: 2026-11-01 --- ## Overview A text embedding model encodes a string into a vector, usually compared with cosine similarity. The vectors then serve search, clustering, and light classification [arxiv:2201.10005 §1]. Most papers in this wiki share one design: a sentence encoder is run on each side of a pair, and training shapes the vector space so that the pairs the task cares about land close together. They differ in where the pairs come from and what the labels are. Some papers depart from pairs: TSDAE reconstructs a sentence from a damaged copy of itself [arxiv:2104.06979 §3.1], Llama2Vec adapts a decoder LLM with two prompts [arxiv:2312.15503 §3.2], and Sent2Vec predicts each word from the rest of its sentence [arxiv:1703.02507 §2.1]. A sentiment study uses word vectors only as the input layer of a supervised classifier, with no pair objective [arxiv:2006.03541 §4.4]. The multilingual distillation method does use translation pairs, but its target is a teacher's vector rather than a pair label [arxiv:2004.09813 §2]. Three papers go further. One has an LLM write its own training examples [arxiv:2401.00368 §3.1]. One replaces the dense vector with a sparse list of named concept latents [arxiv:2512.10092 §3]. One uses a fixed direction in a sentence-embedding space as a scalar score rather than as a search index [arxiv:2103.11790 eq. 1]. The shared mechanics: a pooled encoder output per input, a similarity (usually cosine), and a loss that pulls positives together and pushes negatives apart [arxiv:2201.10005 §2.1] [arxiv:2209.11055 §3.1]. ## Variants ### Contrastive pre-training on naturally paired data Pairs come for free from the web (neighbouring text) or from code (a docstring and its function). Negatives are the other pairs in the batch. The embedding is the last-layer state at an end delimiter [arxiv:2201.10005 §2.1]. The authors train text models from GPT-family checkpoints and code models from Codex checkpoints [arxiv:2201.10005 §2.2]. Card: [Text and Code Embeddings by Contrastive Pre-Training](/papers/2201.10005). Reported gains are largest on retrieval and classification. On a 7-task linear probe the largest text model reaches 91.8 against 88.3 for DECLUTR [arxiv:2201.10005 tab. 2]. On a sentence-similarity benchmark it reaches 71.1 against 79.8 for SimCSE [arxiv:2201.10005 tab. 4]. ### Weakly supervised contrastive pre-training at scale (E5) Pairs are mined from semi-structured sources: forum posts with replies, questions with answers, encyclopedia sections with their passages, paper titles with abstracts, and web title-passage pairs. A consistency filter then removes weak pairs. It trains a model on about 1.3 billion candidate pairs, ranks each passage against 1 million random passages, and keeps a pair only when its passage ranks in the top two. About 270 million pairs remain [arxiv:2212.03533 §3]. Training uses in-batch negatives, a batch of 32,768 and a temperature of 0.01 [arxiv:2212.03533 §4.1] [arxiv:2212.03533 tab. 11]. Batch size matters here. On base models the BEIR average falls from 51.6 at 32k to 50.2 at 8k and 45.8 at 1k [arxiv:2212.03533 tab. 5]. Unsupervised E5-PT-base averages 42.9 nDCG@10 on BEIR against 41.7 for BM25 [arxiv:2212.03533 tab. 1]. A fine-tuning round on NLI, MS-MARCO and NQ, with hard negatives and teacher distillation, lifts the base model to 60.4 on the 56-dataset English MTEB average [arxiv:2212.03533 tab. 3]. The paper's pre-training took 16 to 64 V100 GPUs for 1 to 2 days [arxiv:2212.03533 tab. 11]. Card: [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533). ### Supervised siamese fine-tuning on labelled pairs (SBERT) A pretrained BERT or RoBERTa is fine-tuned with two weight-tied copies, one per sentence. Each side is mean-pooled into one vector, and the two vectors are compared with cosine similarity. Training uses one of three objectives: a three-way classifier over NLI pairs, applied to the two vectors and their absolute difference; a regression of the cosine onto a gold score; or a triplet hinge [arxiv:1908.10084 §3]. The classifier is used only during training and is dropped at inference [arxiv:1908.10084 §6]. The main gain is speed. The authors estimate that the closest pair among 10,000 sentences takes about 65 hours with a BERT cross-encoder and about 5 seconds with SBERT [arxiv:1908.10084 §1] [arxiv:1908.10084 §8]. Accuracy is competitive but not uniformly higher. SBERT-NLI-large averages 76.55 Spearman over seven STS tasks without STS training data, against 71.22 for the Universal Sentence Encoder [arxiv:1908.10084 tab. 1]. When trained on the STS benchmark, the large BERT model scores 85.64 and the large SBERT model 84.45 [arxiv:1908.10084 tab. 2]. Card: [Sentence-BERT](/papers/1908.10084). ### Small dual encoder for reply selection (ConveRT) A context encoder and a response encoder share one small Transformer, and each side has its own feed-forward head. Outputs are L2-normalised and scored with a scaled cosine [arxiv:1911.03688 §2.1]. It is trained on Reddit (input, response) pairs with an in-batch softmax and label smoothing of 0.2 [arxiv:1911.03688 §2.1] [arxiv:1911.03688 §3]. Compactness is the design goal: 512-dimensional embeddings, six blocks, a subword vocabulary of 31,476 tokens plus 1,000 hash buckets for unknown characters, and quantization-aware training with 8-bit embeddings and 16-bit other weights [arxiv:1911.03688 §2.1]. The authors report a 116 MB model before quantization, against 444 MB for PolyAI-dual [arxiv:1911.03688 tab. 2]. On Reddit, the single-context model scores 68.2 R100@1 and the version that reads up to ten earlier messages scores 71.8, against 61.3 for PolyAI-dual [arxiv:1911.03688 tab. 3]. The authors also claim pre-training takes about 18 hours on 12 GPU nodes [arxiv:1911.03688 §3]. Card: [ConveRT](/papers/1911.03688). ### Denoising reconstruction on unlabelled sentences (TSDAE) No pairs are needed. Each sentence has 60 percent of its tokens deleted [arxiv:2104.06979 app. A] [arxiv:2104.06979 tab. 6]. The encoder compresses the damaged sentence into one [CLS] vector, and a decoder that attends only to that vector must rebuild the original. Encoder and decoder share weights, except for the decoder's cross-attention [arxiv:2104.06979 §3.1] [arxiv:2104.06979 §3.2] [arxiv:2104.06979 app. A]. The decoder is discarded after training, and the encoder alone embeds text [arxiv:2104.06979 §3.2]. The method is aimed at specialised domains. Its unsupervised average over four domain tasks is 55.2, against 52.4 for Contrastive Tension (CT) [arxiv:2104.06979 tab. 2]. On SciDocs it reaches 74.0 against 67.6 for CT [arxiv:2104.06979 tab. 2]. On the general STS benchmark it reaches 66.0 against 73.9 for CT [arxiv:2104.06979 tab. 3]. Card: [TSDAE](/papers/2104.06979). ### Averaged unigram and bigram vectors trained without pairs (Sent2Vec) No encoder network is involved. Each sentence vector is the mean of learned vectors for its unigrams and bigrams [arxiv:1703.02507 §2.1] [arxiv:1703.02507 eq. 2]. Training takes each word of a sentence as a target and predicts it from the rest of the sentence, with negative sampling and no fixed window [arxiv:1703.02507 §2.1] [arxiv:1703.02507 §2.3]. Bigrams are hashed into a fixed number of buckets [arxiv:1703.02507 §2.2]. The authors add a bigram dropout step during training, and a sentence has no dropout at inference [arxiv:1703.02507 §2.4]. The paper reports the unigram+bigram model at 82.0 average accuracy on six supervised tasks, trained on Toronto books. SkipThought, which reads word order, scores 83.8 on the same table [arxiv:1703.02507 tab. 1]. On unsupervised STS 2014 and SICK 2014 the average Spearman and Pearson are 0.65 and 0.67 [arxiv:1703.02507 tab. 2]. Training on the Toronto books corpus takes about 1.2 hours on one node with 30 threads [arxiv:1703.02507 tab. 3]. Card: [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](/papers/1703.02507). The released trainer's defaults differ from the paper's table in learning rate, n-gram order and an L1 penalty, so the card's "Paper vs code" section lists the flags to set. ### Bi-encoder trained with cross-encoder silver labels (Augmented SBERT) A slow, accurate cross-encoder is fine-tuned on a small gold set. It then labels new sentence pairs, and the fast bi-encoder trains on gold plus these soft-labelled pairs [arxiv:2010.08240 §3.1]. The paper's experiments show that which pairs are chosen matters more than how many. BM25 retrieval gives most of the gain at the smallest silver set, and random pairs often hurt [arxiv:2010.08240 §6.1] [arxiv:2010.08240 tab. 11]. The same recipe can adapt to an unlabelled target domain with no target labels [arxiv:2010.08240 §3.2]. Card: [Augmented SBERT](/papers/2010.08240). ### Multilingual student distilled from an English teacher (multilingual distillation) An English sentence encoder, already tuned on NLI and STS data, is the fixed teacher. A multilingual student, XLM-RoBERTa, is trained on parallel sentences. For each pair, the loss adds two squared distances to the teacher's English vector: one from the student's vector of the source and one from the student's vector of the translation [arxiv:2004.09813 §2] [arxiv:2004.09813 §3]. The student therefore places each language in the teacher's English space. The training data are OPUS corpora and bilingual dictionaries [arxiv:2004.09813 §3]. Results are the authors' own. The paraphrase-trained teacher's student averages 84.6 on monolingual STS 2017 and 83.7 on cross-lingual STS 2017, against 75.4 and 67.0 for LASER and 76.4 and 73.5 for LaBSE [arxiv:2004.09813 tab. 1] [arxiv:2004.09813 tab. 2]. Its BUCC bitext mining F1 is 88.6, below LASER at 93.0 and LaBSE at 93.5 [arxiv:2004.09813 tab. 3]. The authors attribute the gap to the method targeting similarity rather than exact translation [arxiv:2004.09813 §4.2]. On Tatoeba for four low-resource languages the NLI-trained student beats LASER in both directions [arxiv:2004.09813 tab. 4]. LASER was trained on Tatoeba, so that comparison may favour LASER's numbers [arxiv:2004.09813 app. A]. The data size matters: EN-DE reaches 71.5 with 1,000 TED2020 pairs and 80.4 with the full set [arxiv:2004.09813 tab. 6]. Card: [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](/papers/2004.09813). ### Few-shot contrastive fine-tuning with a linear head (SetFit) Starting from a pretrained sentence encoder, the body is fine-tuned on pairs built from a handful of labelled examples. Same-label pairs are positive and different-label pairs are negative. A logistic-regression head is then trained on the tuned embeddings. No prompts are used [arxiv:2209.11055 §3.1]. Card: [Efficient Few-Shot Learning Without Prompts](/papers/2209.11055). Results are in the table under "When to use which". The 110M-parameter body is close to a 3B prompt-based model at 8 examples per class and ahead of it at 64 [arxiv:2209.11055 tab. 2]. ### LLM backbone adapted without labels (Llama2Vec) The starting point is LLaMA-2-7B, a decoder-only model. Adaptation runs on Wikipedia passages. One forward pass reads the passage followed by two prompts: one asks the model to rebuild the passage, the other to predict the next one. Each prompt's end state is read as an embedding. A custom attention mask stops the two prompts from attending to each other [arxiv:2312.15503 §3.2] [arxiv:2312.15503 fig. 2]. Fine-tuning then uses contrastive learning with LoRA on MS MARCO [arxiv:2312.15503 §4.1]. The authors say one joint pass saves almost half the cost of two separate passes [arxiv:2312.15503 §3.2]. Reported results: MS MARCO passage MRR@10 of 43.1, against 41.2 for RepLLaMA [arxiv:2312.15503 tab. 1]. Zero-shot BEIR average NDCG@10 of 56.4, against 53.9 for RepLLaMA [arxiv:2312.15503 tab. 3]. The released code differs from the paper in its read-out and its adaptation loss, so implement from the card's "Paper vs code" section, not from the paper alone. Card: [Llama2Vec](/papers/2312.15503). ### LLM writes its own training data, then tunes a 7B decoder (E5-Mistral) A proprietary LLM is prompted to invent search-style tasks. For each of four length-based groups it first proposes about 20 one-sentence task definitions, then writes one JSON example per task, in 93 languages. The paper reports about 500k examples after filtering [arxiv:2401.00368 §3.1] [arxiv:2401.00368 §4.1]. Mistral-7B is then fine-tuned with LoRA at rank 16 for one epoch, using InfoNCE at a fixed temperature of 0.02 [arxiv:2401.00368 §3.2] [arxiv:2401.00368 §4.2] [arxiv:2401.00368 eq. 3]. Only queries carry a task instruction; documents do not. The embedding is the last-layer state at the end-of-sequence token [arxiv:2401.00368 §3.2]. Removing the instruction costs 4.2 MTEB points [arxiv:2401.00368 tab. 5]. Results are the authors' own. Trained on synthetic data alone, the model averages 63.1 on the 56-dataset English MTEB, against 62.3 for E5-large-v2 [arxiv:2401.00368 tab. 1]. With the labelled data added, it reaches 66.6, against 64.2 for BGE-large-en-v1.5 [arxiv:2401.00368 tab. 1]. It is weaker on low-resource languages: 68.4 on Swahili MIRACL dev, against 74.9 for mE5-large [arxiv:2401.00368 tab. 2]. The released repository holds only evaluation code, so the training recipe is paper-only. The paper does not discuss the terms of the generator's outputs, so check them before you reproduce the data pipeline. Card: [Improving Text Embeddings with Large Language Models](/papers/2401.00368). ### Multilingual multi-function encoder with long inputs (M3-Embedding) One XLM-RoBERTa-based encoder yields three representations from one pass: a normalised [CLS] vector for dense search, a learned non-negative weight per token for lexical matching, and a normalised vector for every token for multi-vector scoring [arxiv:2402.03216 §3.2]. Candidates can be re-ranked by a weighted sum of the three scores, with weights set per benchmark [arxiv:2402.03216 eq. 1] [arxiv:2402.03216 §3.2]. Training has three stages: RetroMAE pre-training, unsupervised contrastive learning on the dense head, and fine-tuning of all three heads with self-distillation, where the combined score is the teacher [arxiv:2402.03216 §3.3] [arxiv:2402.03216 app. B.1]. Inputs of up to 8,192 tokens are encoded in sub-batches with gradient checkpointing [arxiv:2402.03216 §3.4] [arxiv:2402.03216 alg. 1]. An inference-only pooling over [CLS] tokens inserted every 256 tokens is also proposed for long text [arxiv:2402.03216 app. B.2]. The gains are reported on the authors' own tables. On the MIRACL dev set the all-heads score averages 71.5 nDCG@10, against 66.6 for mE5-large [arxiv:2402.03216 tab. 1]. On MLDR, the MultiLongDoc test split, it reaches 65.0 against 42.6 for E5-mistral-7b [arxiv:2402.03216 tab. 3]. Self-distillation matters most for the sparse head: 53.9 with it and 36.7 without, on MIRACL [arxiv:2402.03216 tab. 5]. Card: [M3-Embedding](/papers/2402.03216). ### Pre-trained word vectors as classifier features (survey) Not a training method. The survey lists word embeddings as one feature option for text classification, next to bag-of-words and TF-IDF. The embeddings it names are Word2Vec, GloVe and FastText, plus contextual representations [arxiv:1904.08067 §2.4] [arxiv:1904.08067 tab. 1]. Bag of words counts words over a fixed vocabulary and ignores order [arxiv:1904.08067 §2.3]. The survey reports no experiments. Its comparisons are qualitative, and it notes that pre-trained GloVe, trained on Wikipedia, can underperform TF-IDF on short messages [arxiv:1904.08067 §1] [arxiv:1904.08067 §6]. Card: [Text Classification Algorithms: A Survey](/papers/1904.08067). ### Word vectors inside a supervised sentiment classifier (comparative study) Not a pair-based embedding method. Learned word vectors, width 300 over a vocabulary of 15,000 words and sequences of up to 40 tokens, form the input layer of a CNN or an LSTM trained on polarity labels. The paper compares that input against TF-IDF features [arxiv:2006.03541 §4.3] [arxiv:2006.03541 §4.4]. Evaluation uses 10-fold cross-validation over eight tweet and review datasets [arxiv:2006.03541 §4.1] [arxiv:2006.03541 §4.2]. Word vectors beat TF-IDF for the CNN on all eight datasets. On Sentiment140 the CNN scores 0.8006 with word vectors and 0.7669 with TF-IDF, and on Cornell Movie Reviews 0.7137 against 0.6787 [arxiv:2006.03541 tab. 2]. The LSTM with word vectors is the most accurate model on six of the eight datasets. It also trains about nine times longer than the CNN on Sentiment140 [arxiv:2006.03541 tab. 2] [arxiv:2006.03541 tab. 7]. The authors report no baseline outside their three models and no variance across runs [arxiv:2006.03541 §4.2]. Card: [Sentiment Analysis Based on Deep Learning: A Comparative Study](/papers/2006.03541). ### Frozen embeddings of structured records as classifier features (travel behaviour) Not a pair method. Each record is rendered as plain text, here a trip with travel times and costs for each mode and some traveller attributes. An LLM embedding of that text is computed once per record. A classical head is then fitted on the vectors: multinomial logit, random forest or a neural network [arxiv:2312.00819 §3.4] [arxiv:2312.00819 §4.2]. The encoder is not trained. The paper gives no settings for the heads, and its prose does not name the embedding model [arxiv:2312.00819 §3.4] [arxiv:2312.00819 §3.2]. The embedding dimension is not stated [arxiv:2312.00819 §3.1]. Card: [Large Language Models for Travel Behavior Prediction](/papers/2312.00819). Results are the authors' own, on a mode-choice task and a trip-purpose task, with 200-row balanced test sets averaged over five random samples [arxiv:2312.00819 §4.1] [arxiv:2312.00819 tab. 3]. Embedding heads are reported only with 10 training samples. Mode accuracy rises for each head: MNL from 0.469 to 0.494, RF from 0.455 to 0.509, NN from 0.462 to 0.532. On trip purpose, NN rises from 0.268 to 0.296 [arxiv:2312.00819 tab. 3]. The ± spreads overlap in five of the six pairs. The exception is the RF trip-purpose baseline, whose spread is zero [arxiv:2312.00819 tab. 3]. The paper reports no significance tests [arxiv:2312.00819 §4]. At 1,000 samples, no embedding rows are reported, so the paper gives no evidence for that size [arxiv:2312.00819 tab. 3]. ### Embeddings as a measurement tool (not a method) A study of LLM-edited essays embeds drafts with off-the-shelf embedding models and projects them onto a two-dimensional PCA plane per condition [arxiv:2603.18161 §3.2] [arxiv:2603.18161 §4.2]. The paper contributes no embedding training. Card: [How LLMs Distort Our Written Language](/papers/2603.18161). Do not copy its plane construction as a retrieval method. The paper does not state that the planes are comparable across rows, and the authors' code fits one plane per row [arxiv:2603.18161 §3.2]. ### Sparse concept latents as an interpretable embedding (SAE) Not a trained encoder. A reader LLM runs over the text, and a sparse autoencoder trained on one of its layers splits each hidden state into features. The paper uses Llama 3.3 70B at layer 50, with a public SAE of 65,536 latents [arxiv:2512.10092 §4]. A document's vector is the maximum activation of each latent over its tokens [arxiv:2512.10092 §3]. Each latent carries a short label, so every dimension can be read and compared [arxiv:2512.10092 §3]. The paper uses these vectors for dataset diffing, concept co-occurrence, clustering along chosen axes and property-based retrieval [arxiv:2512.10092 §4]. Its API accepts at most 2048 tokens, so longer documents need chunking, which the paper does not describe [arxiv:2512.10092 §4]. On six retrieval datasets, the SAE ranking beats the OpenAI-plus-LLM baseline on four: responses 0.418 against 0.373 MAP, reasoning traces 0.409 against 0.333, biology abstracts 0.529 against 0.524 and short stories 0.468 against 0.406. It trails on prompts, 0.361 against 0.412, and on The Pile, 0.443 against 0.446 [arxiv:2512.10092 §4.4]. The values are before LLM reranking. The temperature and phrase count were chosen by best average on the same queries that are reported, which may inflate the gain [arxiv:2512.10092 §4.4]. Cost is lower than pure-LLM hypothesis generation, by 2 to 8 times in the paper's own pipeline [arxiv:2512.10092 §4.1]. Card: [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092). ### A single direction as a scorer (moral direction) Not a search index. Each of 64 verbs is placed in ten question frames and embedded with BERT-large using SBERT pooling. The verb's ten vectors are averaged, and the first principal component of the verb vectors is taken as a direction [arxiv:2103.11790 §Identifying the Moral Direction of Language Models] [arxiv:2103.11790 suppl. B.2]. A phrase's score is its coordinate on that direction [arxiv:2103.11790 eq. 1]. Against human yes-rates, the authors report r = 0.78 and 0.79 [arxiv:2103.11790 §BERT's MoralDirection Strongly Correlates with Human Moral Norms]. The score then filters candidate tokens during GPT-2 decoding. With threshold 0, 1.17% of prompts get a generation above toxicity 0.9, against 9.82% for GPT-2 [arxiv:2103.11790 §Reducing Neural Toxic Degeneration in Language Models]. These are author-reported. The correlation depends on the verb set. Three random sets of 64 verbs give r = 0.64, 0.60 and -0.01 [arxiv:2103.11790 suppl. B.2]. The filter's threshold is tied to one normalising constant and one encoder. The paper gives no tuning procedure, so a new encoder or verb set means re-tuning it. See the card's Pitfalls. Card: [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](/papers/2103.11790). ## When to use which | Situation | Use | Why | |---|---|---| | Large corpus of naturally paired text or code, big compute for large batches, search or linear-probe use | Contrastive pre-training [arxiv:2201.10005] | Needs no labels. Batch size is the main lever: 84.7 against 71.4 at batch 12288 versus 1536 on one NQ dev run [arxiv:2201.10005 tab. 9] | | Weak-pair corpus of hundreds of millions of pairs, multi-GPU budget, general-purpose retrieval and MTEB-style use | E5-style pre-training [arxiv:2212.03533] | Unsupervised BEIR 42.9 against 41.7 for BM25; fine-tuned MTEB 60.4 [arxiv:2212.03533 tab. 1] [arxiv:2212.03533 tab. 3]. Dropping the batch from 32k to 8k costs about 1.4 BEIR points on base models [arxiv:2212.03533 tab. 5] | | Conversational reply ranking under tight memory or latency limits, with Reddit-like reply pairs | ConveRT [arxiv:1911.03688] | Authors' size is 116 MB before quantization [arxiv:1911.03688 tab. 2]. Multi-context reaches 71.8 on Reddit [arxiv:1911.03688 tab. 3]. The single-context model scores 38.2 on DSTC7-Ubuntu, against 71.2 for multi-context [arxiv:1911.03688 tab. 5] | | Unlabelled in-domain sentences only, and the target is retrieval or paraphrase in a specialised domain rather than STS | TSDAE [arxiv:2104.06979] | Four-domain average 55.2 against 52.4 for CT [arxiv:2104.06979 tab. 2]. Do not use it for general STS: 66.0 against 73.9 for CT [arxiv:2104.06979 tab. 3] | | Only a few labelled examples per class (8 or 64) and you want a classifier without writing prompts | SetFit [arxiv:2209.11055] | Reported N = 8 average 62.3 ± 4.9 against 58.3 for ADAPET; N = 64 average 75.3 ± 1.3 against 73.8 for ADAPET [arxiv:2209.11055 tab. 2]. Variance at N = 8 is high | | A labelled pair set, a large unlabelled pool, and you need a fast bi-encoder for pairwise scoring | AugSBERT with BM25 pair selection [arxiv:2010.08240] | Smallest silver sets, and the best average score [arxiv:2010.08240 §6.1] [arxiv:2010.08240 tab. 11]. Random pairs can fall below the no-silver baseline [arxiv:2010.08240 tab. 5] | | Target domain has no labels, a labelled source domain exists | AugSBERT domain adaptation [arxiv:2010.08240 §3.2] | One pair reaches 0.875 against 0.505 for the source-only model [arxiv:2010.08240 tab. 6]. Other pairs fall short of the in-domain upper bound, for example 0.602 against 0.715 | | GPU budget for a 7B model, labelled search data for fine-tuning, and zero-shot BEIR transfer as the goal | Llama2Vec [arxiv:2312.15503] | BEIR average 56.4 against 53.9 for RepLLaMA [arxiv:2312.15503 tab. 3]. The model has 7B parameters, so serving costs far more than the encoders above | | Labelled English sentence pairs or triplets, and you need cosine search or clustering over a large collection without running a cross-encoder per pair | SBERT [arxiv:1908.10084] | Mean-pooled siamese encoder; SBERT-NLI-large averages 76.55 on STS without STS training [arxiv:1908.10084 tab. 1]. On cross-topic argument similarity it trails BERT, 50.65 against 57.23 [arxiv:1908.10084 tab. 3] | | No labelled pairs, access to a strong LLM for data generation, a 7B model you can serve, and English-heavy retrieval | E5-Mistral [arxiv:2401.00368] | Synthetic data only gives MTEB 63.1, against 62.3 for E5-large-v2 [arxiv:2401.00368 tab. 1]. Weaker on low-resource languages: Swahili 68.4 against 74.9 for mE5-large [arxiv:2401.00368 tab. 2]. The generator's terms are not discussed in the paper | | Need to see which named concepts drive a document's place in a corpus or a comparison, with a reader LLM and a released SAE, and inputs under 2048 tokens | Sparse concept latents [arxiv:2512.10092] | Retrieval MAP close to dense baselines, and not uniformly higher: 0.418 against 0.373 on responses, 0.361 against 0.412 on prompts [arxiv:2512.10092 §4.4]. Cost 2 to 8 times lower than pure-LLM hypothesis generation [arxiv:2512.10092 §4.1]. Not a drop-in replacement for a dense index | | A fixed scalar score is enough, such as a filter in decoding, and an encoder is already in use | Single-direction score [arxiv:2103.11790] | Correlation with human ratings 0.78 and 0.79 with the paper's verbs, but 0.64, 0.60 and -0.01 with other verb sets [arxiv:2103.11790 suppl. B.2]. Each candidate needs an encoder pass per step. Test on your own verbs first | | Multilingual retrieval over documents up to 8,192 tokens, and you want dense, sparse and multi-vector scores from one model | M3-Embedding [arxiv:2402.03216] | All-heads score 71.5 on MIRACL dev and 65.0 on MLDR [arxiv:2402.03216 tab. 1] [arxiv:2402.03216 tab. 3]. On MIRACL the multi-vector head re-ranks the dense top 200 [arxiv:2402.03216 §4.1], and the paper does not discuss its storage cost [arxiv:2402.03216 §3.2] | | Text classification with a classical classifier (SVM, random forest) and no budget for training a neural encoder | Bag of words or TF-IDF features, with word vectors as an option to test [arxiv:1904.08067 §2.3] [arxiv:1904.08067 §2.4] | The survey reports no measured numbers [arxiv:1904.08067 §6]. Its advice on word vectors versus TF-IDF is qualitative, and pre-trained GloVe can trail TF-IDF on short messages [arxiv:1904.08067 §6] | | Labelled sentiment or polarity data in the thousands, a CNN or LSTM trained from scratch, and no pre-trained encoder | CNN on learned word vectors, in place of TF-IDF [arxiv:2006.03541] | CNN with word vectors beats CNN with TF-IDF on all eight datasets: 0.8006 against 0.7669 on Sentiment140 [arxiv:2006.03541 tab. 2]. The LSTM is more accurate on six datasets but trains about nine times longer on Sentiment140 [arxiv:2006.03541 tab. 7]. Author-reported, with no variance [arxiv:2006.03541 §4.2] | | No labels and no pairs, cheap CPU training, and a fixed-size vector for classification or similarity with a linear model on top | Sent2Vec [arxiv:1703.02507] | Unigram+bigram model at 82.0 supervised average against 83.8 for SkipThought, which reads word order [arxiv:1703.02507 tab. 1]. About 1.2 hours on Toronto books with 30 threads [arxiv:1703.02507 tab. 3]. Use the card's flags, since the released defaults differ from the paper | | A strong English sentence encoder already exists, parallel data exists for the target languages, and vectors must be comparable across languages | Multilingual distillation [arxiv:2004.09813] | STS 2017 cross-lingual average 83.7 against 73.5 for LaBSE [arxiv:2004.09813 tab. 2]. Do not use it for bitext mining: BUCC F1 is 88.6, against 93.5 for LaBSE [arxiv:2004.09813 tab. 3] | | Roughly 10 labelled examples per task, a classical head (MNL, RF or NN) is wanted, and each record can be written as text | Frozen LLM embeddings with a classical head [arxiv:2312.00819 §3.4] | Mode accuracy at 10 samples: 0.469 to 0.494 for MNL, 0.455 to 0.509 for RF, 0.462 to 0.532 for NN [arxiv:2312.00819 tab. 3]. The spreads overlap in five of six pairs, so the gains are not established. Zero-shot GPT-3.5 reaches 0.586 on the same mode task [arxiv:2312.00819 tab. 3]. Not evidence for larger training sets | | Task is sentence similarity (STS-like) with no tuning | Test before use | Contrastive pre-training trails SimCSE on STS [arxiv:2201.10005 tab. 4]. TSDAE trails CT by 7.9 points on STS-B [arxiv:2104.06979 tab. 3] | Notes that apply to every row: - Test on your own task. The sources report checkpoints chosen for search or classification, and their results may not transfer [arxiv:2201.10005 §3.4.2]. - Budget for training cost. Every AugSBERT silver pair needs a cross-encoder forward pass during training, but serving speed matches SBERT [arxiv:2010.08240 §5]. - Some papers report only a subset of their hyperparameters. The contrastive-pretraining paper gives no optimiser, learning rate or steps [arxiv:2201.10005 §2.2]. The E5 paper gives no data-construction code, and its repository holds evaluation code only [arxiv:2212.03533 tab. 11]. ## Implementation sketch The common core is an encoder, a cosine similarity, and a pair loss. The variants differ in how pairs and targets are built. TSDAE and Llama2Vec do not fit this template directly; their own objectives are sketched after it. ```pseudocode # Shared core: embed both sides with one encoder, compare with cosine [arxiv:2201.10005 §2.1] # Input: pairs (text_a, text_b, target); target is 1.0, 0.0 or a soft score # Output: fine-tuned encoder def train_bi_encoder(pairs, encoder, loss_fn): # Batches of pairs; in-batch negatives are optional [arxiv:2201.10005 §2.2] for batch in batches(pairs, size=16): # Embed both sides with the shared encoder [arxiv:2209.11055 §3.1] emb_a = encoder.encode([a for a, _, _ in batch]) emb_b = encoder.encode([b for _, b, _ in batch]) # Cosine similarity pulled toward the target [arxiv:2209.11055 §3.1] loss = loss_fn(cosine(emb_a, emb_b), [t for _, _, t in batch]) encoder.update(loss) return encoder ``` Variant-specific pair construction: ```pseudocode # Contrastive pre-training: diagonal of the batch is positive [arxiv:2201.10005 §2.2] def in_batch_targets(M): # Other pairs in the batch act as negatives; no labels needed return range(M) # SetFit: same label is positive (1.0), different label negative [arxiv:2209.11055 §3.1] def setfit_pairs(texts, labels): # Each labelled example can be paired with many others return [(a, b, float(la == lb)) for a, la in zip(texts, labels) for b, lb in zip(texts, labels)] # AugSBERT: cross-encoder scores become soft targets for unlabelled pairs [arxiv:2010.08240 §3.1] def silver_pairs(candidates, cross_encoder): # Keep candidates chosen by BM25 top-k rather than random pairs [arxiv:2010.08240 §6.1] return [(a, b, cross_encoder.score(a, b)) for a, b in candidates] # E5-style weak pairs: keep a pair only if its passage ranks in the top two [arxiv:2212.03533 §3] def consistency_filter(pairs, rank_of_passage, pool, k=2): # Rank each passage against a pool of random passages; drop weak pairs return [(q, p) for q, p in pairs if rank_of_passage(q, p, pool) <= k] ``` Objectives that are not pair-based: ```pseudocode # TSDAE: rebuild the sentence from a damaged copy through one vector [arxiv:2104.06979 §3.1] [§3.2] def tsdae_loss(encoder, decoder, sentence_ids, ratio=0.6): # Drop each token with probability ratio [arxiv:2104.06979 app. A] damaged = [t for t in sentence_ids if random() >= ratio] # The decoder sees only this one vector, as memory of length 1 [arxiv:2104.06979 §3.2] vec = encoder(damaged)[:, None, :] # Teacher forcing: predict each token of the original [arxiv:2104.06979 §3.1] return cross_entropy(decoder(sentence_ids[:-1], vec), sentence_ids[1:]) ``` Scoring with several heads at once: ```pseudocode # M3 hybrid score: weighted sum of the three head scores [arxiv:2402.03216 eq. 1] def hybrid_score(dense_score, lex_score, multi_score, w_dense, w_lex, w_multi): # Weights are set per benchmark [arxiv:2402.03216 §3.2] # (1, 0.3, 1) is the MIRACL "All" setting [arxiv:2402.03216 §4.1] return w_dense * dense_score + w_lex * lex_score + w_multi * multi_score ``` ```pseudocode # Llama2Vec adaptation: two prompt-end states from one pass [arxiv:2312.15503 §3.2] def llama2vec_adapt_loss(text_ids, next_ids, lm_head, self_block, next_block): # Joint prompt; the two blocks may not attend to each other [arxiv:2312.15503 fig. 2] hidden = llm(text_ids, self_block, next_block, mask=blocked_pairs) # Embedding per block: the state at each block's end [arxiv:2312.15503 §3.2] e_self, e_next = hidden[end_self], hidden[end_next] # Each block predicts its target tokens; how the terms combine is not stated return token_nll(lm_head(e_self), text_ids) + token_nll(lm_head(e_next), next_ids) ``` ```pseudocode # E5-Mistral query format: task instruction on queries only [arxiv:2401.00368 eq. 1] def format_query(task, query): # Documents are encoded with no prefix, so the index is never rebuilt [arxiv:2401.00368 §3.2] return "Instruct: " + task + "\nQuery: " + query ``` ```pseudocode # Sparse-latent document vector: max over tokens of each latent [arxiv:2512.10092 §3] def sae_doc_vector(acts): # acts: (tokens, latents), after the SAE nonlinearity; one value per latent # A latent that fires anywhere in the text shows up in the vector return acts.max(axis=0) ``` ```pseudocode # Single-direction score: coordinate of a phrase on PC1 [arxiv:2103.11790 eq. 1] def direction_score(phrase_vec, direction): # Plain projection as in the equation; the released code first subtracts the PCA mean # Sign and scale are set separately; see the card's Pitfalls return dot(phrase_vec, direction) ``` ## Contradictions The papers do not contradict each other's headline results, since they cover different tasks. Discrepancies inside single papers, or between a paper and its repository, that affect reuse: - **SetFit learning rate.** The paper states 1e-3 for the body [arxiv:2209.11055 §4.4]. The official code's default is 2e-5 [arxiv:2209.11055 §4.4]. The paper gives no reason for the gap. - **Contrastive loss and NQ gain.** The paper's prose and its listing disagree on whether the two directional losses are summed or averaged [arxiv:2201.10005 §2.2]. The NQ relative gain in the text cannot be reproduced from the printed table [arxiv:2201.10005 tab. 5]. - **AugSBERT gap.** The text gives a range of gains over the SBERT baseline that the table does not reproduce [arxiv:2010.08240 tab. 5]. - **E5-large size.** Section 5.4 gives 300M parameters, Table 10 gives 330M [arxiv:2212.03533 §5.4] [arxiv:2212.03533 tab. 10]. Use the table value. - **E5-base BEIR average.** Table 2 gives 48.7 [arxiv:2212.03533 tab. 2]. The repository's README gives 48.8. The difference is probably a later snapshot, but this cannot be settled from the repository. - **TSDAE deletion rate.** The paper's ratio 0.6 means 60 percent of tokens are affected [arxiv:2104.06979 app. A]. In the repository, the STS-B and file-based examples keep each word with probability 0.6, so they delete about 40 percent. The README's setting does delete 60 percent. See the [TSDAE](/papers/2104.06979) card's "Paper vs code" section. - **Sent2Vec L1 variants.** The paper reports small gains from an L1 penalty, at 82.1 and 83.4 supervised averages [arxiv:1703.02507 tab. 6]. The released code has no L1 step, so those rows cannot be reproduced from it. See the [Sent2Vec](/papers/1703.02507) card's "Paper vs code" section. - **Distillation teacher and schedule.** The paper trains for up to 20 epochs with a BERT-based NLI and STS teacher [arxiv:2004.09813 §3]. The example script uses a DistilRoBERTa paraphrase teacher and 5 epochs. The [Multilingual distillation](/papers/2004.09813) card's "Paper vs code" section lists these and related gaps. - **TSDAE gain over CT.** The text says 2.6 points on average [arxiv:2104.06979 §6]. Table 2 gives 55.2 against 52.4, a gap of 2.8 [arxiv:2104.06979 tab. 2]. - **ConveRT hardware.** Section 3 describes 12 GPU nodes with one K80 each [arxiv:1911.03688 §3]. Section 4 describes 12 T4 GPUs [arxiv:1911.03688 §4]. - **ConveRT size.** Table 2 gives 59 MB after quantization. The stated 8-bit and 16-bit widths applied to its parameter counts give about 42 MB [arxiv:1911.03688 tab. 2]. - **Llama2Vec table averages.** The printed Ada* average in Table 5 is 57.4, and the N2S average is 56.1. The per-dataset values in the same table average to 56.4 and 55.0 [arxiv:2312.15503 tab. 5] [arxiv:2312.15503 tab. 3]. - **Llama2Vec equation 5.** Equation 5 puts the query in the SELF slot and the answer in the NEXT slot, the reverse of the text [arxiv:2312.15503 §3.2]. Follow the text, which the released code also does. - **Llama2Vec read-out.** The paper reads a single special-token state per block [arxiv:2312.15503 §3.2]. The released code and README use the mean of eight special-token states at inference and in fine-tuning. See the [Llama2Vec](/papers/2312.15503) card's "Paper vs code" section. - **Word vectors versus TF-IDF.** The survey notes that pre-trained GloVe, trained on Wikipedia, can underperform TF-IDF on short messages [arxiv:1904.08067 §6]. The sentiment comparison finds word vectors ahead of TF-IDF for the CNN on all eight tweet and review datasets [arxiv:2006.03541 tab. 2]. The settings differ: the survey reports no experiments, and the comparison's vector source is itself unclear [arxiv:2006.03541 §4.4]. Not resolved here. - **Word-vector initialisation in the sentiment comparison.** One passage says Word2Vec trains the initial vectors. Another says the embedding layer starts from random weights and learns during training. Figure 9 marks the embedding parameters as non-trainable. The paper does not say which setup produced its results [arxiv:2006.03541 §4.4] [arxiv:2006.03541 fig. 9]. See the card's Pitfalls section. - **SBERT triplet margin.** The paper sets the margin to 1 [arxiv:1908.10084 §3]. The official library's default is 5, and its Wikipedia-sections example does not override it [arxiv:1908.10084 §3]. See the [Sentence-BERT](/papers/1908.10084) card's "Paper vs code" section. - **SBERT optimizer.** The paper says Adam [arxiv:1908.10084 §3.1]. The library's `fit` default is AdamW with weight decay 0.01 [arxiv:1908.10084 §3.1]. - **M3 temperature.** The paper does not state the InfoNCE temperature [arxiv:2402.03216 eq. 2]. The research code defaults to 0.02, and the newer package's model class defaults to 1 [arxiv:2402.03216 eq. 2]. - **M3 multi-vector scope.** The paper says the multi-vector head uses the entire output embeddings [arxiv:2402.03216 §3.2]. The code drops the first position, which is [CLS] [arxiv:2402.03216 §3.2]. The [M3-Embedding](/papers/2402.03216) card's "Paper vs code" section lists this and related gaps. - **Moral direction verb count.** The main text says 54 verbs enter the PCA [arxiv:2103.11790 §Methods: The MoralDirection of language models]. The supplement lists 64 [arxiv:2103.11790 suppl. B.2], and the released code uses that list. The paper does not reconcile the two. - **Moral filter minimum.** The experiments keep at least 5 candidates [arxiv:2103.11790 §Experimental setup]. The released class defaults to 30. See the [Large Pre-trained Language Models](/papers/2103.11790) card's "Paper vs code" section. - **Zero-shot BEIR numbers.** Llama2Vec's 56.4 is a zero-shot average [arxiv:2312.15503 tab. 3]. E5-Mistral's 56.9 uses labelled training data as well as synthetic data [arxiv:2401.00368 tab. 3]. The two are not directly comparable as zero-shot results. ## Papers - [Text and Code Embeddings by Contrastive Pre-Training](/papers/2201.10005): unsupervised contrastive pre-training of text and code encoders on paired web and code data. - [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533): E5, contrastive pre-training on about 270M filtered web pairs with a 32k batch, then fine-tuning. - [Sentence-BERT](/papers/1908.10084): siamese BERT with mean pooling, trained on NLI, STS or triplets, for cosine comparison of sentences. - [ConveRT](/papers/1911.03688): compact Reddit-pretrained dual encoder for reply selection, with quantization-aware training. - [Sentiment Analysis Based on Deep Learning: A Comparative Study](/papers/2006.03541): compares a DNN, CNN and LSTM for sentiment on eight datasets, with learned word vectors or TF-IDF as input. - [TSDAE](/papers/2104.06979): unsupervised sentence encoder trained by denoising reconstruction through a single sentence vector. - [Augmented SBERT](/papers/2010.08240): labels new pairs with a cross-encoder to train a bi-encoder; pair selection drives the gain. - [Efficient Few-Shot Learning Without Prompts](/papers/2209.11055): contrastive fine-tuning of a sentence encoder with a logistic-regression head for few-shot classification. - [Llama2Vec](/papers/2312.15503): adapts LLaMA-2-7B into a dense retriever without labels, with two prompts read from one pass, then LoRA fine-tuning on MS MARCO. - [M3-Embedding](/papers/2402.03216): multilingual encoder with dense, sparse and multi-vector heads, self-distillation and inputs up to 8,192 tokens. - [Improving Text Embeddings with Large Language Models](/papers/2401.00368): an LLM writes about 500k training examples in 93 languages; Mistral-7B is tuned with contrastive loss on them and labelled data. - [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092): document vectors of named SAE concept latents, max-pooled over tokens, for diffing, clustering and retrieval. - [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](/papers/2103.11790): a moral direction, the first principal component of verb embeddings, used as a score and as a decoding filter. - [Large Language Models for Travel Behavior Prediction](/papers/2312.00819): frozen LLM embeddings of rendered trip records fed to MNL, RF or NN, compared with zero-shot prompting at 10 and 1,000 samples. - [Text Classification Algorithms: A Survey](/papers/1904.08067): surveys word embeddings (Word2Vec, GloVe, FastText) as classifier features beside bag-of-words and TF-IDF, with qualitative comparisons only. - [How LLMs Distort Our Written Language](/papers/2603.18161): uses embeddings only as a measurement tool for semantic shift in LLM-edited text. - [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](/papers/1703.02507): Sent2Vec, a sentence vector as the mean of unigram and bigram vectors, trained without labels. - [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](/papers/2004.09813): a multilingual XLM-R student is trained to match an English teacher's vectors on parallel sentences.