Text embedding

Overview

A text embedding model encodes a string into a vector, usually compared with cosine similarity. The vectors then serve search, clustering, and light classification 2201.10005 §1. Most papers in this wiki share one design: a sentence encoder is run on each side of a pair, and training shapes the vector space so that the pairs the task cares about land close together. They differ in where the pairs come from and what the labels are. Some papers depart from pairs: TSDAE reconstructs a sentence from a damaged copy of itself 2104.06979 §3.1, Llama2Vec adapts a decoder LLM with two prompts 2312.15503 §3.2, and Sent2Vec predicts each word from the rest of its sentence 1703.02507 §2.1. A sentiment study uses word vectors only as the input layer of a supervised classifier, with no pair objective 2006.03541 §4.4. The multilingual distillation method does use translation pairs, but its target is a teacher's vector rather than a pair label 2004.09813 §2. Three papers go further. One has an LLM write its own training examples 2401.00368 §3.1. One replaces the dense vector with a sparse list of named concept latents 2512.10092 §3. One uses a fixed direction in a sentence-embedding space as a scalar score rather than as a search index 2103.11790 eq. 1.

The shared mechanics: a pooled encoder output per input, a similarity (usually cosine), and a loss that pulls positives together and pushes negatives apart 2201.10005 §2.1 2209.11055 §3.1.

Variants

Contrastive pre-training on naturally paired data

Pairs come for free from the web (neighbouring text) or from code (a docstring and its function). Negatives are the other pairs in the batch. The embedding is the last-layer state at an end delimiter 2201.10005 §2.1. The authors train text models from GPT-family checkpoints and code models from Codex checkpoints 2201.10005 §2.2. Card: Text and Code Embeddings by Contrastive Pre-Training.

Reported gains are largest on retrieval and classification. On a 7-task linear probe the largest text model reaches 91.8 against 88.3 for DECLUTR 2201.10005 tab. 2. On a sentence-similarity benchmark it reaches 71.1 against 79.8 for SimCSE 2201.10005 tab. 4.

Weakly supervised contrastive pre-training at scale (E5)

Pairs are mined from semi-structured sources: forum posts with replies, questions with answers, encyclopedia sections with their passages, paper titles with abstracts, and web title-passage pairs. A consistency filter then removes weak pairs. It trains a model on about 1.3 billion candidate pairs, ranks each passage against 1 million random passages, and keeps a pair only when its passage ranks in the top two. About 270 million pairs remain 2212.03533 §3. Training uses in-batch negatives, a batch of 32,768 and a temperature of 0.01 2212.03533 §4.1 2212.03533 tab. 11.

Batch size matters here. On base models the BEIR average falls from 51.6 at 32k to 50.2 at 8k and 45.8 at 1k 2212.03533 tab. 5. Unsupervised E5-PT-base averages 42.9 nDCG@10 on BEIR against 41.7 for BM25 2212.03533 tab. 1. A fine-tuning round on NLI, MS-MARCO and NQ, with hard negatives and teacher distillation, lifts the base model to 60.4 on the 56-dataset English MTEB average 2212.03533 tab. 3. The paper's pre-training took 16 to 64 V100 GPUs for 1 to 2 days 2212.03533 tab. 11. Card: Text Embeddings by Weakly-Supervised Contrastive Pre-training.

Supervised siamese fine-tuning on labelled pairs (SBERT)

A pretrained BERT or RoBERTa is fine-tuned with two weight-tied copies, one per sentence. Each side is mean-pooled into one vector, and the two vectors are compared with cosine similarity. Training uses one of three objectives: a three-way classifier over NLI pairs, applied to the two vectors and their absolute difference; a regression of the cosine onto a gold score; or a triplet hinge 1908.10084 §3. The classifier is used only during training and is dropped at inference 1908.10084 §6.

The main gain is speed. The authors estimate that the closest pair among 10,000 sentences takes about 65 hours with a BERT cross-encoder and about 5 seconds with SBERT 1908.10084 §1 1908.10084 §8. Accuracy is competitive but not uniformly higher. SBERT-NLI-large averages 76.55 Spearman over seven STS tasks without STS training data, against 71.22 for the Universal Sentence Encoder 1908.10084 tab. 1. When trained on the STS benchmark, the large BERT model scores 85.64 and the large SBERT model 84.45 1908.10084 tab. 2. Card: Sentence-BERT.

Small dual encoder for reply selection (ConveRT)

A context encoder and a response encoder share one small Transformer, and each side has its own feed-forward head. Outputs are L2-normalised and scored with a scaled cosine 1911.03688 §2.1. It is trained on Reddit (input, response) pairs with an in-batch softmax and label smoothing of 0.2 1911.03688 §2.1 1911.03688 §3. Compactness is the design goal: 512-dimensional embeddings, six blocks, a subword vocabulary of 31,476 tokens plus 1,000 hash buckets for unknown characters, and quantization-aware training with 8-bit embeddings and 16-bit other weights 1911.03688 §2.1. The authors report a 116 MB model before quantization, against 444 MB for PolyAI-dual 1911.03688 tab. 2.

On Reddit, the single-context model scores 68.2 R100@1 and the version that reads up to ten earlier messages scores 71.8, against 61.3 for PolyAI-dual 1911.03688 tab. 3. The authors also claim pre-training takes about 18 hours on 12 GPU nodes 1911.03688 §3. Card: ConveRT.

Denoising reconstruction on unlabelled sentences (TSDAE)

No pairs are needed. Each sentence has 60 percent of its tokens deleted 2104.06979 app. A 2104.06979 tab. 6. The encoder compresses the damaged sentence into one [CLS] vector, and a decoder that attends only to that vector must rebuild the original. Encoder and decoder share weights, except for the decoder's cross-attention 2104.06979 §3.1 2104.06979 §3.2 2104.06979 app. A. The decoder is discarded after training, and the encoder alone embeds text 2104.06979 §3.2.

The method is aimed at specialised domains. Its unsupervised average over four domain tasks is 55.2, against 52.4 for Contrastive Tension (CT) 2104.06979 tab. 2. On SciDocs it reaches 74.0 against 67.6 for CT 2104.06979 tab. 2. On the general STS benchmark it reaches 66.0 against 73.9 for CT 2104.06979 tab. 3. Card: TSDAE.

Averaged unigram and bigram vectors trained without pairs (Sent2Vec)

No encoder network is involved. Each sentence vector is the mean of learned vectors for its unigrams and bigrams 1703.02507 §2.1 [arxiv:1703.02507 eq. 2]. Training takes each word of a sentence as a target and predicts it from the rest of the sentence, with negative sampling and no fixed window 1703.02507 §2.1 1703.02507 §2.3. Bigrams are hashed into a fixed number of buckets 1703.02507 §2.2. The authors add a bigram dropout step during training, and a sentence has no dropout at inference 1703.02507 §2.4.

The paper reports the unigram+bigram model at 82.0 average accuracy on six supervised tasks, trained on Toronto books. SkipThought, which reads word order, scores 83.8 on the same table 1703.02507 tab. 1. On unsupervised STS 2014 and SICK 2014 the average Spearman and Pearson are 0.65 and 0.67 1703.02507 tab. 2. Training on the Toronto books corpus takes about 1.2 hours on one node with 30 threads 1703.02507 tab. 3. Card: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features.

The released trainer's defaults differ from the paper's table in learning rate, n-gram order and an L1 penalty, so the card's "Paper vs code" section lists the flags to set.

Bi-encoder trained with cross-encoder silver labels (Augmented SBERT)

A slow, accurate cross-encoder is fine-tuned on a small gold set. It then labels new sentence pairs, and the fast bi-encoder trains on gold plus these soft-labelled pairs 2010.08240 §3.1. The paper's experiments show that which pairs are chosen matters more than how many. BM25 retrieval gives most of the gain at the smallest silver set, and random pairs often hurt 2010.08240 §6.1 2010.08240 tab. 11. The same recipe can adapt to an unlabelled target domain with no target labels 2010.08240 §3.2. Card: Augmented SBERT.

Multilingual student distilled from an English teacher (multilingual distillation)

An English sentence encoder, already tuned on NLI and STS data, is the fixed teacher. A multilingual student, XLM-RoBERTa, is trained on parallel sentences. For each pair, the loss adds two squared distances to the teacher's English vector: one from the student's vector of the source and one from the student's vector of the translation 2004.09813 §2 2004.09813 §3. The student therefore places each language in the teacher's English space. The training data are OPUS corpora and bilingual dictionaries 2004.09813 §3.

Results are the authors' own. The paraphrase-trained teacher's student averages 84.6 on monolingual STS 2017 and 83.7 on cross-lingual STS 2017, against 75.4 and 67.0 for LASER and 76.4 and 73.5 for LaBSE 2004.09813 tab. 1 2004.09813 tab. 2. Its BUCC bitext mining F1 is 88.6, below LASER at 93.0 and LaBSE at 93.5 2004.09813 tab. 3. The authors attribute the gap to the method targeting similarity rather than exact translation [arxiv:2004.09813 §4.2]. On Tatoeba for four low-resource languages the NLI-trained student beats LASER in both directions 2004.09813 tab. 4. LASER was trained on Tatoeba, so that comparison may favour LASER's numbers 2004.09813 app. A. The data size matters: EN-DE reaches 71.5 with 1,000 TED2020 pairs and 80.4 with the full set 2004.09813 tab. 6. Card: Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation.

Few-shot contrastive fine-tuning with a linear head (SetFit)

Starting from a pretrained sentence encoder, the body is fine-tuned on pairs built from a handful of labelled examples. Same-label pairs are positive and different-label pairs are negative. A logistic-regression head is then trained on the tuned embeddings. No prompts are used 2209.11055 §3.1. Card: Efficient Few-Shot Learning Without Prompts.

Results are in the table under "When to use which". The 110M-parameter body is close to a 3B prompt-based model at 8 examples per class and ahead of it at 64 2209.11055 tab. 2.

LLM backbone adapted without labels (Llama2Vec)

The starting point is LLaMA-2-7B, a decoder-only model. Adaptation runs on Wikipedia passages. One forward pass reads the passage followed by two prompts: one asks the model to rebuild the passage, the other to predict the next one. Each prompt's end state is read as an embedding. A custom attention mask stops the two prompts from attending to each other 2312.15503 §3.2 2312.15503 fig. 2. Fine-tuning then uses contrastive learning with LoRA on MS MARCO 2312.15503 §4.1. The authors say one joint pass saves almost half the cost of two separate passes 2312.15503 §3.2.

Reported results: MS MARCO passage MRR@10 of 43.1, against 41.2 for RepLLaMA 2312.15503 tab. 1. Zero-shot BEIR average NDCG@10 of 56.4, against 53.9 for RepLLaMA 2312.15503 tab. 3. The released code differs from the paper in its read-out and its adaptation loss, so implement from the card's "Paper vs code" section, not from the paper alone. Card: Llama2Vec.

LLM writes its own training data, then tunes a 7B decoder (E5-Mistral)

A proprietary LLM is prompted to invent search-style tasks. For each of four length-based groups it first proposes about 20 one-sentence task definitions, then writes one JSON example per task, in 93 languages. The paper reports about 500k examples after filtering 2401.00368 §3.1 2401.00368 §4.1. Mistral-7B is then fine-tuned with LoRA at rank 16 for one epoch, using InfoNCE at a fixed temperature of 0.02 2401.00368 §3.2 2401.00368 §4.2 2401.00368 eq. 3. Only queries carry a task instruction; documents do not. The embedding is the last-layer state at the end-of-sequence token 2401.00368 §3.2. Removing the instruction costs 4.2 MTEB points 2401.00368 tab. 5.

Results are the authors' own. Trained on synthetic data alone, the model averages 63.1 on the 56-dataset English MTEB, against 62.3 for E5-large-v2 2401.00368 tab. 1. With the labelled data added, it reaches 66.6, against 64.2 for BGE-large-en-v1.5 2401.00368 tab. 1. It is weaker on low-resource languages: 68.4 on Swahili MIRACL dev, against 74.9 for mE5-large 2401.00368 tab. 2. The released repository holds only evaluation code, so the training recipe is paper-only. The paper does not discuss the terms of the generator's outputs, so check them before you reproduce the data pipeline. Card: Improving Text Embeddings with Large Language Models.

Multilingual multi-function encoder with long inputs (M3-Embedding)

One XLM-RoBERTa-based encoder yields three representations from one pass: a normalised [CLS] vector for dense search, a learned non-negative weight per token for lexical matching, and a normalised vector for every token for multi-vector scoring 2402.03216 §3.2. Candidates can be re-ranked by a weighted sum of the three scores, with weights set per benchmark 2402.03216 eq. 1 2402.03216 §3.2. Training has three stages: RetroMAE pre-training, unsupervised contrastive learning on the dense head, and fine-tuning of all three heads with self-distillation, where the combined score is the teacher 2402.03216 §3.3 2402.03216 app. B.1. Inputs of up to 8,192 tokens are encoded in sub-batches with gradient checkpointing 2402.03216 §3.4 2402.03216 alg. 1. An inference-only pooling over [CLS] tokens inserted every 256 tokens is also proposed for long text 2402.03216 app. B.2.

The gains are reported on the authors' own tables. On the MIRACL dev set the all-heads score averages 71.5 nDCG@10, against 66.6 for mE5-large 2402.03216 tab. 1. On MLDR, the MultiLongDoc test split, it reaches 65.0 against 42.6 for E5-mistral-7b 2402.03216 tab. 3. Self-distillation matters most for the sparse head: 53.9 with it and 36.7 without, on MIRACL 2402.03216 tab. 5. Card: M3-Embedding.

Pre-trained word vectors as classifier features (survey)

Not a training method. The survey lists word embeddings as one feature option for text classification, next to bag-of-words and TF-IDF. The embeddings it names are Word2Vec, GloVe and FastText, plus contextual representations 1904.08067 §2.4 1904.08067 tab. 1. Bag of words counts words over a fixed vocabulary and ignores order 1904.08067 §2.3. The survey reports no experiments. Its comparisons are qualitative, and it notes that pre-trained GloVe, trained on Wikipedia, can underperform TF-IDF on short messages 1904.08067 §1 1904.08067 §6. Card: Text Classification Algorithms: A Survey.

Word vectors inside a supervised sentiment classifier (comparative study)

Not a pair-based embedding method. Learned word vectors, width 300 over a vocabulary of 15,000 words and sequences of up to 40 tokens, form the input layer of a CNN or an LSTM trained on polarity labels. The paper compares that input against TF-IDF features 2006.03541 §4.3 2006.03541 §4.4. Evaluation uses 10-fold cross-validation over eight tweet and review datasets 2006.03541 §4.1 2006.03541 §4.2.

Word vectors beat TF-IDF for the CNN on all eight datasets. On Sentiment140 the CNN scores 0.8006 with word vectors and 0.7669 with TF-IDF, and on Cornell Movie Reviews 0.7137 against 0.6787 2006.03541 tab. 2. The LSTM with word vectors is the most accurate model on six of the eight datasets. It also trains about nine times longer than the CNN on Sentiment140 2006.03541 tab. 2 2006.03541 tab. 7. The authors report no baseline outside their three models and no variance across runs 2006.03541 §4.2. Card: Sentiment Analysis Based on Deep Learning: A Comparative Study.

Frozen embeddings of structured records as classifier features (travel behaviour)

Not a pair method. Each record is rendered as plain text, here a trip with travel times and costs for each mode and some traveller attributes. An LLM embedding of that text is computed once per record. A classical head is then fitted on the vectors: multinomial logit, random forest or a neural network 2312.00819 §3.4 2312.00819 §4.2. The encoder is not trained. The paper gives no settings for the heads, and its prose does not name the embedding model 2312.00819 §3.4 2312.00819 §3.2. The embedding dimension is not stated 2312.00819 §3.1. Card: Large Language Models for Travel Behavior Prediction.

Results are the authors' own, on a mode-choice task and a trip-purpose task, with 200-row balanced test sets averaged over five random samples 2312.00819 §4.1 2312.00819 tab. 3. Embedding heads are reported only with 10 training samples. Mode accuracy rises for each head: MNL from 0.469 to 0.494, RF from 0.455 to 0.509, NN from 0.462 to 0.532. On trip purpose, NN rises from 0.268 to 0.296 2312.00819 tab. 3. The ± spreads overlap in five of the six pairs. The exception is the RF trip-purpose baseline, whose spread is zero 2312.00819 tab. 3. The paper reports no significance tests 2312.00819 §4. At 1,000 samples, no embedding rows are reported, so the paper gives no evidence for that size 2312.00819 tab. 3.

Embeddings as a measurement tool (not a method)

A study of LLM-edited essays embeds drafts with off-the-shelf embedding models and projects them onto a two-dimensional PCA plane per condition 2603.18161 §3.2 2603.18161 §4.2. The paper contributes no embedding training. Card: How LLMs Distort Our Written Language. Do not copy its plane construction as a retrieval method. The paper does not state that the planes are comparable across rows, and the authors' code fits one plane per row 2603.18161 §3.2.

Sparse concept latents as an interpretable embedding (SAE)

Not a trained encoder. A reader LLM runs over the text, and a sparse autoencoder trained on one of its layers splits each hidden state into features. The paper uses Llama 3.3 70B at layer 50, with a public SAE of 65,536 latents 2512.10092 §4. A document's vector is the maximum activation of each latent over its tokens 2512.10092 §3. Each latent carries a short label, so every dimension can be read and compared 2512.10092 §3. The paper uses these vectors for dataset diffing, concept co-occurrence, clustering along chosen axes and property-based retrieval 2512.10092 §4. Its API accepts at most 2048 tokens, so longer documents need chunking, which the paper does not describe 2512.10092 §4.

On six retrieval datasets, the SAE ranking beats the OpenAI-plus-LLM baseline on four: responses 0.418 against 0.373 MAP, reasoning traces 0.409 against 0.333, biology abstracts 0.529 against 0.524 and short stories 0.468 against 0.406. It trails on prompts, 0.361 against 0.412, and on The Pile, 0.443 against 0.446 2512.10092 §4.4. The values are before LLM reranking. The temperature and phrase count were chosen by best average on the same queries that are reported, which may inflate the gain 2512.10092 §4.4. Cost is lower than pure-LLM hypothesis generation, by 2 to 8 times in the paper's own pipeline 2512.10092 §4.1. Card: Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit.

A single direction as a scorer (moral direction)

Not a search index. Each of 64 verbs is placed in ten question frames and embedded with BERT-large using SBERT pooling. The verb's ten vectors are averaged, and the first principal component of the verb vectors is taken as a direction 2103.11790 §Identifying the Moral Direction of Language Models 2103.11790 suppl. B.2. A phrase's score is its coordinate on that direction 2103.11790 eq. 1. Against human yes-rates, the authors report r = 0.78 and 0.79 2103.11790 §BERT's MoralDirection Strongly Correlates with Human Moral Norms. The score then filters candidate tokens during GPT-2 decoding. With threshold 0, 1.17% of prompts get a generation above toxicity 0.9, against 9.82% for GPT-2 2103.11790 §Reducing Neural Toxic Degeneration in Language Models. These are author-reported.

The correlation depends on the verb set. Three random sets of 64 verbs give r = 0.64, 0.60 and -0.01 2103.11790 suppl. B.2. The filter's threshold is tied to one normalising constant and one encoder. The paper gives no tuning procedure, so a new encoder or verb set means re-tuning it. See the card's Pitfalls. Card: Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do.

When to use which

Notes that apply to every row:

• Test on your own task. The sources report checkpoints chosen for search or classification, and their results may not transfer 2201.10005 §3.4.2.
• Budget for training cost. Every AugSBERT silver pair needs a cross-encoder forward pass during training, but serving speed matches SBERT 2010.08240 §5.
• Some papers report only a subset of their hyperparameters. The contrastive-pretraining paper gives no optimiser, learning rate or steps 2201.10005 §2.2. The E5 paper gives no data-construction code, and its repository holds evaluation code only 2212.03533 tab. 11.

Implementation sketch

The common core is an encoder, a cosine similarity, and a pair loss. The variants differ in how pairs and targets are built. TSDAE and Llama2Vec do not fit this template directly; their own objectives are sketched after it.

# Shared core: embed both sides with one encoder, compare with cosine  [arxiv:2201.10005 §2.1]
# Input:  pairs (text_a, text_b, target); target is 1.0, 0.0 or a soft score
# Output: fine-tuned encoder

def train_bi_encoder(pairs, encoder, loss_fn):
    # Batches of pairs; in-batch negatives are optional  [arxiv:2201.10005 §2.2]
    for batch in batches(pairs, size=16):
        # Embed both sides with the shared encoder  [arxiv:2209.11055 §3.1]
        emb_a = encoder.encode([a for a, _, _ in batch])
        emb_b = encoder.encode([b for _, b, _ in batch])
        # Cosine similarity pulled toward the target  [arxiv:2209.11055 §3.1]
        loss = loss_fn(cosine(emb_a, emb_b), [t for _, _, t in batch])
        encoder.update(loss)
    return encoder

Variant-specific pair construction:

# Contrastive pre-training: diagonal of the batch is positive  [arxiv:2201.10005 §2.2]
def in_batch_targets(M):
    # Other pairs in the batch act as negatives; no labels needed
    return range(M)

# SetFit: same label is positive (1.0), different label negative  [arxiv:2209.11055 §3.1]
def setfit_pairs(texts, labels):
    # Each labelled example can be paired with many others
    return [(a, b, float(la == lb)) for a, la in zip(texts, labels)
            for b, lb in zip(texts, labels)]

# AugSBERT: cross-encoder scores become soft targets for unlabelled pairs  [arxiv:2010.08240 §3.1]
def silver_pairs(candidates, cross_encoder):
    # Keep candidates chosen by BM25 top-k rather than random pairs  [arxiv:2010.08240 §6.1]
    return [(a, b, cross_encoder.score(a, b)) for a, b in candidates]

# E5-style weak pairs: keep a pair only if its passage ranks in the top two  [arxiv:2212.03533 §3]
def consistency_filter(pairs, rank_of_passage, pool, k=2):
    # Rank each passage against a pool of random passages; drop weak pairs
    return [(q, p) for q, p in pairs if rank_of_passage(q, p, pool) <= k]

Objectives that are not pair-based:

# TSDAE: rebuild the sentence from a damaged copy through one vector  [arxiv:2104.06979 §3.1] [§3.2]
def tsdae_loss(encoder, decoder, sentence_ids, ratio=0.6):
    # Drop each token with probability ratio  [arxiv:2104.06979 app. A]
    damaged = [t for t in sentence_ids if random() >= ratio]
    # The decoder sees only this one vector, as memory of length 1  [arxiv:2104.06979 §3.2]
    vec = encoder(damaged)[:, None, :]
    # Teacher forcing: predict each token of the original  [arxiv:2104.06979 §3.1]
    return cross_entropy(decoder(sentence_ids[:-1], vec), sentence_ids[1:])

Scoring with several heads at once:

# M3 hybrid score: weighted sum of the three head scores  [arxiv:2402.03216 eq. 1]
def hybrid_score(dense_score, lex_score, multi_score, w_dense, w_lex, w_multi):
    # Weights are set per benchmark  [arxiv:2402.03216 §3.2]
    # (1, 0.3, 1) is the MIRACL "All" setting  [arxiv:2402.03216 §4.1]
    return w_dense * dense_score + w_lex * lex_score + w_multi * multi_score

# Llama2Vec adaptation: two prompt-end states from one pass  [arxiv:2312.15503 §3.2]
def llama2vec_adapt_loss(text_ids, next_ids, lm_head, self_block, next_block):
    # Joint prompt; the two blocks may not attend to each other  [arxiv:2312.15503 fig. 2]
    hidden = llm(text_ids, self_block, next_block, mask=blocked_pairs)
    # Embedding per block: the state at each block's end  [arxiv:2312.15503 §3.2]
    e_self, e_next = hidden[end_self], hidden[end_next]
    # Each block predicts its target tokens; how the terms combine is not stated
    return token_nll(lm_head(e_self), text_ids) + token_nll(lm_head(e_next), next_ids)

# E5-Mistral query format: task instruction on queries only  [arxiv:2401.00368 eq. 1]
def format_query(task, query):
    # Documents are encoded with no prefix, so the index is never rebuilt  [arxiv:2401.00368 §3.2]
    return "Instruct: " + task + "\nQuery: " + query

# Sparse-latent document vector: max over tokens of each latent  [arxiv:2512.10092 §3]
def sae_doc_vector(acts):
    # acts: (tokens, latents), after the SAE nonlinearity; one value per latent
    # A latent that fires anywhere in the text shows up in the vector
    return acts.max(axis=0)

# Single-direction score: coordinate of a phrase on PC1  [arxiv:2103.11790 eq. 1]
def direction_score(phrase_vec, direction):
    # Plain projection as in the equation; the released code first subtracts the PCA mean
    # Sign and scale are set separately; see the card's Pitfalls
    return dot(phrase_vec, direction)

Contradictions

The papers do not contradict each other's headline results, since they cover different tasks. Discrepancies inside single papers, or between a paper and its repository, that affect reuse:

• SetFit learning rate. The paper states 1e-3 for the body 2209.11055 §4.4. The official code's default is 2e-5 2209.11055 §4.4. The paper gives no reason for the gap.
• Contrastive loss and NQ gain. The paper's prose and its listing disagree on whether the two directional losses are summed or averaged 2201.10005 §2.2. The NQ relative gain in the text cannot be reproduced from the printed table 2201.10005 tab. 5.
• AugSBERT gap. The text gives a range of gains over the SBERT baseline that the table does not reproduce 2010.08240 tab. 5.
• E5-large size. Section 5.4 gives 300M parameters, Table 10 gives 330M 2212.03533 §5.4 2212.03533 tab. 10. Use the table value.
• E5-base BEIR average. Table 2 gives 48.7 2212.03533 tab. 2. The repository's README gives 48.8. The difference is probably a later snapshot, but this cannot be settled from the repository.
• TSDAE deletion rate. The paper's ratio 0.6 means 60 percent of tokens are affected 2104.06979 app. A. In the repository, the STS-B and file-based examples keep each word with probability 0.6, so they delete about 40 percent. The README's setting does delete 60 percent. See the TSDAE card's "Paper vs code" section.
• Sent2Vec L1 variants. The paper reports small gains from an L1 penalty, at 82.1 and 83.4 supervised averages 1703.02507 tab. 6. The released code has no L1 step, so those rows cannot be reproduced from it. See the Sent2Vec card's "Paper vs code" section.
• Distillation teacher and schedule. The paper trains for up to 20 epochs with a BERT-based NLI and STS teacher 2004.09813 §3. The example script uses a DistilRoBERTa paraphrase teacher and 5 epochs. The Multilingual distillation card's "Paper vs code" section lists these and related gaps.
• TSDAE gain over CT. The text says 2.6 points on average 2104.06979 §6. Table 2 gives 55.2 against 52.4, a gap of 2.8 2104.06979 tab. 2.
• ConveRT hardware. Section 3 describes 12 GPU nodes with one K80 each 1911.03688 §3. Section 4 describes 12 T4 GPUs 1911.03688 §4.
• ConveRT size. Table 2 gives 59 MB after quantization. The stated 8-bit and 16-bit widths applied to its parameter counts give about 42 MB 1911.03688 tab. 2.
• Llama2Vec table averages. The printed Ada* average in Table 5 is 57.4, and the N2S average is 56.1. The per-dataset values in the same table average to 56.4 and 55.0 2312.15503 tab. 5 2312.15503 tab. 3.
• Llama2Vec equation 5. Equation 5 puts the query in the SELF slot and the answer in the NEXT slot, the reverse of the text 2312.15503 §3.2. Follow the text, which the released code also does.
• Llama2Vec read-out. The paper reads a single special-token state per block 2312.15503 §3.2. The released code and README use the mean of eight special-token states at inference and in fine-tuning. See the Llama2Vec card's "Paper vs code" section.
• Word vectors versus TF-IDF. The survey notes that pre-trained GloVe, trained on Wikipedia, can underperform TF-IDF on short messages 1904.08067 §6. The sentiment comparison finds word vectors ahead of TF-IDF for the CNN on all eight tweet and review datasets [arxiv:2006.03541 tab. 2]. The settings differ: the survey reports no experiments, and the comparison's vector source is itself unclear 2006.03541 §4.4. Not resolved here.
• Word-vector initialisation in the sentiment comparison. One passage says Word2Vec trains the initial vectors. Another says the embedding layer starts from random weights and learns during training. Figure 9 marks the embedding parameters as non-trainable. The paper does not say which setup produced its results 2006.03541 §4.4 2006.03541 fig. 9. See the card's Pitfalls section.
• SBERT triplet margin. The paper sets the margin to 1 1908.10084 §3. The official library's default is 5, and its Wikipedia-sections example does not override it 1908.10084 §3. See the Sentence-BERT card's "Paper vs code" section.
• SBERT optimizer. The paper says Adam 1908.10084 §3.1. The library's fit default is AdamW with weight decay 0.01 1908.10084 §3.1.
• M3 temperature. The paper does not state the InfoNCE temperature 2402.03216 eq. 2. The research code defaults to 0.02, and the newer package's model class defaults to 1 2402.03216 eq. 2.
• M3 multi-vector scope. The paper says the multi-vector head uses the entire output embeddings 2402.03216 §3.2. The code drops the first position, which is [CLS] 2402.03216 §3.2. The M3-Embedding card's "Paper vs code" section lists this and related gaps.
• Moral direction verb count. The main text says 54 verbs enter the PCA 2103.11790 §Methods: The MoralDirection of language models. The supplement lists 64 2103.11790 suppl. B.2, and the released code uses that list. The paper does not reconcile the two.
• Moral filter minimum. The experiments keep at least 5 candidates 2103.11790 §Experimental setup. The released class defaults to 30. See the Large Pre-trained Language Models card's "Paper vs code" section.
• Zero-shot BEIR numbers. Llama2Vec's 56.4 is a zero-shot average 2312.15503 tab. 3. E5-Mistral's 56.9 uses labelled training data as well as synthetic data 2401.00368 tab. 3. The two are not directly comparable as zero-shot results.

Papers

• Text and Code Embeddings by Contrastive Pre-Training: unsupervised contrastive pre-training of text and code encoders on paired web and code data.
• Text Embeddings by Weakly-Supervised Contrastive Pre-training: E5, contrastive pre-training on about 270M filtered web pairs with a 32k batch, then fine-tuning.
• Sentence-BERT: siamese BERT with mean pooling, trained on NLI, STS or triplets, for cosine comparison of sentences.
• ConveRT: compact Reddit-pretrained dual encoder for reply selection, with quantization-aware training.
• Sentiment Analysis Based on Deep Learning: A Comparative Study: compares a DNN, CNN and LSTM for sentiment on eight datasets, with learned word vectors or TF-IDF as input.
• TSDAE: unsupervised sentence encoder trained by denoising reconstruction through a single sentence vector.
• Augmented SBERT: labels new pairs with a cross-encoder to train a bi-encoder; pair selection drives the gain.
• Efficient Few-Shot Learning Without Prompts: contrastive fine-tuning of a sentence encoder with a logistic-regression head for few-shot classification.
• Llama2Vec: adapts LLaMA-2-7B into a dense retriever without labels, with two prompts read from one pass, then LoRA fine-tuning on MS MARCO.
• M3-Embedding: multilingual encoder with dense, sparse and multi-vector heads, self-distillation and inputs up to 8,192 tokens.
• Improving Text Embeddings with Large Language Models: an LLM writes about 500k training examples in 93 languages; Mistral-7B is tuned with contrastive loss on them and labelled data.
• Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit: document vectors of named SAE concept latents, max-pooled over tokens, for diffing, clustering and retrieval.
• Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do: a moral direction, the first principal component of verb embeddings, used as a score and as a decoding filter.
• Large Language Models for Travel Behavior Prediction: frozen LLM embeddings of rendered trip records fed to MNL, RF or NN, compared with zero-shot prompting at 10 and 1,000 samples.
• Text Classification Algorithms: A Survey: surveys word embeddings (Word2Vec, GloVe, FastText) as classifier features beside bag-of-words and TF-IDF, with qualitative comparisons only.
• How LLMs Distort Our Written Language: uses embeddings only as a measurement tool for semantic shift in LLM-edited text.
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features: Sent2Vec, a sentence vector as the mean of unigram and bigram vectors, trained without labels.
• Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation: a multilingual XLM-R student is trained to match an English teacher's vectors on parallel sentences.