Text Embeddings by Weakly-Supervised Contrastive Pre-training

In plain words

Search, clustering and "find similar things" all work better when every piece of text is turned into a list of numbers, and texts with similar meaning get similar lists. Older text models were built for other jobs and do this poorly without extra tuning. The usual fix is to train on pairs of texts that belong together, such as a question and its answer, but labelled pairs are scarce and pairs scraped automatically are often noisy.

This paper builds a large set of text pairs from forums, encyclopedias, papers and web pages, and throws out the weakest pairs with a cleaning step. A model is then trained to pull each pair together and push apart the other texts in the same training batch. The result is a general-purpose embedding model, called E5, that works without labels. A short extra training round on a few labelled datasets improves it further.

TL;DR

• E5 is a BERT-family encoder trained with a contrastive loss on CCPairs, a cleaned set of about 270M text pairs §3 tab. 9.
• Negatives come from the same batch, and the batch is large (32,768) §4.1 tab. 11.
• Author-reported: unsupervised E5-PT-base beats BM25 on the 15 public BEIR datasets (average nDCG@10 42.9 vs 41.7) tab. 1.
• Author-reported: fine-tuned E5-base averages 60.4 on the 56 English MTEB datasets, above GTR-xxl (59.0) and Sentence-T5-xxl (59.5) tab. 3.
• Pitfall: the code in the repository covers evaluation only. Pre-training, data construction and fine-tuning cannot be checked against code.

Problem

Pre-trained language models give token-level or sentence-level features, but they are not ideal for retrieval and matching, where one vector per text is wanted for speed and simplicity §1. Contrastive training works well for single-vector embeddings, but labelled pair data is small. Automatically built pairs (inverse cloze task, random cropping, neighbouring text spans) are plentiful but noisy, and according to the paper they do not match BM25 on BEIR without further fine-tuning §1 §2. The authors' premise is that data quality, not only quantity, decides the outcome §3 §4.

Method

Overview. The method has two stages, pre-training on unlabelled pairs and optional fine-tuning on labelled data, and the resulting embeddings are used without further training of the encoder for retrieval and classification §4 fig. 1.

Stage 0: build CCPairs §3.

1. Collect (query, passage) pairs from semi-structured sources: Reddit (post, comment), Stackexchange (question, upvoted answer), Wikipedia (entity name plus section title, passage), Scientific papers (title and abstract, citations), Common Crawl and news (title, passage) §3. Subsets from existing datasets are reused §3 app. A.
2. Apply heuristic filters: drop Reddit comments longer than 4096 characters or with score below 1, and drop Common Crawl passages with high perplexity §3. This leaves about 1.3 billion pairs, mostly from Reddit and Common Crawl §3.
3. Remove pairs that also occur in the evaluation sets, by exact string match app. A.
4. Consistency filter §3: train a model on the 1.3B pairs; rank each pair's passage against a pool of 1 million random passages; keep the pair only if its passage is in the top k = 2. About 270M pairs remain §3. The authors motivate this with the memorisation of clean labels before noisy ones §3.
5. S2ORC pairs are down-weighted to a sample weight of 0.3 to avoid over-fitting the scientific domain app. A.

Stage 1: contrastive pre-training §4.1.

• Bi-encoder: one shared pre-trained Transformer encodes queries and passages. Each text is average-pooled over its output tokens to get a fixed-size vector §4.1.
• Score: cosine similarity divided by a temperature τ = 0.01 §4.1 eq. 2.
• Loss: InfoNCE over one positive and the negatives §4.1 eq. 1.
• Negatives: the other passages in the batch (in-batch negatives). The authors report that this beat MoCo when the batch is large §4.1 tab. 8.
• Asymmetry: the prefix "query: " is added to queries and "passage: " to passages, so one encoder can serve both roles §4.1. For pairs where neither side is clearly the query (citation pairs), the side is chosen at random §4.1.

# Contrastive pre-training with in-batch negatives  [§4.1] [eq. 1] [eq. 2]
# Input:  B (query, passage) texts; B = 32768 in pre-training  [tab. 11]
# Output: scalar loss

def pretraining_loss(queries, passages, temperature):
    # Prefixes break the symmetry between the two roles  [§4.1]
    queries = ["query: " + text for text in queries]
    passages = ["passage: " + text for text in passages]
    # One shared encoder, average pooling over tokens  [§4.1]
    query_vecs = normalize(mean_pool(encoder(queries)))    # (B, d)
    passage_vecs = normalize(mean_pool(encoder(passages)))  # (B, d)
    # Cosine similarity divided by the temperature  [eq. 2]
    scores = query_vecs @ passage_vecs.T / temperature      # (B, B)
    # Other passages in the batch are the negatives; the positive
    # for query i is passage i, on the diagonal  [§4.1] [eq. 1]
    return cross_entropy(scores, targets=arange(B)).mean()

Stage 2: fine-tuning on labelled data §4.2.

• Datasets: NLI (in the SimCSE release), MS-MARCO passage ranking and Natural Questions (NQ), concatenated §4.2 §5.1.
• Hard negatives: MS-MARCO and NQ use mined hard negatives and cross-encoder scores reused from SimLM. NLI contradiction sentences serve as hard negatives §4.2 §5.1.
• Distillation: the student's probabilities are trained toward the cross-encoder teacher's probabilities with a KL term, plus α times the contrastive loss, with α = 0.2 §4.2 eq. 3 tab. 11.
• Per example: 7 hard negatives. NLI has only one hard negative per example, so 6 sentences are sampled at random from the corpus §5.1.

# Fine-tuning loss  [§4.2] [eq. 3]
# Input:  a query, its positive, hard negatives (7 per example)  [§5.1]
# Output: scalar loss

def finetuning_loss(query, positive, hard_negatives, teacher_probs):
    # Student probabilities over the candidates; the candidate set
    # for the KL term is not stated  [§4.2]
    student_probs = softmax(scores(query, [positive] + hard_negatives))
    # Distil the cross-encoder's soft labels  [§4.2] [eq. 3]
    distill = kl_divergence(teacher_probs, student_probs)
    # Contrastive loss on the hard labels, alpha = 0.2  [tab. 11]
    contrast = info_nce(query, positive, hard_negatives)
    return distill + alpha * contrast

Use as an embedding §4.3.

• Zero-shot retrieval: index passage embeddings offline, embed the query, and return the top-k by cosine similarity §4.3.
• Few-shot classification: train a linear classifier on frozen embeddings with a few labelled examples §4.3.
• Zero-shot classification: write each label as a sentence from a manual template and predict the nearest label embedding. The example in the paper uses "movie review: " as the input prefix and a "terrible/great movie review" label text §4.3.
• Semantic similarity: cosine similarity, evaluated by rank correlation §4.3.
• Clustering: standard algorithms such as k-means §4.3.
• For tasks other than zero-shot classification and retrieval, the query prefix embedding is used by default §4.3.

Hyperparameters

Values for the three sizes are listed in the order small / base / large. Model sizes are in tab. 10. The paper's text also gives these settings in §5.1 and app. B.

Results

Author-reported. Setup: BEIR (15 public datasets, nDCG@10) and MTEB (56 English datasets, mean over categories) §5.2. Emphasis marks from the paper are omitted; "-" means not reported.

Table 1: unsupervised, BEIR nDCG@10 tab. 1. cpt₃₀₀M is available only through a paid API and is missing some datasets. LaPraDor uses no BM25 ensemble tab. 1.

The paper states that E5-PT-base beats BM25 by 1.2 points on average, that this is the first unsupervised model to do so on BEIR, and that E5-PT-large reaches 44.2 §5.3.

Table 2: supervised fine-tuning, BEIR nDCG@10 tab. 2. ANCE, ColBERT and Contriever values come from Izacard et al. tab. 2. MS-MARCO and NQ are in-domain for the fine-tuned E5 models; the other datasets are zero-shot transfer §5.3.

Table 3: MTEB English, category means tab. 3. Glossary: Class. = Classification, Clust. = Clustering, PairClass. = Pair Classification, Retr. = Retrieval, Summ. = Summarization. Category counts: 12, 11, 3, 4, 15, 10, 1 (56 datasets). The caption refers readers to app. B for the detailed version, but in the LaTeX order that table is tab. 13 in app. C.

The paper states that E5-base (110M) is competitive with GTR-xxl and Sentence-T5-xxl (4.8B) and that E5-large is more than 10× smaller than those two §1 §5.4. It also says that the difference between BERT-FT-base and E5-base isolates the effect of contrastive pre-training on CCPairs §5.4.

Table 4: zero-shot text classification, SST-2 dev accuracy tab. 4.

Table 5: batch size, base models tab. 5.

Table 6: labelled fine-tuning mix, base models tab. 6.

Table 7: consistency filter, base models tab. 7. The top two rows train on 1M random pairs; the bottom two on all pairs.

Table 8: negative sampling, base models tab. 8.

Table 9: CCPairs sources after filtering tab. 9. The "random example" column of the paper is not reproduced here.

Table 10: model configurations tab. 10.

Table 11: hyperparameters tab. 11.

PT = contrastive pre-training (E5-PT); FT = fine-tuning (E5).

Table 12: in-domain results tab. 12. "Target pre-train" marks models that first receive extra pre-training on the target corpus, before supervised fine-tuning. The NQ columns follow DPR's passage retrieval setup. MS-MARCO results use RocketQA document titles app. B.

Table 13: per-dataset MTEB, non-retrieval tasks tab. 13. The 15 retrieval datasets are omitted here because they are the BEIR datasets in Tables 1 and 2.

Negative results app. C. Adding one BM25 hard negative per pair gave about 0.5 BEIR points on a 15M-pair set but was too slow at 250M+ pairs app. C. RoBERTa initialisation was worse than BERT on most BEIR datasets app. C. An auxiliary MLM loss on 25% of pairs gave results on par with no auxiliary loss at higher cost app. C.

Paper vs code

The official repository is microsoft/unilm, MIT-licensed, at commit 31c5b904ca1bf2afb4c234a6675c683a4e5fc7cd, with the E5 code in e5/. The e5/ folder contains evaluation code only: BEIR and MTEB runners, a pooling and prompt helper, and model-name mappings. It has no pre-training, CCPairs construction, consistency-filter or fine-tuning code, so the training stages below are not checked against code. The README also lists later E5 models (v2, multilingual, Mistral-based) README.md.

Checked and consistent:

• Average pooling for the E5 models §4.1 model_config.py utils.py.
• Prefixes "query: " and "passage: " for retrieval §4.1 app. B mteb_beir_eval.py mteb_beir_eval.py.
• Quora: "query: " on both sides app. B mteb_beir_eval.py.
• Evaluation maximum length 512 app. B utils.py.
• L2 normalisation of embeddings before scoring mteb_beir_eval.py mteb_except_retrieval_eval.py. The paper's cosine similarity is consistent with this.

Discrepancies:

• BEIR average for E5-base: README 48.8 README.md; Table 2 48.7 tab. 2 §5.3. The E5-small (46.0) and E5-large (50.0) values match. The README is a later snapshot, so the difference is probably a version difference. This cannot be settled from the code here.
• Titles: the BEIR evaluation prepends each corpus document's title to its text mteb_beir_eval.py. The paper says titles are used for MS-MARCO in-domain results only app. B and says nothing about BEIR corpus documents.
• Classification: for tasks typed "Classification", the code disables L2 normalisation mteb_except_retrieval_eval.py. The paper does not state this.
• Precision: the evaluation loads the model in float16 and encodes under autocast mteb_beir_eval.py mteb_beir_eval.py. The paper does not state evaluation precision.
• MTEB version: the code pins mteb==1.0.2 requirements.txt. The paper does not state the MTEB version it used.

No other discrepancies found in the evaluation code.

Pitfalls

• E5-large size. §5.4 says "300M parameters" for E5-large §5.4. Table 10 gives 330M tab. 10. The abstract's "40x more parameters" is the comparison of E5-base (110M) with the 4.8B GTR-xxl and Sentence-T5-xxl §1 §5.4. Use the table value.
• Equation 3 weighting. The text calls the fine-tuning loss a "linear interpolation" §4.2, but eq. 3 is written as D_KL + α·L_cont, with no weight on the KL term. Table 11 gives α = 0.2 tab. 11. Whether the KL term is weighted (and by what) is not stated. The code for fine-tuning is not in the repository.
• Teacher and candidates for distillation. The paper does not say how the cross-encoder scores are turned into probabilities, over which candidate set the KL is computed, or whether NLI has a teacher §4.2 §5.1. The text says NLI uses contradiction sentences as hard negatives §4.2.
• Consistency filter. The paper does not name the model used for the filter, or say whether the pair's own passage is in the 1M pool before ranking, or how the pool is sampled §3. The value k = 2 comes from "manual inspection" §3, and iterated filtering is left for future work §3.
• Warmup and decay. The schedule is "linear decay" with warmup §5.1. The decay horizon, final learning rate, Adam betas and epsilon, and dropout are not stated.
• Cross-device negatives. The batch of 32,768 is spread across 16 to 64 GPUs §5.1 tab. 11. Whether negatives are gathered across devices is not stated. The 32k count of negatives holds only if they are.
• Truncation and positions. Pre-training uses a maximum length of 128 tokens tab. 11, fine-tuning 192 tab. 11, and evaluation 512 app. B. Position embeddings are frozen during training, which is what makes the longer evaluation length possible app. B. The paper does not say how many texts exceed the training limits.
• Quality filters are coarse. The Reddit length and score rules, the Common Crawl perplexity cut-off, and the exact contamination check §3 app. A are described qualitatively. Contamination removal is by exact string match only app. A, so near-duplicates may remain.
• MS-MARCO titles. BEIR MS-MARCO has no titles, so its results are expected to be lower than the in-domain numbers, which use RocketQA titles app. B tab. 12.
• In-domain vs zero-shot in averages. In the fine-tuned BEIR average tab. 2, MS-MARCO and NQ are in-domain for E5 §5.3. Compare averages only with that in mind.
• Compute. Pre-training took 1 to 2 days on 16 to 64 V100 GPUs §5.1 tab. 11. CCPairs construction (about 1.3B pairs, then 270M kept) is a large data pipeline the paper describes only in outline §3.
• Reproducing MTEB numbers. Results depend on the MTEB version and on the prompt and normalisation choices above. The paper's table caption refers to app. B for the detailed MTEB table, but that table sits in app. C tab. 3 tab. 13.
• Figure. Figure 1 gives the data pipeline and architecture in one picture fig. 1. This card describes it in words only.

Lineage

• Builds on: InfoNCE loss and in-batch negatives, following SimCLR [Chen et al. 2020] §4.1. Consistency filtering follows Promptagator §3. Hard negatives and cross-encoder scores are reused from SimLM §4.2 §5.1. The backbone is MiniLM for the small model and BERT for the base and large ones §5.1.
• Compared against: BM25 tab. 1; SimCSE, LaPraDor, Contriever and cpt-300M tab. 1; ANCE, ColBERT and GTR tab. 2; GTR-large, Sentence-T5 and BERT-FT tab. 3; pre-batch negatives and MoCo tab. 8; RocketQAv2 and SimLM in-domain tab. 12.
• Superseded by: not stated in this paper. The code README lists later E5 variants (see Paper vs code).