M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

In plain words

Search systems find documents in a few different ways. One way compares the meaning of a query and a document as single vectors. Another matches the exact words they share, weighted by how important each word is. A third compares every word of the query with every word of the document. Most embedding models support only one of these, and most were built for English and short texts.

M3-Embedding is one model that supports all three ways. It handles text in more than 100 languages and inputs of up to 8,192 tokens, and it produces the three kinds of scores from the same pass over the text. During training, the combined score acts as a teacher for the three scoring modes, so each mode learns from the others. The authors also changed how training batches are formed, so long texts fit in memory and batches can still be large.

Why it matters: one model can serve several retrieval styles, and the card's results section shows how each style does on multilingual, cross-lingual and long-document benchmarks.

TL;DR

M3-Embedding is a multilingual text encoder with three retrieval heads: a dense vector, learned sparse token weights, and per-token vectors for late interaction. Its hybrid score is a weighted sum of the three. Training combines a contrastive loss per head with a self-distillation term whose teacher is the weighted sum. The paper reports the best averages on MIRACL, MKQA and MLDR when all three heads are combined, with the numbers author-reported and on the paper's own splits.

Problem

Most text embedding models were tailored for English, leaving few options for other languages. Each model usually supports one retrieval function, while retrieval systems often combine several. Long-document retrieval is hard to train because the training cost grows with input length, and most embedding models accept only short inputs §1.

Method

M3-Embedding combines three properties: many working languages, three retrieval functions on one encoder, and inputs from sentences up to 8,192 tokens §3. The backbone is an XLM-RoBERTa model further pre-trained with RetroMAE, with the maximum position extended to 8,192 §3.3 app. B.1.

1. Data. Three sources: unsupervised pairs mined from multilingual corpora (about 1.2 billion pairs over 194 languages, plus 2,655 cross-lingual correspondences from translation corpora); labelled fine-tuning data for English, Chinese and other languages; and synthetic long-document pairs (MultiLongDoc, 41.4K training examples) made by asking GPT-3.5 for one question per sampled passage §3.1 tab. 8 app. A.2 tab. 7.

2. Three heads on one encoder. Dense: the normalized [CLS] state. Sparse: a learned linear map to one non-negative weight per token, with the maximum weight kept for repeated terms. Multi-vector: a learned projection of every token, normalized §3.2.

3. Hybrid score. Candidates are re-ranked by a weighted sum of the three relevance scores §3.2 eq. 1.

   PseudocodeCopy

   # Hybrid re-ranking score  [eq. 1]
   def hybrid_score(dense_score, lex_score, multi_score, weights):
       # The three weights depend on the benchmark  [§3.2] [§4.1]
       w_dense, w_lex, w_multi = weights
       return (w_dense * dense_score
               + w_lex * lex_score
               + w_multi * multi_score)

4. Three heads in code form. The encoder output gives all three representations in one pass §3.2.

   PseudocodeCopy

   # Three retrieval heads on one encoder  [§3.2]
   # Input:  token_ids (L,), hidden (L, d) from the encoder
   # Output: dense (d,), sparse (V,), multi (L, d)
   
   def encode_three_modes(token_ids, hidden):
       # Dense: normalized [CLS] state  [§3.2]
       dense = normalize(hidden[0])
       # Sparse: ReLU token weight, max over repeats of a term  [§3.2]
       weights = relu(hidden @ w_lex)                 # (L,)  w_lex: (d,)
       sparse = zeros(V)
       for position in range(L):
           term = token_ids[position]
           sparse[term] = max(sparse[term], weights[position])
       # Multi-vector: normalized projection of every token  [§3.2]
       multi = normalize(hidden @ W_mul)              # (L, d)  W_mul: (d, d)
       return dense, sparse, multi

5. Self-knowledge distillation loss. Each head gets an InfoNCE loss eq. 2. The integrated score is the teacher eq. 3. A joint loss averages the per-head losses with weights eq. 4. The distillation losses eq. 5 are averaged with weights eq. 6. The final loss is the mean of the joint loss and the distillation loss §3.3.

   PseudocodeCopy

   # Self-knowledge distillation loss  [§3.3]  [eq. 2-6]
   # Scores are (B, B*N); targets hold each query's positive index
   def m3_loss(dense, lex, multi, targets, step, self_distill_start):
       # Integrated score, weights (1, 0.3, 1)  [eq. 3]
       inter = dense + 0.3 * lex + multi
       # Per-head InfoNCE losses, over the passages in the batch  [eq. 2]
       loss_dense = cross_entropy(dense, targets)
       loss_lex = cross_entropy(lex, targets)
       loss_multi = cross_entropy(multi, targets)
       loss_inter = cross_entropy(inter, targets)
       # Joint loss with lambda = (1, 0.1, 1), averaged over 4  [eq. 4]
       loss = (loss_dense + 0.1 * loss_lex + loss_multi + loss_inter) / 4
       # Warm-up: no distillation term yet  [app. B.1]
       if step <= self_distill_start:
           return loss
       # Teacher: softmax of the detached integrated score  [eq. 5]
       teacher = softmax(stop_gradient(inter))
       # Soft-target cross-entropy per head, lambda-weighted  [eq. 5] [eq. 6]
       distill_dense = soft_cross_entropy(teacher, dense)
       distill_lex = soft_cross_entropy(teacher, lex)
       distill_multi = soft_cross_entropy(teacher, multi)
       distill = (distill_dense + 0.1 * distill_lex + distill_multi) / 3
       # Final loss: mean of the joint and distilled losses  [§3.3]
       return (loss + distill) / 2

6. Multi-stage training. Stage 1: RetroMAE pre-training of XLM-RoBERTa on Pile, Wudao and mC4 (184 million samples in 105 languages) app. B.1. Stage 2: unsupervised contrastive learning on the dense head only §3.3 [app. B.1] fig. 2. Stage 3: fine-tuning all three heads with self-distillation, after an initial warm-up of about 6,000 steps on the three heads §3.3 app. B.1.

7. Efficient batching. Training data is grouped by sequence length, and each mini-batch is drawn from one group to reduce padding §3.4 fig. 3. A long batch is encoded in sub-batches with gradient checkpointing, so the intermediate activations are dropped and the embeddings are kept §3.4 [alg. 1]. Embeddings are then shared across GPUs to enlarge the in-batch negatives §3.4 fig. 3.

   PseudocodeCopy

   # Split-batch encoding with gradient checkpointing  [alg. 1]  [app. B.3]
   # Without checkpointing, sub-batch activations stay in memory  [app. B.3]
   model.enable_gradient_checkpointing()
   
   all_embeddings = []
   for batch in loader:
       # sub_batch_size is not stated in the paper
       for sub_batch in split(batch, sub_batch_size):
           all_embeddings.append(model(sub_batch))
   
   # Join the sub-batch outputs in order  [alg. 1]
   embeddings = concatenate(all_embeddings)

8. MCLS (no training on long text). At inference only, insert a [CLS] token every 256 tokens and average the [CLS] states into one vector [app. B.2] §3.4.

   PseudocodeCopy

   # MCLS: long-document vector, inference only  [app. B.2]
   def mcls_embedding(token_ids):
       # Insert a [CLS] token every 256 tokens; placement not stated  [app. B.2]
       chunked_ids, cls_positions = insert_cls_every(token_ids, 256)
       hidden = model(chunked_ids)
       # Average the [CLS] states into one vector  [app. B.2]
       return mean(hidden[cls_positions])

Figures: fig. 1 is captioned "Characters of M3-Embedding" and is not described further here. fig. 2 shows the multi-stage training process §3.3. fig. 3 shows length-grouped sampling, gradient checkpointing and cross-GPU broadcasting. fig. 4 shows the language and length distribution of the unsupervised data app. A.1. fig. 5 shows NarrativeQA results across sequence lengths app. C.1.

Hyperparameters

Values in the paper's appendix are reproduced as stated. "not stated" means the paper does not give the value.

Name	Value	Where	Notes
Pre-training base	XLM-RoBERTa (footnote links the large checkpoint)	§3.3 app. B.1	RetroMAE applied; max position 8,192
RetroMAE data	Pile, Wudao, mC4; 184M samples; 105 languages	app. B.1
RetroMAE max sequence length	8,192	app. B.1
RetroMAE learning rate	7e-5	app. B.1
RetroMAE batch	32, gradient accumulation over 16 steps	app. B.1
RetroMAE steps and hardware	20,000 steps on 32 A100 (40 GB)	app. B.1
Unsupervised max length, query / passage	512 / 8,192	app. B.1
Unsupervised learning rate	5e-5	app. B.1
Unsupervised warm-up ratio	0.1	app. B.1
Unsupervised weight decay	0.01	app. B.1
Unsupervised steps	25,000	app. B.1
Unsupervised batch	per length range, unsupervised column	tab. 9	Hardware: 96 A800 (80 GB) app. B.1
Unsupervised negatives	in-batch; embeddings broadcast across GPUs	§3.4
Fine-tuning group size	1 positive and 7 negatives per query	app. B.1	Negatives are hard negatives from ANCE §3.3
Fine-tuning batch	per length range, fine-tuning column	tab. 9	Hardware: 24 A800 (80 GB) app. B.1
Fine-tuning warm-up before distillation	about 6,000 steps on dense, sparse and multi-vector	app. B.1	Distillation starts after this phase §3.3
Training score weights	w1 = 1, w2 = 0.3, w3 = 1	§3.3	Used in the integrated score eq. 3
Training loss weights	lambda1 = 1, lambda2 = 0.1, lambda3 = 1	§3.3	Used in eq. 4 and eq. 6
Fine-tuning learning rate	not stated	app. B.1
Fine-tuning optimizer	not stated	app. B.1
Fine-tuning max length, query / passage	not stated	app. B.1	Table 9 length buckets reach 7,000 to 8,192 tokens tab. 9
Fine-tuning total steps or epochs	not stated	app. B.1
InfoNCE temperature tau	not stated	eq. 2	Code default 0.02, see Paper vs code
Shuffle augmentation	applied to long passages, probability written as 0.2%	app. A.1	Splits text into three segments and shuffles them
Gradient checkpointing and sub-batches	on, sub-batches for long inputs	§3.4 alg. 1	Sub-batch size not stated
Retrieval candidates, MIRACL	dense top-1000 (Faiss), sparse top-1000 (Lucene)	§4.1	Multi-vector re-ranks dense top-200
Score weights, MIRACL	Dense+Sparse (1, 0.3, 0); All (1, 0.3, 1)	§4.1	Dense+Sparse re-ranks the union of dense and sparse top-1000
Score weights, MKQA	same as MIRACL	§4.2
Score weights, MLDR	Dense+Sparse (0.2, 0.8, 0); All (0.15, 0.5, 0.35)	§4.3
MCLS	[CLS] every 256 tokens; mean of [CLS] states	app. B.2	Placement inside each block not stated
Projection sizes	W_lex in R^(d x 1); W_mul in R^(d x d)	§3.2	d is the hidden size

Results

Author-reported. Benchmarks and setup: MIRACL dev set, 18 languages, nDCG@10 §4.1 tab. 1; MKQA, 25 languages, passages from the BEIR NQ corpus, Recall@100 §4.2 tab. 2; MLDR test, 13 languages, nDCG@10 §4.3 tab. 3; NarrativeQA, English, nDCG@10 §4.3 tab. 4. The numbers below are copied from the paper's tables and were not re-run.

Headline averages (paper's best baseline versus M3 Dense and M3 All):

Benchmark	Metric	Best baseline (average)	M3 Dense	M3 All	Source
MIRACL dev, 18 languages	nDCG@10	mE5-large 66.6	69.2	71.5	tab. 1
MKQA, 25 languages	Recall@100	mE5-large 70.9	75.1	75.5	tab. 2
MLDR test, 13 languages, up to 8,192 tokens	nDCG@10	E5-mistral-7b 42.6	52.5	65.0	tab. 3
NarrativeQA, English	nDCG@10	text-embedding-3-large 51.6	48.7	61.7	tab. 4

Notes from the paper's text:

• On MIRACL, M3 Dense already beats the baselines on the average and in most languages. Sparse beats BM25 in every language, and multi-vector adds to dense. Dense+Sparse beats each single head, and All is the best row §4.1.
• On MKQA the gaps are smaller than on MIRACL. The sparse head is weak here because a query and its passage share few terms when they are in different languages §4.2.
• On MLDR, sparse is about 10 points above dense (62.2 vs 52.5), and multi-vector is 5.1 points above dense §4.3 tab. 3.
• Text-embedding-3-large is in the NarrativeQA table but not in the MLDR table tab. 3 tab. 4. On NarrativeQA, M3 Dense (48.7) trails it (51.6), while All (61.7) leads tab. 4.
• Ablation, self-knowledge distillation (MIRACL nDCG@10 averages): with it, Dense 69.2, Sparse 53.9, Multi-vector 70.5; without it, Dense 68.7, Sparse 36.7, Multi-vector 69.3 tab. 5 tab. 14.
• Ablation, training stages (MIRACL dense nDCG@10): fine-tune only 60.5; RetroMAE then fine-tune 66.1; RetroMAE, unsupervised, then fine-tune 69.2 tab. 6 tab. 15.
• MCLS without long-text training: Dense 41.2 becomes 45.0 on MLDR §4.3 tab. 3.

Table 1: MIRACL dev, nDCG@10 tab. 1

Model	Avg	ar	bn	en	es	fa	fi	fr	hi	id	ja	ko	ru	sw	te	th	zh	de	yo
BM25	31.9	39.5	48.2	26.7	7.7	28.7	45.8	11.5	35.0	29.7	31.2	37.1	25.6	35.1	38.3	49.1	17.5	12.0	56.1
mDPR	41.8	49.9	44.3	39.4	47.8	48.0	47.2	43.5	38.3	27.2	43.9	41.9	40.7	29.9	35.6	35.8	51.2	49.0	39.6
mContriever	43.1	52.5	50.1	36.4	41.8	21.5	60.2	31.4	28.6	39.2	42.4	48.3	39.1	56.0	52.8	51.7	41.0	40.8	41.5
mE5-large	66.6	76.0	75.9	52.9	52.9	59.0	77.8	54.5	62.0	52.9	70.6	66.5	67.4	74.9	84.6	80.2	56.0	56.4	78.3
E5-mistral-7b	63.4	73.3	70.3	57.3	52.2	52.1	74.7	55.2	52.1	52.7	66.8	61.8	67.7	68.4	73.9	74.0	54.0	54.1	79.7
OpenAI-3 (text-embedding-3-large)	54.9	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
M3 Dense	69.2	78.4	80.0	56.9	56.1	60.9	78.6	58.3	59.5	56.1	72.8	69.9	70.1	78.7	86.2	82.6	62.7	56.7	81.8
M3 Sparse	53.9	67.1	68.9	43.8	38.6	45.1	65.4	35.3	48.2	48.9	56.1	61.5	44.5	57.9	79.1	70.9	36.1	32.5	70.0
M3 Multi-vec	70.5	79.6	81.0	59.3	57.8	62.0	80.1	59.4	61.5	58.3	74.5	71.2	71.2	79.1	87.9	83.0	63.7	58.0	82.4
M3 Dense+Sparse	70.4	79.6	80.7	58.8	58.1	62.3	79.7	58.0	62.9	58.3	73.9	71.2	69.8	78.5	87.2	83.1	63.5	57.7	83.3
M3 All	71.5	80.2	81.5	59.6	59.7	63.4	80.4	61.2	63.3	59.0	75.2	72.1	71.7	79.6	88.1	83.7	64.9	59.8	83.5

Table 2: MKQA cross-lingual, Recall@100 tab. 2

Language	BM25	mDPR	mContriever	mE5-large	E5-mistral-7b	OpenAI-3	M3 Dense	M3 Sparse	M3 Multi-vec	M3 Dense+Sparse	M3 All
ar	18.9	48.2	58.2	68.7	59.6	65.6	71.1	23.5	71.4	71.1	71.5
da	49.3	67.4	73.9	77.4	77.8	73.6	77.2	55.4	77.5	77.4	77.6
de	35.4	65.8	71.7	76.9	77.0	73.6	76.2	43.3	76.3	76.4	76.3
es	43.4	66.8	72.6	76.4	77.4	73.9	76.4	50.6	76.6	76.7	76.9
fi	46.3	56.2	70.2	74.0	72.0	72.7	75.1	51.1	75.3	75.3	75.5
fr	45.3	68.2	72.8	75.5	78.0	74.1	76.2	53.9	76.4	76.6	76.6
he	26.9	49.7	63.8	69.6	47.2	58.1	72.4	31.1	72.9	72.5	73.0
hu	38.2	60.4	69.7	74.7	75.0	71.2	74.7	44.6	74.6	74.9	75.0
it	45.2	66.0	72.3	76.8	77.1	73.6	76.0	52.5	76.4	76.3	76.5
ja	24.5	60.3	64.8	71.5	65.1	71.9	75.0	31.3	75.1	75.0	75.2
km	27.8	29.5	26.8	28.1	34.3	33.9	68.6	30.1	69.1	68.8	69.2
ko	27.9	50.9	59.7	68.1	59.4	63.9	71.6	31.4	71.7	71.6	71.8
ms	55.9	65.5	74.1	76.3	77.2	73.3	77.2	62.4	77.4	77.4	77.4
nl	56.2	68.2	73.7	77.8	79.1	74.2	77.4	62.4	77.6	77.7	77.6
no	52.1	66.7	73.5	77.3	76.6	73.3	77.1	57.9	77.2	77.4	77.3
pl	40.8	63.3	71.6	76.7	77.1	72.7	76.3	46.1	76.5	76.3	76.6
pt	44.9	65.5	72.0	73.5	77.5	73.7	76.3	50.9	76.4	76.5	76.4
ru	33.2	62.7	69.8	76.8	75.5	72.0	76.2	36.9	76.4	76.2	76.5
sv	54.6	66.9	73.2	77.6	78.3	74.0	76.9	59.6	77.2	77.4	77.4
th	37.8	53.8	66.9	76.0	67.4	65.2	76.4	42.0	76.5	76.5	76.6
tr	45.8	59.1	71.1	74.3	73.0	71.8	75.6	51.8	75.9	76.0	76.0
vi	46.6	63.4	70.9	75.4	70.9	71.1	76.6	51.8	76.7	76.8	76.9
zh_cn	31.0	63.7	68.1	56.6	69.3	70.7	74.6	35.4	74.9	74.7	75.0
zh_hk	35.0	62.8	68.0	58.1	65.1	69.6	73.8	39.8	74.1	74.0	74.3
zh_tw	33.5	64.0	67.9	58.1	65.8	69.7	73.5	37.7	73.5	73.6	73.6
Avg	39.9	60.6	67.9	70.9	70.1	69.5	75.1	45.3	75.3	75.3	75.5

Table 3: MLDR test, nDCG@10 tab. 3

Model	Max length	Avg	ar	de	en	es	fr	hi	it	ja	ko	pt	ru	th	zh
BM25	8192	53.6	45.1	52.6	57.0	78.0	75.7	43.7	70.9	36.2	25.7	82.6	61.3	33.6	34.6
mDPR	512	23.5	15.6	17.1	23.9	34.1	39.6	14.6	35.4	23.7	16.5	43.3	28.8	3.4	9.5
mContriever	512	31.0	25.4	24.2	28.7	44.6	50.3	17.2	43.2	27.3	23.6	56.6	37.7	9.0	15.3
mE5-large	512	34.2	33.0	26.9	33.0	51.1	49.5	21.0	43.1	29.9	27.1	58.7	42.4	15.9	13.2
E5-mistral-7b	8192	42.6	29.6	40.6	43.3	70.2	60.5	23.2	55.3	41.6	32.7	69.5	52.4	18.2	16.8
text-embedding-ada-002	8191	32.5	16.3	34.4	38.7	59.8	53.9	8.0	46.5	28.6	20.7	60.6	34.8	9.0	11.2
Jina-embeddings-v2-base-en	8192	-	-	-	37.0	-	-	-	-	-	-	-	-	-	-
M3 Dense	8192	52.5	47.6	46.1	48.9	74.8	73.8	40.7	62.7	50.9	42.9	74.4	59.5	33.6	26.0
M3 Sparse	8192	62.2	58.7	53.0	62.1	87.4	82.7	49.6	74.7	53.9	47.9	85.2	72.9	40.3	40.5
M3 Multi-vec	8192	57.6	56.6	50.4	55.8	79.5	77.2	46.6	66.8	52.8	48.8	77.5	64.2	39.4	32.7
M3 Dense+Sparse	8192	64.8	63.0	56.4	64.2	88.7	84.2	52.3	75.8	58.5	53.1	86.0	75.6	42.9	42.0
M3 All	8192	65.0	64.7	57.9	63.8	86.8	83.9	52.2	75.5	60.1	55.7	85.4	73.8	44.7	40.0
M3 Dense, no long-doc fine-tuning	8192	41.2	35.4	35.2	37.5	64.0	59.3	28.8	53.1	41.7	29.8	63.5	51.1	19.5	16.5
M3 Dense, no long-doc fine-tuning, MCLS	8192	45.0	37.9	43.3	41.2	67.7	64.6	32.0	55.8	43.4	33.1	67.8	52.8	27.2	18.2

Table 4: NarrativeQA, nDCG@10 tab. 4

Model	Max length	nDCG@10
mDPR	512	16.3
mContriever	512	23.3
mE5-large	512	24.2
E5-mistral-7b	8192	49.9
text-embedding-ada-002	8191	41.1
text-embedding-3-large	8191	51.6
Jina-embeddings-v2-base-en	8192	39.4
M3 Dense	8192	48.7
M3 Sparse	8192	57.5
M3 Multi-vec	8192	55.4
M3 Dense+Sparse	8192	60.1
M3 All	8192	61.7

Table 5: self-knowledge distillation ablation, MIRACL dev, nDCG@10 tab. 5

Model	Dense	Sparse	Multi-vec
M3 with self-knowledge distillation	69.2	53.9	70.5
M3 without self-knowledge distillation	68.7	36.7	69.3

Table 6: training-stage ablation, MIRACL dev, dense nDCG@10 tab. 6

Training	MIRACL
Fine-tune	60.5
RetroMAE + Fine-tune	66.1
RetroMAE + Unsup + Fine-tune	69.2

Table 7: MultiLongDoc specification tab. 7

Language	Source	Train	Dev	Test	Corpus	Avg. doc length (tokens)
ar	Wikipedia	1,817	200	200	7,607	9,428
de	Wikipedia, mC4	1,847	200	200	10,000	9,039
en	Wikipedia	10,000	200	800	200,000	3,308
es	Wikipedia, mC4	2,254	200	200	9,551	8,771
fr	Wikipedia	1,608	200	200	10,000	9,659
hi	Wikipedia	1,618	200	200	3,806	5,555
it	Wikipedia	2,151	200	200	10,000	9,195
ja	Wikipedia	2,262	200	200	10,000	9,297
ko	Wikipedia	2,198	200	200	6,176	7,832
pt	Wikipedia	1,845	200	200	6,569	7,922
ru	Wikipedia	1,864	200	200	10,000	9,723
th	mC4	1,970	200	200	10,000	8,089
zh	Wikipedia, Wudao	10,000	200	800	200,000	4,249
Total	--	41,434	2,600	3,800	493,709	4,737

Table 8: training data sizes tab. 8

Data source	Language	Size
MTP	EN, ZH	291.1M
S2ORC, Wikipedia	EN	48.3M
xP3, mC4, CC-News	Multi-lingual	488.4M
NLLB, CCMatrix	Cross-lingual	391.3M
CodeSearchNet	Text-code	344.1K
Total unsupervised	--	1.2B
MS MARCO, HotpotQA, NQ, NLI, etc.	EN	1.1M
DuReader, T2-Ranking, NLI-zh, etc.	ZH	386.6K
MIRACL, Mr.TyDi	Multi-lingual	88.9K
MultiLongDoc	Multi-lingual	41.4K

Table 9: total batch size by sequence length tab. 9

Length range (tokens)	Unsupervised	Fine-tuning
0-500	67,200	1,152
500-1000	54,720	768
1000-2000	37,248	480
2000-3000	27,648	432
3000-4000	21,504	336
4000-5000	17,280	336
5000-6000	15,072	288
6000-7000	12,288	240
7000-8192	9,984	192

Table 10: maximum batch size per device, split-batch on or off tab. 10

Split-batch	1024	4096	8192
Off	262	25	6
On	855	258	130

Table 11: BM25 with different tokenizers tab. 11

Method	Tokenizer	MIRACL	MKQA	MLDR
BM25	Lucene analyzer	38.5	40.9	64.1
BM25	XLM-R	31.9	39.9	53.6
M3 Sparse	XLM-R	53.9	45.3	62.2
M3 All	XLM-R	71.5	75.5	65.0

Appendix tables with per-language recall and ablations are cited, not repeated: MIRACL Recall@100 tab. 12 (averages: BM25 67.3, mE5-large 94.1, M3 Dense 95.5, M3 All 96.4); MKQA Recall@20 tab. 13 (averages: BM25 28.1, mE5-large 63.5, M3 Dense 67.8, M3 All 68.8); full per-language ablations tab. 14 and tab. 15.

Paper vs code

Official repository: FlagOpen/FlagEmbedding, linked from the abstract's footnote. Its LICENSE is MIT, copyright holder staoxiao. Commit fd1a2bdf69488ffebe0327999d4400d8c8058a0b. The training code is in research/BGE_M3/, and a newer package version is in FlagEmbedding/finetune/embedder/encoder_only/m3/. Citations below are to the repo paths at that commit. Where the two versions differ, both are cited.

Agreements, checked against the paper:

• Dense head: normalized [CLS] state §3.2 modeling.py modeling.py.
• Sparse head: ReLU weight per token, max over repeats, dot product over shared terms §3.2 modeling.py modeling.py modeling.py.
• Loss mixing: integrated score with weights 1, 0.3, 1 §3.3 modeling.py and joint loss over 4 with lambda 0.1 on sparse §3.3 modeling.py. Distillation averaged over 3, then halved with the joint loss §3.3 modeling.py.
• Batch sizes in Table 9 equal per-device values in code times the GPU count: 48 × 24 = 1,152 for 0-500 fine-tuning, and 700 × 96 = 67,200 for 0-500 unsupervised tab. 9 data.py data.py. This is inferred from code and matches the GPU counts in app. B.1.
• Batches drawn from one length group §3.4 data.py.
• Seven negatives per query app. B.1 arguments.py (group size 8 = 1 positive + 7 negatives).

Discrepancies and gaps:

1. Multi-vector drops the first token. Paper: "entire output embeddings" §3.2. Code takes positions 1 onward, so [CLS] is excluded modeling.py and modeling.py. The score is averaged over the query positions after [CLS], which includes the final end token modeling.py.
2. Projections have biases. Paper's W_lex and W_mul are plain matrices §3.2. Code uses linear layers with bias modeling.py.
3. Special tokens are zeroed in the sparse head. Not stated in the paper. Code zeroes CLS, EOS, PAD and UNK modeling.py.
4. Warm-up before distillation. Paper: about 6,000 steps app. B.1. Code default is -1, which means no warm-up arguments.py. The example script sets 0 m3.sh. The gate is a strict comparison on the step counter modeling.py.
5. Teacher is detached. Not stated in the paper. Code stops gradients through the teacher modeling.py.
6. Distillation form. Paper's Eq. 5 is a soft cross-entropy eq. 5. The newer package names the same computation "kl_div" but computes minus the sum of teacher times log-softmax of the student, which is a soft cross-entropy AbsModeling.py. The teacher softmax runs over every passage in the batch, not over each query's group modeling.py. The paper does not state the softmax domain.
7. Temperature. Paper: tau is not stated eq. 2. Research code default is 0.02 arguments.py. The newer package's model class defaults to 1 modeling.py. The example script uses 0.02 m3.sh.
8. Unified fine-tuning is off by default. Paper's method trains all three heads. Both code bases default the flag to False arguments.py arguments.py.
9. Cross-device negatives are off by default. Paper: embeddings are broadcast to enlarge in-batch negatives §3.4. Default is off arguments.py. The example script turns it on m3.sh.
10. Negatives. Paper: ANCE hard negatives §3.3. The training loader samples negatives at random from the list given in the data data.py. The repository has a hard-negative mining script, scripts/hn_mine.py, which I did not audit.
11. External teacher path. The code can take teacher scores from a reranker modeling.py data.py. The paper's teacher is the internal integrated score §3.3. The example script turns this path off m3.sh.
12. Sub-batch sizes. Paper gives no sub-batch size. It reports maximum batch per device with and without split-batch tab. 10 and a gain of over 20 times at 8,192 tokens §3.4. Code maps input length to a sub-batch size modeling.py.
13. Shuffle augmentation. Paper: probability written as "0.2%" app. A.1. Code default is 0.0 arguments.py. The three- segment shuffle matches data.py.
14. Score weights. Paper: weights are per benchmark §3.2 §4.1 §4.3. The README example uses [0.4, 0.2, 0.4] README.md. This is a different setting, not a contradiction.
15. MCLS has no code. A search of the repository for "mcls", "multi_cls" and "multiple cls" found nothing, so the MCLS method is not verifiable here.
16. Learning rate, step count and maximum lengths for fine-tuning are not in the paper. The example script uses learning rate 1e-5 and 512-token limits m3.sh m3.sh. The script's own comments mark its batch size, epochs and GPU count as test settings m3.sh.
17. Evaluation pipeline. The paper's evaluation (Pyserini, Faiss, Lucene, re-ranking) is not audited here; the claims rest on the paper's tables.

No official code for MCLS or the paper's evaluation runs was found.

Pitfalls

• Temperature is not in the paper. The code uses 0.02 and the newer model class uses 1. Set it explicitly and record the value you used.
• Fine-tuning hyperparameters are missing. The paper gives no fine-tuning learning rate, optimizer, step count or maximum lengths app. B.1. Table 9's length buckets reach 8,192 tokens, so long passages were trained on, but the limit used is not stated.
• Batch sizes are global totals. Table 9 values need 24 GPUs (fine-tuning) or 96 GPUs (unsupervised). On fewer GPUs, scale per-device batches to match the totals you want, and note the effect on in-batch negatives.
• "Second stage" is used for two different runs. §3.3 calls the self-distillation fine-tuning the second stage, while app. B.1 uses the same phrase for the 96-GPU unsupervised run. Table 9's unsupervised column matches the 96-GPU count, so the appendix's "second stage" is the unsupervised run.
• Training memory for sparse. The training path builds a (batch, length, vocabulary) tensor and takes a max over length modeling.py. The inference path uses a scatter-reduce instead modeling.py. The vocabulary size is the model config value and is not stated in the paper.
• Multi-vector cost. Each token keeps a vector of the projection size, which is the hidden size unless colbert_dim is set modeling.py. The paper uses multi-vector only as a re-ranker over dense top-200 on MIRACL, citing its cost §3.2 §4.1. The storage cost is not discussed.
• Flags that change the method. Left at defaults, the research code trains dense only (unified_finetuning False), keeps self-distillation off, has no warm-up if it is enabled (self_distill_start_step -1), and uses no cross-device negatives. Set all four explicitly.
• Do not feed external teacher scores. Keep the teacher-score path off to match the paper's internal teacher modeling.py.
• Language coverage is inconsistent. The abstract says more than 100 languages [abstract]. The data section says 194 languages §3.1. The pre-training sample covers 105 languages app. B.1. The paper says per-language variation was not analysed [Limitations].
• Compare BM25 on the same tokenizer. BM25 with the Lucene analyzer scores higher than BM25 on the XLM-R tokenizer (MIRACL 38.5 vs 31.9) tab. 11. The paper's main BM25 row uses the XLM-R tokenizer §4.1.
• Sparse is weak across languages. On MKQA, sparse matches only the words shared by query and passage, which are few across languages §4.2.
• Hybrid weights are task-specific. Use the paper's per-benchmark weights as starting points, not as defaults §3.2 §4.1 §4.3.
• Synthetic data source. MultiLongDoc questions come from GPT-3.5 §3.1 app. A.2. The paper does not discuss the terms of use for that output.
• Author-reported results on the paper's splits. The MIRACL table is the dev set tab. 1. The NarrativeQA result is English only §4.3.

Lineage

• Builds on: XLM-RoBERTa for the encoder and RetroMAE for pre-training §3.3 app. B.1. Multi-vector scoring follows ColBERT §3.2. Hard negatives follow ANCE §3.3. The distillation target is a soft label from the model's own heads §3.3.
• Compared against: BM25, mDPR, mContriever, mE5-large, E5-mistral-7b, text-embedding-3-large (OpenAI-3), text-embedding-ada-002 and Jina-v2-base-en §4 tab. 1 tab. 2 tab. 3 tab. 4.
• Superseded by: not known from content already in this wiki.

Related

ConceptText embeddingFixed-size vectors for text or code, so similar inputs sit close. Variants differ by training signal: weak pairs, denoising, silver labels, teacher distillation, few-shot tuning, n-gram averaging, LLM-written data or an unlabelled LLM. Frozen embeddings can feed a classical classifier when labels are scarce. Sparse concept latents are interpretable. Pick by labels, compute and serving budget.ConceptDense retrievalRanks documents by one query vector against precomputed document vectors, a nearest-neighbour lookup. Variants differ in what they train on or generate first: labelled search data, LLM-written queries or tasks, unlabelled LLM adaptation, conversation pairs, multilingual data, a generator trained jointly, LM-scored examples or reasoning text. Pick by labels, generation budget and serving size.ConceptKnowledge distillationA student is trained to match a teacher's outputs, not only labels. Targets can be class probabilities, stored top-K probabilities, feature maps, pairwise relations, EMA-teacher patch tokens, density at 3D points, few-step sampler outputs or written rationales. Trade-off: a teacher pass or stored targets, often an extra training run, and gains that depend on teacher and benchmark.ConceptLearned sparse retrievalScores a query against a document with learned, non-negative weights on vocabulary terms, so only shared terms count. One card so far: M3-Embedding's sparse head, trained with its dense and multi-vector heads. Strong on long and English documents in the paper, weak across languages, and it needs self-distillation to work well.Paper · 2026-05-11Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and WhyTraining-free diagnostic that scores, per token and branch node, whether a GKD distillation gradient points toward success. The ideal gradient comes from targeted student rollouts; the cosine alignment is higher on failing rollouts, and the best teacher context depends on student size and task. Author-reported on Qwen3-0.6B/1.7B (BoolQ, MMLU, AIME 2025).Paper · 2026-05-08KL for a KL: On-Policy Distillation with Control Variate BaselineSubtracts a detached baseline from each token's teacher-student log-ratio reward in on-policy distillation. The baseline is the per-token negative reverse KL, taken from the existing forward pass: no critic, no extra rollouts, gradient still unbiased. A top-k baseline is cheaper. Author-reported: up to +3 average points over OPD on Qwen3-1.7B, matching full-vocabulary OPD.

Source and license

Adapted from “M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation” by Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu (arXiv:2402.03216), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/FlagOpen/FlagEmbedding at commit fd1a2bdf6948 (MIT). Any quoted code is shown under that license.