---
kind: concept
slug: learned-sparse-retrieval
title: Learned sparse retrieval
summary: >-
  Scores a query against a document with learned, non-negative weights on vocabulary terms, so only shared terms count. One card so far: M3-Embedding's sparse head, trained with its dense and multi-vector heads. Strong on long and English documents in the paper, weak across languages, and it needs self-distillation to work well.
tags: [retrieval, embeddings, lexical-matching, multilingual]
papers: ["2402.03216"]
related: [dense-retrieval, text-embedding, knowledge-distillation, nearest-neighbor-search, retrieval-augmented-generation, multilingual-evaluation]
status: draft
updated_at: 2026-10-19
---

## Overview

A learned sparse retriever gives each query and each document a weight for
some vocabulary terms, and zero for the rest. The score adds up the products of
the weights for the terms both sides share [arxiv:2402.03216 §3.2]. Unlike a
dense vector, each non-zero entry names a term, so matching words stay visible
in the score. The weights are learned, not counted.

The invariant across variants is that every term weight is non-negative and
the score depends only on terms present on both sides. The wiki has one card
for this concept so far, the sparse head of M3-Embedding [arxiv:2402.03216 §3.2].
This page covers that variant. Other variants will be added when their cards
are compiled.

## Variants

### Sparse head on a shared multilingual encoder (M3-Embedding)

One XLM-RoBERTa-based encoder, extended to 8,192 tokens, has a sparse head. A
learned linear map gives one scalar per token, passed through a ReLU. A term
that appears more than once keeps its largest weight [arxiv:2402.03216 §3.2]
[arxiv:2402.03216 tab. 11]. The sparse head is trained together with a dense
head and a multi-vector head. The training target is the weighted sum of all
three scores, which the sparse head learns from through self-distillation
[arxiv:2402.03216 §3.3]. Card:
[M3-Embedding](/papers/2402.03216).

Reported results, author-reported, on the paper's own splits, sparse head
alone against the dense head:

- MIRACL dev, 18 languages, nDCG@10: sparse 53.9, dense 69.2, all three heads
  71.5 [arxiv:2402.03216 tab. 1]. The sparse head beats BM25 with the same
  XLM-R tokenizer in all 18 languages [arxiv:2402.03216 tab. 1] [arxiv:2402.03216 §4.1].
- MLDR test, 13 languages, documents up to 8,192 tokens, nDCG@10: sparse 62.2,
  dense 52.5, dense plus sparse 64.8, all three 65.0 [arxiv:2402.03216 tab. 3].
- NarrativeQA, English, nDCG@10: sparse 57.5, dense 48.7, the best listed
  baseline 51.6 [arxiv:2402.03216 tab. 4].
- MKQA, 25 languages, Recall@100: sparse 45.3, dense 75.1, all three 75.5
  [arxiv:2402.03216 tab. 2].

Points to check before you implement:

- The sparse head needs self-distillation. On MIRACL its average is 36.7
  without it and 53.9 with it, while the dense head barely moves (68.7 and 69.2)
  [arxiv:2402.03216 tab. 5].
- Use it inside a hybrid. In every benchmark table in the paper, sparse alone
  scores below the combined heads [arxiv:2402.03216 tab. 1] [arxiv:2402.03216 tab. 2]
  [arxiv:2402.03216 tab. 3] [arxiv:2402.03216 tab. 4].
- Hybrid weights are set per benchmark: (1, 0.3, 0) for dense plus sparse on
  MIRACL and (0.2, 0.8, 0) on MLDR [arxiv:2402.03216 §4.1] [arxiv:2402.03216 §4.3].
  Treat them as starting points [arxiv:2402.03216 §3.2].
- The training loss gives the sparse head weight 0.3 in the integrated score
  [arxiv:2402.03216 eq. 3] and 0.1 in the joint and distillation losses
  [arxiv:2402.03216 eq. 4] [arxiv:2402.03216 eq. 6].
- The paper gives no fine-tuning learning rate or maximum length, and the same
  holds for the sparse head [arxiv:2402.03216 app. B.1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Long documents, with query words that appear in the passage | Sparse head, combined with dense in a hybrid score | MLDR nDCG@10: sparse 62.2, dense 52.5, dense plus sparse 64.8 [arxiv:2402.03216 tab. 3] |
| English long-narrative retrieval | Sparse head in a hybrid, checked against the dense head | NarrativeQA nDCG@10: sparse 57.5, dense 48.7 [arxiv:2402.03216 tab. 4]; the paper reports English only [arxiv:2402.03216 §4.3] |
| Query and passage in different languages | Dense head, not sparse | MKQA Recall@100: sparse 45.3, dense 75.1 [arxiv:2402.03216 tab. 2]. The paper says query and passage share few terms across languages [arxiv:2402.03216 §4.2] |
| One stage of training without self-distillation | Do not use the sparse head as trained | MIRACL sparse 36.7 without self-distillation, 53.9 with [arxiv:2402.03216 tab. 5] |
| A sparse-only index with no dense index | Not supported by these results | Sparse alone is below the combined heads in every benchmark table [arxiv:2402.03216 tab. 1] [arxiv:2402.03216 tab. 3] |
| Tight training memory, large vocabulary | Measure memory before you commit | The research code builds a batch by length by vocabulary tensor in training [code: research/BGE_M3/modeling.py#L107-L113]. The vocabulary size is not stated in the paper. |

The card records no index size or serving latency for the sparse head, so
the cost side of this choice is not established by these results.

## Implementation sketch

The sparse core has two steps: encode the text into term weights, and score
two term-weight maps on their shared terms. The hybrid score adds the other
heads on top [arxiv:2402.03216 eq. 1].

```pseudocode
# Sparse encoding: one non-negative weight per token  [arxiv:2402.03216 §3.2]
# Input:  token_ids (L,), hidden (L, d) from the encoder, w_lex (d,)
# Output: sparse, a map from vocabulary term to weight

def encode_sparse(token_ids, hidden, w_lex):
    # Linear map to one scalar per token, then ReLU  [arxiv:2402.03216 §3.2]
    weights = relu(hidden @ w_lex)                 # (L,)
    sparse = {}
    for position, term in enumerate(token_ids):
        # Repeated terms keep the largest weight  [arxiv:2402.03216 §3.2]
        sparse[term] = max(sparse.get(term, 0.0), weights[position])
    # Special tokens: zeroed in the research code, not stated in the paper
    return sparse

# Score: sum of products over shared terms  [arxiv:2402.03216 §3.2]
def score_sparse(query_sparse, doc_sparse):
    return sum(w * doc_sparse[t]
               for t, w in query_sparse.items() if t in doc_sparse)

# Hybrid: weighted sum of the heads  [arxiv:2402.03216 eq. 1]
# MLDR setting puts 0.8 on sparse  [arxiv:2402.03216 §4.3]
```

## Contradictions

Only one card covers this concept, so no two papers disagree. Within the
paper, one tension stands. Table 3 lists BM25 at 53.6 on MLDR, with the XLM-R
tokenizer [arxiv:2402.03216 tab. 3]. Table 11 lists BM25 at 64.1 with the
Lucene analyzer, which is above the sparse head's 62.2 [arxiv:2402.03216 tab. 11].
The paper's main tables use the XLM-R BM25 row [arxiv:2402.03216 §4.1]. The
paper does not discuss the Lucene row against its own sparse head, so both
numbers are kept here.

## Papers

- [M3-Embedding](/papers/2402.03216): one encoder with dense, sparse and multi-vector heads, trained with self-distillation, for many languages and long documents.