Learned sparse retrieval

Overview

A learned sparse retriever gives each query and each document a weight for some vocabulary terms, and zero for the rest. The score adds up the products of the weights for the terms both sides share 2402.03216 §3.2. Unlike a dense vector, each non-zero entry names a term, so matching words stay visible in the score. The weights are learned, not counted.

The invariant across variants is that every term weight is non-negative and the score depends only on terms present on both sides. The wiki has one card for this concept so far, the sparse head of M3-Embedding 2402.03216 §3.2. This page covers that variant. Other variants will be added when their cards are compiled.

Variants

Sparse head on a shared multilingual encoder (M3-Embedding)

One XLM-RoBERTa-based encoder, extended to 8,192 tokens, has a sparse head. A learned linear map gives one scalar per token, passed through a ReLU. A term that appears more than once keeps its largest weight 2402.03216 §3.2 2402.03216 tab. 11. The sparse head is trained together with a dense head and a multi-vector head. The training target is the weighted sum of all three scores, which the sparse head learns from through self-distillation 2402.03216 §3.3. Card: M3-Embedding.

Reported results, author-reported, on the paper's own splits, sparse head alone against the dense head:

• MIRACL dev, 18 languages, nDCG@10: sparse 53.9, dense 69.2, all three heads 71.5 2402.03216 tab. 1. The sparse head beats BM25 with the same XLM-R tokenizer in all 18 languages 2402.03216 tab. 1 2402.03216 §4.1.
• MLDR test, 13 languages, documents up to 8,192 tokens, nDCG@10: sparse 62.2, dense 52.5, dense plus sparse 64.8, all three 65.0 2402.03216 tab. 3.
• NarrativeQA, English, nDCG@10: sparse 57.5, dense 48.7, the best listed baseline 51.6 2402.03216 tab. 4.
• MKQA, 25 languages, Recall@100: sparse 45.3, dense 75.1, all three 75.5 2402.03216 tab. 2.

Points to check before you implement:

• The sparse head needs self-distillation. On MIRACL its average is 36.7 without it and 53.9 with it, while the dense head barely moves (68.7 and 69.2) 2402.03216 tab. 5.
• Use it inside a hybrid. In every benchmark table in the paper, sparse alone scores below the combined heads 2402.03216 tab. 1 2402.03216 tab. 2 2402.03216 tab. 3 2402.03216 tab. 4.
• Hybrid weights are set per benchmark: (1, 0.3, 0) for dense plus sparse on MIRACL and (0.2, 0.8, 0) on MLDR 2402.03216 §4.1 2402.03216 §4.3. Treat them as starting points 2402.03216 §3.2.
• The training loss gives the sparse head weight 0.3 in the integrated score 2402.03216 eq. 3 and 0.1 in the joint and distillation losses 2402.03216 eq. 4 2402.03216 eq. 6.
• The paper gives no fine-tuning learning rate or maximum length, and the same holds for the sparse head 2402.03216 app. B.1.

When to use which

The card records no index size or serving latency for the sparse head, so the cost side of this choice is not established by these results.

Implementation sketch

The sparse core has two steps: encode the text into term weights, and score two term-weight maps on their shared terms. The hybrid score adds the other heads on top 2402.03216 eq. 1.

# Sparse encoding: one non-negative weight per token  [arxiv:2402.03216 §3.2]
# Input:  token_ids (L,), hidden (L, d) from the encoder, w_lex (d,)
# Output: sparse, a map from vocabulary term to weight

def encode_sparse(token_ids, hidden, w_lex):
    # Linear map to one scalar per token, then ReLU  [arxiv:2402.03216 §3.2]
    weights = relu(hidden @ w_lex)                 # (L,)
    sparse = {}
    for position, term in enumerate(token_ids):
        # Repeated terms keep the largest weight  [arxiv:2402.03216 §3.2]
        sparse[term] = max(sparse.get(term, 0.0), weights[position])
    # Special tokens: zeroed in the research code, not stated in the paper
    return sparse

# Score: sum of products over shared terms  [arxiv:2402.03216 §3.2]
def score_sparse(query_sparse, doc_sparse):
    return sum(w * doc_sparse[t]
               for t, w in query_sparse.items() if t in doc_sparse)

# Hybrid: weighted sum of the heads  [arxiv:2402.03216 eq. 1]
# MLDR setting puts 0.8 on sparse  [arxiv:2402.03216 §4.3]

Contradictions

Only one card covers this concept, so no two papers disagree. Within the paper, one tension stands. Table 3 lists BM25 at 53.6 on MLDR, with the XLM-R tokenizer 2402.03216 tab. 3. Table 11 lists BM25 at 64.1 with the Lucene analyzer, which is above the sparse head's 62.2 2402.03216 tab. 11. The paper's main tables use the XLM-R BM25 row 2402.03216 §4.1. The paper does not discuss the Lucene row against its own sparse head, so both numbers are kept here.

Papers

• M3-Embedding: one encoder with dense, sparse and multi-vector heads, trained with self-distillation, for many languages and long documents.