wikillm

Paper · 2026-10-29 · cs.CL, cs.IR

Glance: The Generator Is Its Own Reranker

Glance adds a rank-16 LoRA decision adapter and a 256-d pointer head to one 9B generator. It scores each candidate passage with a short probe branched from one shared prefill, then the same weights answer. Author-reported: after 39 min of MS MARCO training, nDCG@10 is 0.741 on SciFact and 0.345 on NFCorpus, vs 0.721 and 0.326 for bge-reranker-v2-m3.

Passage rerankingRetrieval-augmented generationLow-rank adaptation (LoRA and variants)KV-cache managementrerankingretrieval-augmented-generationlorakv-cacheshared-prefillhybrid-attention

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A retrieval system usually needs two models: one that sorts the search results by relevance, and one that writes the answer from the best ones. Each model has to be loaded, and each one reads the candidate passages again.

This paper asks whether the answer-writing model can do the sorting too. It stays one model. A small add-on, about 0.1 GiB per extra skill, teaches it to judge each candidate passage. All the passages are read once, and then a short "is this relevant?" check is run for each passage against that same stored reading. The same model then writes the answer.

The authors report that after a short training run on search data, this single-model setup matches or beats a dedicated reranker on two search benchmarks, and gives better answers on one question-answering set. They are clear that it is a proof of concept on small tests, and that a dedicated reranker is still faster per query.

TL;DR

Glance equips one frozen 9B backbone with a rank-16 LoRA decision adapter and a 256-dimensional pointer head. Candidate passages are prefilled once, and each passage gets a short probe that branches from that shared cache; the head turns each probe into a score §1 §3.2 tab. 1. The same backbone then answers with the adapter switched off §3.5. Author-reported results are nDCG@10 0.741 (SciFact) and 0.345 (NFCorpus) after 39 minutes of MS MARCO training, against 0.721 and 0.326 for bge-reranker-v2-m3 tab. 3. Scoring is slower per query than a cross-encoder §4.2 §5.

Problem

RAG pipelines often serve a separate reranker model and a separate generator. Each reranker encodes every query-passage pair, so model residency and passage processing are duplicated §1. The paper's premise is that the generator reads the same passages anyway, so its computation can be shared §1. Long-context models also overlook evidence that sits mid-context, which inputs, which the authors use as a reason to keep reranking §2.

Method

Overview. Glance has two variants. They differ in where the adapter runs and in how the answer is generated §3.1 fig. 1.

  • Main. The adapter runs on every token. Candidates are prefilled and probed, the best n are kept, those passages are prefilled a second time with the LoRA bypassed, and the answer is generated from that second pass §3.1.
  • Activated. The LoRA runs only on probe tokens. Candidates are prefilled with the base weights, probes are scored through the adapter, and the base weights then extend the original cache to produce the answer. The extended cache still holds all k candidates §3.1 §4.7.

The main experiments use the main variant, except §4.7 §3.1.

  1. Decision adapter. A rank-16 LoRA on the frozen backbone plus a 256-dimensional pointer head. Only these are trained §1 §3.2 tab. 1.
  2. Shared state. The k candidate passages are written as numbered blocks, "[Passage i] p_i", and prefilled once §3.3.
  3. Probes. In the packed_all format (the one used for all main results), each candidate gets one probe, a short yes/no question phrased as "Does Passage i contain information that answers or supports the query: q?" §3.3. Each probe is about 30 tokens. Probes continue from the end of the state, with positions that pick up there, and never attend to each other §3.3.
  4. Score. The pointer head compares the hidden state at the decision token with the hidden state at the end of each option, and returns a softmax over the options. Yes/no questions use the options {false, true} §3.2. The paper does not say which option's probability is the score; see Paper vs code.
  5. One pass. Using Kev's shared-prefix path, the state runs once and every probe starts from a copy of the state's cache at its end. For attention layers this is the keys and values. For the Gated DeltaNet layers it is the convolution window and the recurrent state §3.4. Because a DeltaNet state folds in the whole prefix sequentially, so per-passage caches cannot be stitched together in any order. The main variant therefore reprocesses only the selected passages (about 1.5k tokens) §3.4.
  6. Answer. Main variant: prefill the top-n passages again, with the adapter off, using the model's chat template, then decode greedily for at most 48 new tokens §3.1 §3.5 §4.5. Adapter off means the LoRA layers are bypassed, and the same weights run as the base model §3.5.
  7. Training. Start from the general decision adapter. Train on 4,000 MS MARCO training queries. Each record has one positive and 7 to 19 BM25 hard negatives, shuffled, and each probe carries a 0/1 label. One epoch is 500 optimizer steps over 55,598 labelled probes §3.6 tab. 1.
  8. Activated variant. The adapter stays disabled during the state pass and is enabled only for the probe branches, from the first probe token onward. Training and inference both do this, through a modified version of Kev's shared-prefix routine. The patched state cache matches the base model's bitwise in a 0.8B check §3.4 §4.7.
Pseudocode
# Shared-prefill reranking with one resident backbone  [§3.1] [§3.3]
# Input:  query (str), passages (list of k str)
# Output: one relevance probability per passage

def rerank(query, passages):
    # Write all candidates once as numbered blocks  [§3.3]
    state = join_numbered(passages)
    # One probe per passage, each a short question  [§3.3]
    probes = [make_probe(query, i) for i in range(k)]
    # One prefill of the state, then all probes from its cache  [§3.1] [§3.4]
    logits = forward_shared_prefix(adapter_on, state, probes)
    # Pointer head turns each probe into a score over its options  [§3.2]
    return [softmax_over_options(z) for z in logits]
Pseudocode
# Main pipeline: rerank, select, re-prefill, answer  [§3.1] [§4.5]
# Input:  query, candidates (k passages), n (top-n)
# Output: answer text

def answer_main(query, candidates, n):
    # Score every candidate with the adapter on  [§3.1]
    scores = rerank(query, candidates)
    # Keep the n highest-scoring passages, in rank order  [§3.1]
    selected = top_n(candidates, scores, n)
    # Prefill the chosen passages again, adapter disabled  [§3.1] [§3.5]
    cache = prefill_chat(selected, query, adapter_on=False)
    # Greedy decoding, at most 48 new tokens  [§4.5]
    return greedy_decode(cache, max_new_tokens=48)
Pseudocode
# Training step on packed records, one 0/1 label per probe  [§3.6]
# Input:  records, each a query with its passages and labels
# Output: updated LoRA weights and pointer head

def training_step(records):
    losses = []
    for record in records:
        # The state is computed once per record  [§3.4] [§3.6]
        logits = forward_shared_prefix(adapter_on, record.state, record.probes)
        # Loss function is not named in the paper  [§3.6]
        losses.append(loss_fn(logits, record.labels))
    # LoRA lr 5e-5, head lr 1e-4, batch 8, bf16  [§3.6]
    return optimizer_step(mean(losses))

Hyperparameters

NameValueWhereNotes
BackboneMiMo-V2.6-Distill-Qwen-9B, frozen, bf16§4.1Hybrid Gated DeltaNet and full attention, 3:1 §3.4
LoRA rank16§3.2 tab. 1Target modules not stated in paper; code uses "all" training_config.json
Pointer head width256§1 tab. 1Code: head_dim 256 training_config.json
Initialisationgeneral decision adapter (zero-shot, 20 min, 4.4k records)§4.1 tab. 1The training mix includes no MS MARCO, BEIR or HotpotQA examples §4.1
Training data4,000 MS MARCO records, 8 to 20 passages each§3.6 tab. 11 positive plus 7 to 19 BM25 hard negatives §3.6
Training length500 updates per epoch, covering 55,598 labelled probes and 7.7M tokens in forward passes§3.639 minutes on one B200 §3.6
LoRA learning rate5e-5§3.6Code: 5e-05 training_config.json
Pointer head learning rate1e-4§3.6Code: head_lr 1e-04 training_config.json
Batch size8 records per optimizer step§3.6Code: batch 1 with grad accumulation 8 training_config.json
Precisionbf16§3.6Peak GPU memory 21 GiB §3.6
Warm-up, weight decay, LR schedulenot stated§3.6Code sets weight_decay 0.01 and 1 epoch training_config.json
Trained-long data+450 MS MARCO records, 20 to 80 passagestab. 1 tab. 4Warm-started from trained, 14 min tab. 1
Control and activated runs1,200 MS MARCO records, 150 stepstab. 1 §4.7Activated: probe tokens only; H100, 17 min §4.7
RetrievalBM25 (bm25s), top-20; passages truncated to 220 words§4.1Scaling study uses top-k with 100-word passages §4.4
DatasetsSciFact (300 test queries), NFCorpus (323 test queries), nDCG@10§4.1BEIR §4.1
Candidate counts in scalingk in {10, 20, 50, 100, 200}§4.4k=200 runs on the chunked path tab. 4
Generationgreedy, at most 48 new tokens, top-3 passages§4.5Answer prompt not given in the paper
Significancepaired bootstrap, 10,000 resamples, 95% CI§4.5BEIR differences not significance-tested §4.1
Order test4 random orders plus BM25 order§4.8100 SciFact queries, 220-word passages
Context caps (wrapper)max state 8,192; max branch 16,384; max packed 32,768 tokensglance_rag.pyCode only; not in paper

Results

Author-reported. Setup: BM25 top-20 candidates, zero-shot or MS MARCO-trained adapter, nDCG@10. Latencies are medians per query on the GPU named in the section, and are not comparable across sections §4.1.

Zero-shot, single pass vs separate calls tab. 2 §4.2. Shared GPU: RTX PRO 6000.

MethodSciFact nDCG@10SciFact ms/queryNFCorpus nDCG@10NFCorpus ms/query
BM250.662--0.306--
bge-reranker-v2-m30.721310.32660
Decision 9B, separate (k calls)0.72922100.3232485
Glance packed_all (1 pass)0.7156140.341670
Glance packed_q (1 pass)0.7217730.328847
Glance listwise (1 pass)0.6845440.331573

Zero-shot vs trained (39 min of MS MARCO) tab. 3 §4.3. On a B200, the trained packed_all variant takes 420 ms per query on SciFact and 411 ms on NFCorpus.

MethodSciFact zero-shotSciFact trainedNFCorpus zero-shotNFCorpus trained
BM250.6620.6620.3060.306
bge-reranker-v2-m30.7210.7210.3260.326
Glance packed_all (1 pass)0.7150.7410.3410.345
Glance packed_q (1 pass)0.7210.7380.3280.338
Glance listwise (1 pass)0.6840.6920.3310.334

Scaling with candidates in one pass tab. 4 fig. 2 §4.4. SciFact, first 100 queries, passages trimmed to 100 words. Peak memory is in GiB.

ktokens per passzero-shottrainedtrained-longbge-v2-m3ms/querypeak GiB
102.2k0.6510.653--0.62625215.4
204.3k0.6620.6580.6740.62416516.7
5010.8k0.6370.6740.6910.64241725.5
10021.6k0.5840.6820.6970.637104956.4
20043.4k----0.7170.6591579020.5

The k=200 row runs on Kev's chunked path, which does not fit one call tab. 4. Latency and memory for trained-long are on a B200; the k=10 latency is from a separate run of the trained adapter tab. 4.

End to end: HotpotQA distractor, first 300 validation questions tab. 5 §4.5. Every row uses the same 9B backbone as generator. Top-3 passages are given to the generator, and "adapter on" means the decision LoRA is left active while answering. Zero-shot upper block; trained lower block.

Context given to the generatorEMF1R@2R@3rerank ms
(a) all 10 paragraphs0.5330.716------
(b) bge-reranker-v2-m3 top-3, adapter off0.5000.6830.8630.91829
(c) Glance top-3, adapter off0.5430.7180.7620.865378
(b') bge-reranker-v2-m3 top-3, adapter on0.5470.7020.8630.91829
(c') Glance top-3, adapter on0.5900.7570.7620.865378
(d) oracle: 2 gold paragraphs0.6070.789------
(b_t) bge-reranker-v2-m3 top-3, adapter off0.5030.6840.8630.91824
(c_t) Glance trained top-3, adapter off0.5530.7290.8970.962274
(b'_t) bge-reranker-v2-m3 top-3, trained adapter on0.5100.6820.8630.91824
(c'_t) Glance trained top-3, trained adapter on0.5770.7470.8970.962274

With the trained adapter and the adapter off, the F1 gap over the cross-encoder is +0.045, with a paired bootstrap 95% CI of [+0.014, +0.076] §4.5. The paper reports that the zero-shot adapter-off F1 gap (+0.035) is not significant §4.5.

Other reported results (all single-seed or small-sample; see Limitations):

  • Activated variant, retrieval. SciFact and NFCorpus nDCG@10 are 0.740 and 0.351 for activated, against 0.744 and 0.351 for a matched control with the adapter on all tokens (packed_all, k=20) §4.7.
  • Activated variant, cache reuse. On 100 HotpotQA distractor questions, continuing the state cache gives EM/F1 0.61/0.750, against 0.59/0.754 for re-prefilling the top-3. Median answer latency is 138 ms against 181 ms. This mode gives the generator all passages, not the reranked subset §4.7.
  • Skill retention. On 231 public JevBench items, the general adapter scores 184 (79.7%), and the reranking adapter scores 175 (75.8%) §4.6.
  • Memory per decision skill. About 0.1 GiB (LoRA plus head) §4.6 tab. 6.
  • Order sensitivity. On 100 SciFact queries, nDCG@10 ranges from 0.678 (BM25 order) to 0.693 across orders. Averaging over 5 orders gives 0.689. The top passage is the same as in BM25 order in 91% of cases §4.8.
  • Serving in vLLM (prototype). The prototype's probabilities deviate from Kev's by 0.006 on average, with the largest deviation at 0.10. Per-query Spearman correlation is 0.995, with 4 of 400 decisions flipped. Throughput on a shared B200 is 1.54 requests/s, against 0.86 on Kev's sequential route §4.9.

Paper vs code

The official repository is a Hugging Face model repository, not GitHub. It is owned by the airun-ai organisation, which the paper names as the authors' lab, and the paper links it §Availability. Its Apache-2.0 LICENSE file was read. Revision read: b1956ee6e406177677d7df9b2b8376204e3ac798. The code/ folder last changed in a72a6a12ba35be17f637b8223e2b00fda9099dfb. The repository could not be cloned, so the code was read through its raw file URLs. Line numbers were counted by hand from the served text and are not independently verified. Paths are relative to the repository root.

The repository contains only inference and data-preparation code. The training loop is not included: the README says training uses python -m kev.train from the Kev runtime, which is not in the repository. Pointer-head internals are also in Kev. The following differences were found:

  • Default adapter in the wrapper. code/glance_rag.py loads airun-ai/MiMo-V2.6-Distill-Qwen-9B-Decision by default [code: code/glance_rag.py#L26-L27]. The paper names this as the zero-shot general decision adapter §Availability. The trained reranker is in the repository root (the inference.py default is airun-ai/GlanceRAG-MiMo-9B-Reranker) inference.py. The paper says the root is the trained-long adapter §Availability.
  • Training objective and augmentation. The paper uses one binary label per probe and does not name a loss §3.6. training_config.json sets ordinal_objective to ranked_probability_score, and sets augmentation probabilities p_none 0.1, p_distract 0.15, p_none_distract 0.12 and perm_frac 0.3 training_config.json. None of these appear in the paper.
  • Hard-negative fill. The paper says 7 to 19 BM25 hard negatives per record §3.6. code/make_train.py draws k between 8 and 20 [code: code/make_train.py#L30]. When fewer hard negatives exist, it fills the record with negatives from earlier queries, and it skips records while that pool is too small make_train.py. The paper does not describe this.
  • Passage text. make_train.py writes each passage as its title, a newline, then the text, truncated to 220 words make_train.py. The paper does not mention titles.
  • Probe wording. The paper's question is "Does Passage i contain information that answers or supports the query: q?" §3.3. The code puts the query in quotation marks glance_rag.py. The relevance criteria text ("Relevant: answers or directly supports the query" and "Not relevant to the query") is in the code only glance_rag.py.
  • Score definition. The paper does not say which option probability is the score. The code takes the softmax probability of "true" over ["false", "true"] glance_rag.py.
  • Context cap. The wrapper sets max_state to 8,192 tokens [code: code/glance_rag.py#L29]. The 200-candidate runs (43.4k tokens) used Kev's chunked path, which is not in this repository §4.4 tab. 4.
  • Answer step. The paper says the top-n passages use the model's chat template §3.5. The code adds a system prompt and a "Short answer:" line [code: code/glance_rag.py#L30-L31] glance_rag.py. It keeps only the first output line and removes think tags glance_rag.py. The paper does not describe these steps.
  • Answer head. Kev keeps only the backbone, so the wrapper reloads the base model's lm_head from the base checkpoint's safetensors shard [code: code/glance_rag.py#L62-L72]. The paper does not mention this.
  • Order averaging. The paper suggests averaging scores over orders as a remedy §5. The wrapper scores in the order given and does not average glance_rag.py.
  • Training schedule. The paper says 500 steps over 4,000 records §3.6. The config uses batch 1 with gradient accumulation 8 training_config.json, which gives the same number of records per step. Its seed is 7, while make_train.py defaults to 42 make_train.py.
  • Trained-long run. training_metrics.json records 450 records and 57 optimizer steps, with a wall time of about 854 s training_metrics.json. This matches the paper's 450 records and 14 minutes tab. 1.

No discrepancies were found for the adapter-off answer path (the LoRA is bypassed, and generation is greedy with a 48-token limit) [code: code/glance_rag.py#L105] §4.5.

Pitfalls

  • Wrong adapter by default. GlanceRAG() in the wrapper loads the general decision adapter, not the trained reranker in the repository root. Load the reranker explicitly, or scores will not match the paper [code: code/glance_rag.py#L26-L27].
  • Scores are not independent. In packed_all, every probe can see all passages, so each score shifts with the other candidates and with their order. The per-passage probability changes by 10^-4 median, 0.033 at the 90th percentile and 0.36 at most across orders §4.8 §5.
  • Scores are not calibrated across queries. The paper uses per-query rankings. It gives no threshold for comparing scores between queries.
  • Memory grows faster than linearly in k. Each probe branch copies the shared attention cache, so memory scales with k times context length. Peak is 56 GiB at k=100 §4.4 §3.5. The wrapper's max_packed of 32,768 tokens is below the 43.4k tokens used at k=200 glance_rag.py tab. 4.
  • Batching across queries lowers throughput in the current code. Packing 16 queries (k=20) gave 1.57 queries/s, against 2.87 for a single query, and 32 ran out of memory §4.4.
  • Order of the state changes results. Pass candidates in a fixed order and record it. The paper's averaged remedy is not in the code §4.8 §5.
  • Latency is not a like-for-like comparison. The three GPUs (RTX PRO 6000, B200, H100) were partly shared, and the latency is from eager code with no CUDA graphs or fused kernels. Speed-up ratios depend on load §4.1 §5.
  • Statistical strength is limited. BEIR differences are not significance tested, and gaps below about 0.02 nDCG@10 are indicative at n≈300 §4.1. Scaling and activated results are single-seed on 100 questions §5.
  • Cross-table numbers differ. The cross-encoder scores 0.721 on SciFact in tab. 2 but 0.624 to 0.659 in tab. 4, because the subset, passage length and k differ. Do not compare rows across tables.
  • Adapter-on answers are shorter. Answers with the adapter left on average 2.5 words, against 3.1 with the adapter off §4.5. Adapter-on F1 gains may be partly from this change in answer style. The paper's control for this is row (b') §4.5.
  • Activated cache reuse changes the context. Continuing the cache gives the generator all k candidates, not the reranked top-n §4.7. Do not treat it as the same setup as the main pipeline.
  • Hybrid cache copies. DeltaNet state cannot be stitched per passage, so the main pipeline reprocesses the selected passages (about 1.5k tokens) §3.4. Plan for that extra prefill when budgeting latency.
  • Chat behaviour with the adapter on. The paper checked a handful of prompts and found little change. The adapter does shorten HotpotQA answers, and whether a single adapted model can stand in for switching needs wider testing §4.5.

Lineage

  • Builds on: Kev, a typed-decision runtime with shared-prefix execution (the paper cites it as Kev, 2026; it is software, not an arXiv paper) §2 §3.4. LoRA (arXiv id not given in the paper) §3.2. Gated DeltaNet linear attention §3.4. Activated LoRA (arXiv:2504.12397), which the paper adopts for the activated variant §2 §4.7.
  • Compared against: bge-reranker-v2-m3 cross-encoder §4.1; monoT5 and RankGPT-style LLM rerankers §2; in-context reranking (Chen et al., arXiv id not given) §2; HyperRAG (arXiv:2504.02921), which caches document KV offline for decoder rerankers §2.
  • Related: TurboRAG (arXiv:2410.07590) and CacheBlend, which precompute and merge passage KV caches for generation §2.
  • Superseded by: not known.

Source and license

Adapted from “Glance: The Generator Is Its Own Reranker” by Adilet Metinov, Chingiz Arziev (arXiv:2610.40454), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://huggingface.co/airun-ai/GlanceRAG-MiMo-9B-Reranker at commit b1956ee6e406 (Apache-2.0). Any quoted code is shown under that license.