LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

In plain words

Training a model with reinforcement learning on questions about a very long document is hard when the only feedback is whether the final answer is right. A correct answer usually needs several separate passages at once, so a lucky guess rarely contains all of them. Without those passages, the feedback says almost nothing about which passage to look at, and learning stalls.

LongRLVR changes what the model is asked to produce. Before it answers, it writes down the numbers of the passages it relied on. Those numbers earn an extra reward, on top of the reward for the answer. The reward can be checked against the known relevant passages from the training data, so the model gets feedback about its search, not only about its conclusion.

The authors argue that this extra signal is what lets training make progress. They report better scores than plain reinforcement learning on three long-context question-answering benchmarks. All of these numbers are the authors' own.

TL;DR

LongRLVR is GRPO-style reinforcement learning for long-context question answering. The reward is the answer reward plus a context reward built from the chunk IDs the model lists before its answer §2.3. The paper proves that an answer-only reward gives vanishing gradients for choosing each needed chunk §2.2. Training uses 46K synthetic grounded QA pairs for one epoch §3.1. For Qwen2.5-14B-1M, RULER-QA rises from 73.17 with plain RLVR to 88.90 with LongRLVR tab. 1. The released code differs from the paper in batch size, epoch count, KL setting, reward wiring and the answer checker, and the card's Paper vs code section lists each one.

Problem

In long-context RLVR, the reward depends only on whether the final answer is correct. Many long-context questions need several passages together, so a rollout that gets the answer right must already contain most of them. Under this reward, the gradient that would raise the selection probability of any single needed chunk is scaled by the chance that the other needed chunks are already selected. The authors call this chance the activation probability §2.2. They argue it is near zero early in training for many chunks, so the gradient is effectively zero. Their training curves show contextual recall flattening while training reward plateaus fig. 1 §1.

Method

Factorised policy

The long context X is split into N chunks C = {c_1, ..., c_N}. The policy generates a subset Z of chunks and then an answer y. It factorises into a grounding head, which picks Z, and an answer head, which writes y given Z eq. 2 §2.1. The paper's analysis models the grounding head as a log-linear distribution over subsets, with one logit per chunk app. A.1.

Objective

The objective is the expected sum of the answer reward and the context reward over the data and the policy eq. 8 §2.3. The paper states that PPO or GRPO can optimise it §2.3, and its experiments use naive GRPO §3.1.

Training data has the form (X, Q, G, y): context, question, gold evidence chunks and answer §2.4. The model's output has three parts, in this order: reasoning, a tagged list of chunk IDs, then the answer fig. 2 §2.3.

Context reward

The context reward is built from the F-beta score of the chosen chunk set Z against the gold set G eq. 6 §2.3.2:

F_\beta(Z,G) = (1+\beta^2)\,\frac{\mathrm{Precision}\cdot\mathrm{Recall}}{\beta^2\,\mathrm{Precision}+\mathrm{Recall}} eq. 6

The implemented context reward multiplies this score by a gate built from the answer reward eq. 7 §2.3.2:

r_{\text{ctx}} = \eta\,F_\beta(Z,G) + (1-\eta)\,r_{\text{ans}}\,F_\beta(Z,G) eq. 7

The total reward is r_ans + r_ctx eq. 5. The first term of eq. 7 pays for correct grounding whatever the answer. The second pays only when the answer is right, so a good list with a wrong answer earns the smaller term. The paper uses eta = 0.1 and beta = 2 §3.1. The released reward function follows the same steps:

# Reward for one rollout  [§2.3] [eq. 5] [eq. 7]
# Input:  response (str); gold_answer (str); gold_chunks (set of int)
# Output: scalar reward, r_ans + r_ctx

def rollout_reward(response, gold_answer, gold_chunks):
    # Exactly one block per tag, or every term is 0.
    # [code: recipe/dapo/longrl_reward_manager.py#L238-L247]
    used_block = single_block(response, "useful_chunks")
    answer_block = single_block(response, "answer")
    if used_block is None or answer_block is None:
        return 0.0
    # Chunk IDs in written order; repeats are kept.
    # [code: recipe/dapo/longrl_reward_manager.py#L251]
    used_chunks = parse_chunk_ids(used_block)
    # Judge returns 1.0 if the answer matches the gold answer, else 0.0.
    # It also sees the question and the gold chunk text.
    # [code: recipe/dapo/longrl_reward_manager.py#L278]
    answer_reward = judge_matches(answer_block, gold_answer)
    hits = len(set(used_chunks) & gold_chunks)
    # Precision divides by the list length.
    # [code: recipe/dapo/longrl_reward_manager.py#L255]
    precision = 0.0
    if used_chunks:
        precision = hits / len(used_chunks)
    recall = hits / len(gold_chunks)
    # F-beta with beta = 2, written as 5PR / (4P + R).
    # [eq. 6] [code: recipe/dapo/longrl_reward_manager.py#L258-L260]
    f_beta = 0.0
    if precision + recall > 0:
        f_beta = 5 * precision * recall / (4 * precision + recall)
    # Context term; eta = 0.1 is a constant in the code.
    # [eq. 7] [code: recipe/dapo/longrl_reward_manager.py#L281]
    context_reward = 0.1 * f_beta + 0.9 * answer_reward * f_beta
    return answer_reward + context_reward

Theory

Proposition 1 §2.2: for an essential chunk c_j, the gradient of the expected answer reward with respect to its logit is non-negative and at most p_j(1 - p_j) times a bound on the marginal gain of c_j times the activation probability. Here p_j is the selection probability and the activation event is that the other chunks needed with c_j are already selected app. A.2. The proof uses the REINFORCE covariance identity. The paper states that GRPO's group-relative gradient is the same up to a factor (K-1)/K app. A.2.

Proposition 2 §2.3.1: for an additive context reward, the context term adds a dense component, alpha_j p_j(1 - p_j), that does not depend on the activation probability app. A.3. The paper's lower bound on the total gradient needs non-negative covariances between selections of related chunks §2.3.1.

Synthetic data pipeline

Algorithm 1 builds one QA pair per document alg. 1 §2.4:

1. Split the document into chunks, embed them with a sentence encoder, and cluster the embeddings into thematic groups.
2. For each group, a generator model writes candidate questions, answers and evidence sets. A verifier model scores each candidate, and the best candidate per group is kept.
3. Keep the best of those group winners for the document. Its evidence set becomes G.

The settings in the appendix are: documents of 8K to 64K tokens from book, arXiv and code sources app. B.1; 64 sentence-aware segments per document, embedded with BGE-M3 and grouped with DBSCAN app. B.2; 4 clusters per document with at least 4 chunks each, 3 candidate QA pairs per cluster from Qwen3-235B-A22B, a 1 to 10 rating from the same model, two rounds of selection, and a final cut at rating 9 app. B.3. The final set has 46K documents §3.1. Before training, the authors drop questions that the base model answers well with the full context: a Qwen3-235B-A22B judge rating of 8 or higher §3.1.

Hyperparameters

Results

All numbers are author-reported. Setup: each of the three base models is trained with SFT, with naive RLVR (GRPO) and with LongRLVR, on the same 46K synthetic set §3.2. Benchmarks are RULER-QA at 32K, 64K and 128K; LongBench v2 (multiple choice, overall score); and LongReason at 32K, 64K and 128K §3.2. The paper gives no hardware or wall-clock figures in its text.

Averages from tab. 1:

Reference models in the same table tab. 1: Qwen3-14B in thinking mode scores 87.60, 44.9 and 77.50. QwenLong-L1-32B scores 79.47, 41.0 and 80.94. LLaMA-3.1-70B scores 60.73, 25.9 and 57.59. Qwen2.5-72B with YaRN scores 56.20, 43.5 and 72.76.

The paper's text says LongRLVR gives gains over SFT and RLVR for all three models at every benchmark §4.1. It also says the 14B LongRLVR model beats Qwen3-14B on RULER-QA and LongReason §4.1.

Ablations, from the paper's text and figures:

• Answer-only versus context-only versus both fig. 3 §4.2: the answer-only run's contextual recall stalls. The context-only run improves early on LongBench v2 and then degrades. Combining both gives the best and most stable curve.
• Data quality, 7B model, LongBench v2 §4.3: keeping the best-rated QA pairs gives 38.6. The median-rated pairs give 36.6 and the worst-rated give 34.8. Dropping only easy questions is the best filter. Dropping hard questions falls to 35.8, close to no filter at 35.6 fig. 4.
• eta §4.4 fig. 5a: best near 0.1. Pure gated reward (eta 0) is weaker because the early signal is too sparse. Pure ungated reward (eta 1) is weaker still.
• beta §4.4 fig. 5b: best at 2.
• Chunk count at evaluation §4.4 fig. 5c: performance holds from 16 to 128 chunks per document.

Paper vs code

The official code is at https://github.com/real-absolute-AI/LongRLVR, linked from the abstract. Commit d07b3f1 was cloned. The repository has no LICENSE file and its README states no license, so it is cite-only. The README points to the root run_longrl.sh as the training template README.md. A second script under recipe/dapo/ differs from it; both are listed below.

Discrepancies between the paper and the code:

• Epochs. The paper trains for one epoch §3.1. Both scripts set five epochs run_longrl.sh run_longrl.sh.
• Batch size. The paper uses 512 prompts per step §3.1. The root script uses 32 run_longrl.sh and the recipe uses 64 run_longrl.sh. Mini-batch size is 8 in the root script run_longrl.sh and 32 in the recipe run_longrl.sh. The paper does not state a mini-batch size.
• KL penalty. The paper does not mention one. The root script turns it off run_longrl.sh. The recipe turns it on at 0.001 with low-variance KL run_longrl.sh.
• Reward manager name. The recipe script sets longrl_naive run_longrl.sh. The dispatch in main_dapo.py accepts only naive, prime, dapo and longrl, and raises NotImplementedError otherwise main_dapo.py. The root script uses longrl run_longrl.sh, which the dispatch accepts. I read the dispatch but did not run either script.
• Answer reward. The paper calls it verifiable and does not say how the answer is checked in the text read here. The code asks a hosted language model to judge equivalence. It uses the model name default, temperature 0 and at most 512 output tokens, and the endpoint address is hard-coded longrl_reward_manager.py longrl_reward_manager.py. The judge sees up to 12,000 characters of reference chunks longrl_reward_manager.py. Any API error scores 0 longrl_reward_manager.py.
• Fallback parsing. If the judge omits the double brackets, the code looks for the substrings YES and NO longrl_reward_manager.py.
• eta and beta. Both are constants in the reward code, not config values longrl_reward_manager.py longrl_reward_manager.py.
• Malformed output. The code requires exactly one <useful_chunks> block and exactly one <answer> block. Otherwise every term is 0 longrl_reward_manager.py longrl_reward_manager.py. The paper does not say this.
• Repeated chunk IDs. Precision divides by the length of the parsed list, so a repeated ID lowers it longrl_reward_manager.py longrl_reward_manager.py. The paper does not define precision for repeated IDs.
• Clip ratios and temperature. The paper's text does not state clip ratios or rollout temperature. The root script uses clip ratios 0.2 and 0.28 run_longrl.sh and rollout temperature 1.0 run_longrl.sh.
• Clustering and chunking. The paper describes DBSCAN over 64 sentence-aware segments per document app. B.2. The released clustering code uses HDBSCAN with cosine distance, minimum cluster size 4, minimum samples 2 and a similarity threshold of 0.7 clustering.py clustering.py. Its chunker targets 512 tokens by default, with no overlap clustering.py clustering.py. The released pipeline therefore does not reproduce the paper's segmentation.
• Rating cut. The paper's experiment section keeps pairs rated above 9 §3.1. Its appendix discards pairs below 9 app. B.3. At exactly 9 the two readings differ. The rating step in the released code records a score per pair qa_pairs_judge.py, and I found no cut-off in data_gen/.
• Easy-question filter. Not in the repository, so not verified.
• Released evaluation outputs. eval_results/ contains generation files. I did not recompute the Table 1 scores from them, so they are not verified.

No other official repository was found. The abstract link was the only lead.

Pitfalls

• The answer reward is an LLM judge in the code. "Verifiable" in the paper therefore means a model decides correctness. The gate term 0.9 r_ans F_beta passes that judgement straight into the context reward. Any implementation needs its own judge, and its noise and bias enter both terms.
• Malformed output scores zero, even when the answer is right. A correct answer without a valid chunk block earns 0 in the released code longrl_reward_manager.py. The policy can learn to match the tags exactly, so tag format is part of the task.
• The analysed reward is not the implemented one. Propositions 1 and 2 concern a general reward and an additive form with weights alpha_k §2.2 §2.3.1. The reward used in experiments is the gated F-beta form eq. 7. The paper does not measure the activation probability, so the size of the claimed gradient gap is not known.
• Chunking must match between training and use. The model is trained on a fixed chunking. The README warns that inference must use the same chunking strategy README.md. The paper's training uses 64 segments per document, while the released code's default is 512-token chunks. The robustness result covers 16 to 128 chunks at evaluation only §4.4.
• Prose and table disagree on the 14B baseline. The abstract says LongBench v2 rises from 39.8 to 46.5 [abstract]. The table's RLVR row is 39.8 tab. 1. The introduction says 40.2 to 46.5, and 40.2 is the untrained base model's row §1 tab. 1. Use the table values.
• Benchmark rows are not all the same protocol. Standard baselines on LongBench v2 use chain-of-thought prompting. Models that produce reasoning are scored on their final answer §3.2. Thinking-mode Qwen3 rows are not like-for-like with the rest.
• Left truncation. The root script sets prompt truncation to left run_longrl.sh, and the recipe disables filtering of overlong prompts run_longrl.sh. Left truncation removes text from the start of a prompt. If a prompt exceeds 64K tokens, the earliest chunks can be cut while the gold IDs still refer to them. Check prompt lengths before training.
• Data-generation cost is hidden. Generating and judging 46K pairs uses Qwen3-235B-A22B with chain-of-thought prompting app. B.3. The paper gives no GPU count or time. Each training step samples 8 responses for each of 512 prompts, with contexts up to 64K tokens §3.1.
• The step count implied by the paper is roughly 90 per epoch. This is derived from 46K documents divided by 512 prompts per step, not stated in the paper. The released scripts use 32 or 64 prompts per step, which gives many more steps for the same epoch count.

Lineage

• Builds on: reinforcement learning with verifiable rewards as the baseline, and GRPO, which the paper cites as DeepSeekMath §3.1. No arXiv id is given in the paper for GRPO.
• Compared against: SFT on the same data, naive RLVR with GRPO, QwenLong-L1-32B, the Qwen3 series, LLaMA-3.1-70B and Qwen2.5-72B with YaRN §3.2 tab. 1.
• Superseded by: not known.