wikillm

Paper · 2026-10-29 · cs.CL

Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

BookNavigator lets an LLM read a textbook while it writes a proof: table of contents, keyword search and section reading. GRPO adds an annealed bonus for correct proofs that used the book and round-level credit from a frozen judge. Author-reported: Intern-S2-Preview-35B proof accuracy rises from 51.5% to 79.5%, and BrowseComp-Plus and CL-bench gain 1.7 and 2.3 points.

LLM agentRetrieval-augmented generationCredit assignmentPolicy gradientreinforcement-learninggrpoself-learningretrievalagentslong-context

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Give a language model a hard proof exercise and a textbook that covers the material, and the usual approach is to paste in a few passages or the whole book, then ask for the answer. This paper asks whether a model can instead study the book the way a student does. It can look at the table of contents, search for a keyword, and open a whole section, and it can keep doing this while it works out the proof. The exercise's own solution is removed from the book, so the model has to learn the material rather than copy an answer.

Training uses reinforcement learning. A proof earns a reward only when a grader accepts it. Because the model might stop looking things up once it can solve problems on its own, the reward also pays a bonus for correct proofs that used the book at least once. A second, separate judge looks at each lookup, using a frozen copy of the starting model, and labels it useful, neutral or harmful. Harmful lookups receive less credit than the others. The authors then test whether the skill carries over to books and subjects not seen in training, and to two long-context benchmarks that do not use the book tool.

TL;DR

BookNavigator is a three-action interface for a tool-using LLM: get the table of contents, search an index, and read a section §2.3. The policy is trained with GRPO on textbook-based proof problems, with a correct-only exploration bonus that anneals from 1 to 0.25 and round-level tool credit from a frozen judge §2.4§3.1. Author-reported: overall proof accuracy on Intern-S2-Preview-35B rises from 51.5% to 79.5%, above RL-trained no-book and fixed-retrieval baselines §3.2tab. 1. Gains on BrowseComp-Plus (+1.7) and CL-bench (+2.3) are reported without variance §3.4tab. 3. No official code was found (see Paper vs code).

Problem

Models answer knowledge-intensive questions either from parameters or with retrieval, and fixed retrieved or pre-supplied context does not adapt to what the model has already derived §1§3.2. The paper targets proof problems where a derivation needs a prerequisite that may sit several sections away from the exercise §1. Two failure modes are reported:

  • Outcome-only RL reduces textbook use. Training on correctness alone lifts overall accuracy to 74.9%, but only 6.7% of trajectories still call a tool, down from 36.9% before RL §3.5tab. 4.
  • Fixed-retrieval RL can fall below the base model on out-of-task benchmarks. Policies trained under fixed-retrieval protocols stay below Base on BrowseComp-Plus §3.4.

The paper's setting assumes a long textbook (about 208k tokens on average) that the model may need to navigate §3.1app. A.1.

Method

Task and protocol §2.1

Given a proof problem q and a structured textbook \mathcal{B}, the policy alternates reasoning z_t and one action a_t from the textbook-access action space \mathcal{A}. It then receives an observation o_t §2.1:

(z_t, a_t) \sim \pi_\theta(\cdot \mid q, (z_i, a_i, o_i)_{i=1}^{t-1}) eq. 1

The trajectory ends in a terminal round T with no textbook access. The terminal round outputs the final proof \hat y §2.1eq. 2. A judge compares \hat y with a reference solution y^\star that the model cannot see, and returns r(\tau) \in \{-1, 1\} §2.1eq. 3.

Textbook curation §2.2 app. A.2

Each textbook is parsed into chapter-indexed text. Exercises and their solutions are split from the exposition §2.2. Exercises that refer to textbook section numbers are dropped so that each problem stands alone §2.2. The model-visible book is an ordered list of sections s_k = (\iota_k, \ell_k, x_k), where \iota_k is the identifier, \ell_k the title and x_k the retained text §2.2eq. 4.

BookNavigator §2.3

BookNavigator is the only interface between the model and its book §2.3.

  • get_toc returns the complete table of contents as identifier and title pairs §2.3.
  • search_index takes keywords and returns ranked section identifiers with snippets. The score mixes how many query terms a section covers, how close they sit to each other, exact-phrase hits and whether the title matches. The weights are not stated §2.3.
  • read_section returns the full retained text of one section, including any proofs and examples §2.3.

Each action output is appended to the history. The model can therefore change its next action based on what it has already read §2.3.

Reinforcement learning with fine-grained rewards §2.4

Training uses GRPO §2.4app. C.1. Two additions change the reward and the advantage.

Exploration bonus. For each correct trajectory that made at least one successful tool call, add a bonus scaled by \beta_n. The bonus enters only through the trajectory advantage §2.4eq. 5:

R_n(\tau) = r(\tau) + \beta_n\, b(\tau)\, \mathbb{I}[r(\tau) = 1] eq. 5

Here b(\tau) = 1 for a qualifying trajectory, and \beta_n \ge 0 anneals on a cosine schedule with a positive floor §2.4§3.1app. C.1.

Tool credit. A frozen copy of the initial policy labels each non-terminal round as useful, neutral or harmful. The judge sees the problem, the history through the present observation, and the reasoning that follows it. It does not see the reference solution or the final verdict §2.4eq. 6app. C.2:

c_t = J_{\mathrm{tool}}(q, (z_u, a_u, o_u)_{u=1}^{t}, z_{t+1}) eq. 6

The terminal round is given a neutral score §2.4. The round scores are then redistributed inside each GRPO group. Let m_t be the number of unmasked policy tokens in round t and A the trajectory advantage from R_n(\tau). The round advantage is §2.4eq. 7:

\bar c = \frac{\sum_{t=1}^{T} m_t c_t}{\sum_{t=1}^{T} m_t}, \qquad \tilde A_t = A + \alpha\, |A|\,(c_t - \bar c) eq. 7

This centring keeps the token-weighted mean advantage at A §2.4. The advantage \tilde A_t is applied to all policy tokens in round t, including reasoning and tool-call tokens. Tool observations are excluded from the loss §2.4. The GRPO objective is §2.4eq. 8:

\mathcal{J}(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{\mathrm{old}}}} \left[ \frac{1}{\sum_t m_t} \sum_{t=1}^{T} \sum_{j=1}^{m_t} \min\{ \rho_{t,j}(\theta)\tilde A_t,\ \mathrm{clip}(\rho_{t,j}(\theta))\tilde A_t \} \right] eq. 8

The ratio \rho_{t,j} is per token and per round, and the clipping range is given in app. C.1.

Pseudocode

The reward and credit steps as the paper describes them. The paper does not publish code, so these are a reading of eq. 5 to eq. 7 and app. C.1.

Pseudocode
# Trajectory reward with exploration bonus  [§2.4] [eq. 5]
# Input:  correct (bool, r(tau) = 1), tool_ok (bool), beta (float >= 0)
# Output: scalar reward R_n(tau)

def trajectory_reward(correct, tool_ok, beta):
    # r(tau) is +1 for a correct proof and -1 otherwise  [§2.1]
    outcome = 1.0 if correct else -1.0
    # Bonus only for correct proofs with at least one successful tool call  [§2.4]
    bonus = 1.0 if (correct and tool_ok) else 0.0
    return outcome + beta * bonus
Pseudocode
# Round-level advantages for one trajectory  [§2.4] [eq. 6] [eq. 7]
# Input:  A (trajectory advantage), labels (one per round, terminal last),
#         token_counts (unmasked policy tokens per round), alpha
# Output: one advantage per round, applied to all tokens of that round

def round_advantages(A, labels, token_counts, alpha):
    # Harmful scores 0.5; useful and neutral (incl. terminal) score 1.0  [app. C.1]
    scores = [0.5 if label == "harmful" else 1.0 for label in labels]
    # Token-weighted mean over all rounds, terminal round included  [eq. 7]
    mean_score = (sum(m * c for m, c in zip(token_counts, scores))
                  / sum(token_counts))
    # Centred redistribution keeps the token-weighted mean at A  [eq. 7]
    return [A + alpha * abs(A) * (c - mean_score) for c in scores]

Hyperparameters

NameValueWhereNotes
Policy initialisationIntern-S2-Preview-35B§3.1 app. C.1all RL runs start from this model
RL steps50 rollout steps§3.1
Rollout batch128 problem groups, 8 trajectories per problemapp. C.1two mini-batch optimisation steps per batch
OptimiserAdamW, constant LR 1e-6, β1 0.9, β2 0.95, weight decay 0.1app. C.1no warm-up
Gradient clippingglobal norm 1.0app. C.1
Advantage estimatorleave-one-out group advantagesapp. C.1
Importance-weight clip[0.8, 1.28]app. C.1token-level weighting
Maskingtokens with ratio above 2.0 maskedapp. C.1
KL regularisationdisabledapp. C.1
Group-entropy reweighting, whitespace penaltyinherited; activated at a 4% fractionapp. C.1details not stated
Exploration bonus b1 for a correct trajectory with a successful tool call§3.1
Bonus schedule \beta_ncosine from 1 to 0.25 over 50 rollout steps§3.1 §2.4cosine formula not stated
Tool-credit mappingharmful 0.5; useful or neutral 1§3.1
Redistribution strength \alpha1§3.1
Tool judgefrozen initial policy; temperature 0; high reasoning; 65,536-token limitapp. C.2used only for training credit
Correctness judgeGLM-5.2; 8 independent reviews at temperature 1.0; high reasoning; 65,536 tokens each§3.1 app. C.2accept only if all 8 pass
Self-Learning limitsat most 8 interaction turns; 64k-token trajectory cap§3.1
Baseline limits64k-token input budget; 64k-token output§3.1
Top-k Sectionsup to 8 sections, relevance order§3.1
Evaluationmean accuracy over 4 sampled solutions per problem§3.1sampling temperature not stated for the policy
Dataset158 textbooks, 5,183 problems§3.1 app. A.1
Training split4,557 problems from 132 textbooks§3.1
ID-Book split205 problems from 4 training textbooks§3.1held out from training
OOD-Book split155 problems from 7 unseen textbooks§3.1
OOD-Subject split266 problems from 19 probability and statistics textbooks§3.1 app. A.1
Book-tool prompt budgetsone get_toc; at most two distinct searches; at most five calls per turnapp. B.2prompt-level; not the same as the 8-turn cap
BrowseComp-PlusBM25 retrieval; Qwen3-32B judge§3.4 app. C.2official protocol
CL-benchdirect generation, no tools; GPT-5.1 judge at low reasoning effort§3.4 app. C.2official protocol

Results

Author-reported. Setup: 158 textbooks and 5,183 problems; base model Intern-S2-Preview-35B; accuracy in percent. Each protocol is scored on the base model ("Base") and after RL ("+RL") §3.1.

Main proof accuracy, base and after RL tab. 1

ProtocolID-Book BaseID-Book +RLOOD-Book BaseOOD-Book +RLOOD-Subject BaseOOD-Subject +RLOverall BaseOverall +RL
No-Book39.053.868.978.460.874.155.768.5
Top-k Sections43.756.673.181.070.281.362.273.1
Long-Context39.657.169.480.862.177.556.772.2
Oracle Section43.254.470.777.669.079.960.971.0
Self-Learning33.965.565.285.256.886.751.579.5

Deltas over Base are listed in tab. 1. Self-Learning gains +28.0 overall §3.2.

Reference interventions, overall accuracy tab. 2 §3.3

PolicyPaired textbookNo-BookTopic-matched textbookSource removalNon-source removal
Base51.555.749.649.053.0
+RL (Self-Learning)79.571.775.274.679.8

Out-of-task generalisation tab. 3 §3.4

RL protocolBrowseComp-PlusDKRRSAPTEEDSCL-bench overall
Base50.214.516.316.67.514.8
No-Book48.618.415.615.512.616.2
Top-k Sections49.916.116.116.49.615.5
Long-Context49.416.913.415.311.614.9
Oracle Section48.216.113.114.911.114.4
Self-Learning51.919.017.515.912.617.1

Column keys for CL-bench: DKR is the domain-knowledge task, RSA applies a given rule system, PTE follows a procedure, and EDS covers empirical discovery and simulation tab. 3.

RL ablation under the Self-Learning protocol tab. 4 §3.5

RL settingIn-book splitUnseen-book splitUnseen-subject splitAll splitsTrajectories with a tool (%)Calls per trajectory
Base33.965.256.851.536.91.5
RL (correctness only)60.781.981.574.96.70.3
RL + tool credit64.081.182.576.27.00.3
RL + exploration bonus61.883.984.477.079.95.9
Self-Learning (full)65.585.286.779.558.43.0

Paper vs code

No official code. The arXiv abstract page lists no code, data or project link, and the LaTeX source contains no GitHub or code link. Searches: GitHub repository search for BookNavigator returned two unrelated third-party repositories; booknavigator restricted to the InternLM organisation returned none; self-learning textbook reinforcement returned none. No repository belonging to the authors was found, so no code was read.

Pitfalls

  • Reward scale and bonus. The base reward is \pm 1, and the bonus is \beta_n b with b = 1 and \beta_n between 1 and 0.25 §2.4§3.1. A correct proof with a successful tool call therefore earns 1 + \beta_n, up to 2, against 1 for a correct proof without one and -1 for a wrong proof. Tool calls in wrong trajectories earn no bonus eq. 5.
  • "Successful" tool call is undefined. The bonus needs at least one successful tool call §2.4, but the paper does not define success. The book-tool prompt separates an executed call from a successful retrieval, and it asks the policy to judge the match itself app. B.2. Pick a definition and record it.
  • Cosine schedule. The paper gives only the endpoints, 1 and 0.25 over 50 steps, and says cosine with a positive floor §2.4§3.1. The exact formula is not stated.
  • Terminal round in the centre. \bar c is token-weighted over all rounds, including the terminal round with neutral score 1 §2.4eq. 7. A long final proof therefore pulls \bar c toward 1. The paper does not discuss this effect.
  • Zero advantage. The redistribution uses |A|. Groups where all trajectories share one reward have A = 0 and receive no round-level signal from tool credit eq. 7.
  • Judge input. The correctness judge receives the final response with its thinking block removed, split at blank lines into indexed steps app. C.2. The reference solution is provided to the judge, not to the policy app. C.2§2.1.
  • Judge agreement not reported. Both the training reward and the reported evaluation use the GLM-5.2 pessimistic judge §3.1app. C.2. The sections read do not report agreement with human graders or with a second grader.
  • Cost. Each training trajectory can trigger one tool-judge call per non-terminal round and eight final-proof reviews, each with up to 65,536 output tokens app. C.1app. C.2. Compute cost is not reported.
  • Table 2 naming. The "No-Book" column in tab. 2 is the Self-Learning policy with the book removed (71.7%). It is not the No-Book-protocol RL policy (68.5% in tab. 1).
  • Rounding. The text gives a 4.3-point average advantage over No-Book for the fixed baselines before RL §3.2. tab. 1 gives 4.2 (mean of 62.2, 56.7 and 60.9 is 59.9, minus 55.7).
  • Small margins, no variance. The transfer gains are 1.7 and 2.3 points on BrowseComp-Plus and CL-bench tab. 3. The sections read report no RL seed variance and no confidence intervals.
  • Single base model. All RL runs start from Intern-S2-Preview-35B §3.1. The sections read report no other base model.
  • Data and code. No code or data release is stated. The parser, the exercise filter, the search-ranking weights and the BookNavigator environment are not described in enough detail to reproduce exactly §2.2§2.3.
  • Split naming. ID-Book problems come from textbooks that are in the training set, but the 205 problems are held out from training §3.1. The OOD-Book and OOD-Subject splits use textbooks not in the training set §3.1.

Lineage

  • Builds on: GRPO, with leave-one-out advantages and token-level importance weighting §2.4app. C.1. Its group-entropy reweighting and whitespace penalty are inherited from an unnamed advantage estimator app. C.1.
  • Compared against: No-Book, fixed Top-k retrieval, Long-Context and Oracle Section protocols, using the same base model §3.1.
  • Related work named in the paper: Search-R1 and R1-Searcher (search policies trained by outcome RL), KARL, StepSearch, SLATE, IG-Search and OpenClaw-RL (fine-grained agentic RL), and Reflexion, MCTR, LaMer and SEAL (test-time or cross-episode adaptation) §4.
  • Superseded by: not known.

Source and license

Adapted from “Incentivizing Self-Learning in Large Language Models via Reinforcement Learning” by Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen, Wenwei Zhang (arXiv:2610.39671), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.