Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't

In plain words

Large reasoning models get better at maths when they are trained with reinforcement learning: the model tries several answers to a problem, gets a score for each, and is nudged toward the answers that scored better than their peers. The usual recipes for this need big models and large budgets. This paper asks what happens with a small model, here a 1.5-billion-parameter model that already exists, trained on a few thousand carefully chosen maths problems on a modest GPU setup for one day.

The authors use a simple scoring system that needs no learned reward model. An answer is scored by checking whether its final boxed answer matches the known solution, with small bonuses for a clear answer format and a length term that rewards concise correct answers. Their main finding is that small models improve quickly at first, then become unstable and drift when trained longer, for example by producing long, unreadable output or switching languages. The work is a practical map of what helped and what broke, not a new algorithm.

TL;DR

• Small reasoning models (1.5B parameters) can gain a lot on competition-maths benchmarks from a short reinforcement-learning run on a few thousand problems §3.
• Method: GRPO, which scores a group of sampled answers against each other, with rule-based rewards and no critic model §2.
• Author-reported headline: AIME24 46.7% for the Open-RS3 checkpoint, against 28.8% for the starting model and 44.6% for o1-preview tab. 1.
• Cost: about $42 on four A40 GPUs in 24 hours, versus $2,268 to $3,629 for two other 1.5B models tab. 3.
• Limits: quality peaks within 50 to 100 steps and degrades after roughly 150 to 200 steps; truncation at the length cap and language drift are the main failure modes §3 app. B.

Problem

Strong reasoning models are expensive to train, usually with large datasets and multi-GPU budgets §1. Prior small-model reinforcement-learning results used very large datasets (hundreds of thousands to millions of samples) or large compute §1. The paper asks two questions: how small models behave under strict limits on compute and time, and whether a DeepSeek-R1-style reinforcement-learning recipe can raise their reasoning, and if not, what blocks it §1. The stated constraints are 4 NVIDIA A40 GPUs (48 GB each) and a 24-hour window §1 §3.

Method

Overview. The base model is DeepSeek-R1-Distill-Qwen-1.5B. The authors skip supervised fine-tuning and apply reinforcement learning directly §3. They train with GRPO, the algorithm used in DeepSeek-R1 §2, and build a small maths dataset from two public sources §2. The plan was one epoch, but the run was stopped at 500 steps by the time budget §3 app. E.

Dataset curation §2, Dataset Curation.

1. Take the s1 reasoning corpus of 59,029 questions and keep only questions whose solutions contain a boxed answer: 31,323 remain §2, s1 Dataset.
2. Drop trivial questions with DeepSeek-R1-Distill-Qwen-1.5B: 21,533 remain.
3. Drop noisy or multi-part questions with Qwen2.5-7B-Instruct: 18,615 remain. This set is called open-s1 §2, s1 Dataset.
4. Take DeepScaleR (40,315 maths questions) and drop easy questions with Qwen2.5-Math-7B-Instruct: 21,044 remain. This set is called open-deepscaler §2, DeepScaleR Dataset.
5. Combine both sets: 39,659 questions in total §2, Final Dataset.

The three experiments use subsets of this data. Experiment 1 uses open-s1 only (18,615 samples). Experiments 2 and 3 use 7,000 samples: 3,000 from open-s1, 3,000 from open-deepscaler and 1,000 easier problems from raw DeepScaleR §3.

Reinforcement learning §2, Reinforcement Learning Algorithm. For each question, sample a group of G outputs from the old policy and update the policy to maximise a clipped, group-normalised objective minus a KL penalty to the reference model eq. 1 eq. 2. The advantage of each output is its reward standardised within the group eq. 3. G is 6 §3 tab. 5. The clipping range epsilon and the KL weight beta are named but their values are not stated §2.

# One GRPO update for one question  [§2] [eq. 1]
# Input:  question; policy; old_policy; ref_policy; reward_fns (weighted)
# Output: objective value (the optimiser minimises its negative)

def grpo_objective(question, policy, old_policy, ref_policy, reward_fns):
    # Sample G = 6 completions from the old policy  [§3] [tab. 5]
    outputs = sample(old_policy, question, num_generations=6)
    # Weighted sum of rule-based rewards  [§2 Reward Models]
    # [code: recipes/grpo.yaml#L46-L51]
    rewards = [sum(w * f(o) for f, w in reward_fns) for o in outputs]
    # Standardise rewards within the group  [eq. 3]
    advantages = (rewards - mean(rewards)) / std(rewards)
    # Collect one objective term per output
    terms = []
    for output, advantage in zip(outputs, advantages):
        # Probability ratio of the new policy to the old one  [eq. 1]
        policy_ratio = prob(policy, output) / prob(old_policy, output)
        # Clipped surrogate, taking the pessimistic minimum  [eq. 1]
        surrogate = min(
            policy_ratio * advantage,
            clip(policy_ratio, 1 - eps, 1 + eps) * advantage,
        )
        # KL estimate with ratio = ref / policy: r - log(r) - 1  [eq. 2]
        ref_ratio = prob(ref_policy, output) / prob(policy, output)
        kl = ref_ratio - log(ref_ratio) - 1
        terms.append(surrogate - beta * kl)
    # Average over the G outputs  [eq. 1]
    return mean(terms)

Rewards §2, Reward Models. Three rule-based components, with no neural reward model:

• Accuracy: binary, 1 if the answer in the final boxed expression matches the reference, else 0. Unparseable references get 1 so that the example is skipped §2 rewards.py.
• Format: a bonus when the reasoning is closed by a think tag §2 rewards.py.
• Cosine: scales the correctness reward by response length on a cosine schedule. Shorter correct answers score higher; the paper says longer incorrect answers are penalised less §2. The code's values and behaviour are in Pitfalls.

# Cosine length-scaled reward  [§2 Reward Models]
# Input:  completion text; gold answer; config (see Hyperparameters)
# Output: scalar reward

def cosine_reward(completion, gold, config):
    # Correctness, checked with math-verify against the gold answer
    # [code: src/open_r1/rewards.py#L301-L320]
    is_correct = verify(parse(completion), parse(gold))
    # Length in characters, not tokens  [code: src/open_r1/rewards.py#L321]
    progress = len(completion) / config.max_len
    cosine = cos(progress * pi)
    # Pick the value range; wrong answers swap min and max
    # [code: src/open_r1/rewards.py#L327-L333]
    if is_correct:
        low, high = config.min_value_correct, config.max_value_correct
    else:
        low, high = config.max_value_wrong, config.min_value_wrong
    # Cosine interpolation: high at length 0, low at progress 1
    # [code: src/open_r1/rewards.py#L335]
    return low + 0.5 * (high - low) * (1.0 + cosine)

Experiments §3. Three runs, each started from the same base model:

• Experiment 1: open-s1 data, accuracy and format rewards, completions up to 4,096 tokens §3.
• Experiment 2: the 7,000-sample mix, accuracy and format rewards, completions up to 3,584 tokens §3.
• Experiment 3: the same 7,000 samples, with the cosine reward replacing the accuracy reward, completions up to 3,584 tokens. The system prompt also asks for English only §3.

Checkpoints taken for the comparison are Open-RS1 at 100 steps (Experiment 1), Open-RS2 at 50 steps (Experiment 2) and Open-RS3 at 50 steps (Experiment 3) §3.

Hyperparameters

Values are from the paper's appendix table tab. 5 and the repository recipe where both exist. "not stated" means neither the paper nor the code sets the value.

Results

Author-reported. Setup: zero-shot pass@1 on five maths benchmarks (AIME24, MATH-500, AMC23, Minerva, OlympiadBench), evaluated with lighteval; baselines are 1.5B and 7B models, plus o1-preview and Llama-3.1-70B-Instruct §3 app. C app. D. Baseline scores are taken from other papers, not re-run tab. 1 caption.

Bold marks the best score per column among the rows shown in the paper. Averages are over all five benchmarks.

Training cost and data tab. 2 tab. 3. Costs are the paper's estimates from RunPod pricing §3.

For the 7B models tab. 2: Eurus-2-7B-PRIME was trained on 1 × 8 A100 80 GB for 72 h at about $1,088, and Qwen2.5-7B-SimpleRL on 4 × 6 A100 80 GB for 36 h at about $1,633. rStar-Math-7B used a reward model and 7.3M SFT examples on a larger cluster, with no cost estimate. The paper's cost and data figures for the other models come from their papers or GitHub issues tab. 2 tab. 3 captions.

Experiment dynamics §3 fig. 2 fig. 3 fig. 4 fig. 5.

• Experiment 1 (open-s1): AMC23 rose from 63% to 70% and MATH-500 from 83% to 84% in the first 50 to 100 steps. After 200 steps, AMC23 fell below 60% and MATH-500 to 80%. Completion lengths first dropped to about 3,000 tokens, then rose again with unreadable and non-English output §3 fig. 3.
• Experiment 2 (7k mix): completion lengths dropped to about 2,800 tokens. AMC23 rose from 63% to 80% and MATH-500 from 83% to 85% within 50 to 100 steps. After 150 to 200 steps, performance declined, KL became unstable and mixed-language output returned §3 fig. 4.
• Experiment 3 (cosine reward, English prompt): completion lengths stayed between 1,000 and 3,500 tokens. AMC23 rose to 72.5% and MATH-500 to 84.4% within 50 steps, behind Experiment 2's peak. Mixed-language content still appeared after 200 steps §3 fig. 5.

The paper states three insights: rapid early gains that fade with long training; a mix of easy and hard problems helps early but does not fix long-run stability; and the cosine reward stabilises length but does not remove language drift §3.

Paper vs code

Code: official repository knoveleng/open-rs at commit f75524b77bb759a6551dd644f145758b9668ef88, MIT license, linked from the abstract.

• Cosine reward has no formula in the paper, but the code has one. The paper describes it only qualitatively §2. The code uses cosine interpolation over length in characters with a maximum of 1,000 characters rewards.py. Paper and code do not say the same thing about the length unit.
• Direction for wrong answers differs. The paper says longer incorrect solutions are penalised less §2. With the training defaults (wrong: min 0.0, max -0.5) grpo.py, a wrong answer of length 0 gets 0.0 and one of length 1,000 characters gets -0.5, so within that range longer wrong answers are penalised more. The code's docstring agrees with the paper rewards.py. This comes from reading the arithmetic, not from a run.
• Format reward is narrower than described. The paper describes tags around the reasoning §2. The code gives credit only when the text contains exactly one closing think tag preceded and followed by a newline; it does not check the opening tag or the answer tags rewards.py. The system prompt, not the reward, asks for the opening tag.
• Reward configuration in the repository matches only Experiment 3. The paper says Experiments 1 and 2 used accuracy and format rewards §3. The recipe lists format and cosine only grpo.yaml. No recipe for the accuracy-plus-format runs was found in the repository. The code default list does include accuracy grpo.py.
• Training loop is the open-r1 GRPO trainer on a pinned TRL commit. The paper describes GRPO in the abstract form §2 eqs. 1–3. Whether the importance ratio is computed per token or per sequence is not stated; the code uses the TRL trainer at a pinned commit setup.py, whose behaviour was not checked here.
• Evaluation in the code is narrower than the paper's table. eval.sh runs AIME24, MATH-500 and AMC23 by default; Minerva and OlympiadBench are commented out eval.sh. The repository's logs directory does contain a Minerva result for Experiment 1 at 100 steps, scored 0.2904 (29.0%), which matches the paper's 29.0 results_2025-03-15T18-56-57.793761.json. An OlympiadBench result was not checked.
• Checkpoint revisions. Each Open-RS checkpoint has a model-hub revision in eval.sh; for example, Experiment 1 at 100 steps is pinned to commit 0439c48 eval.sh. The revision maps the table rows to saved weights.
• Evaluation settings differ from training. Evaluation samples at temperature 0.6 with top-p 0.95 and up to 32,768 new tokens eval.sh. Training used 0.7 and a cap of 3,584 or 4,096 tokens tab. 5. The paper does not state its evaluation sampling settings.
• Data-filter recipe covers only one model. recipes/data_cleaner.yaml configures Qwen2.5-Math-7B-Instruct, which the paper uses for the DeepScaleR filter data_cleaner.yaml. The repository contains no config for the s1 filtering steps, which the paper attributes to DeepSeek-R1-Distill-Qwen-1.5B and Qwen2.5-7B-Instruct §2. This is a finding from the file list, not an exhaustive search.
• Hyperparameters in the paper's table and the recipe agree for every value that appears in both (learning rate, schedule, batch sizes, G, temperature, lengths, seed, save interval, reward weights) tab. 5 grpo.yaml.

Pitfalls

• Cosine reward wraps around. The code uses cos(length / max_len * pi). Beyond 1,000 characters the cosine turns back up, and at 2,000 characters a completion gets the same reward as an empty one rewards.py. The reward is not clamped to the 0 to 1,000 range. Lengths in the paper are in tokens, so the characters-to-tokens mapping was not verified. Any reimplementation must choose a unit and decide whether to clamp. Expect a non-monotone signal for long outputs.
• Scores for unparseable references. When the gold answer cannot be parsed, both accuracy and cosine rewards return 1.0 for that example rewards.py rewards.py. Those examples are effectively rewarded for any output. The paper does not mention this.
• Answer extraction is strict. The accuracy check prefers boxed expressions and disables fallback extraction without an anchor §2 rewards.py. An answer without a boxed expression may therefore score 0; this was not tested.
• Advantage standardisation is ambiguous. The paper writes the standard deviation of the group rewards eq. 3. It does not say whether this is the population or sample deviation, and the choice matters for small groups of six §2. The paper does not say what happens when all six rewards in a group are equal, where the standard deviation is zero.
• Clip and KL values are missing. Epsilon and beta are not given in the text or the recipe §2 grpo.yaml. Reproducing the stability findings requires choosing them.
• Length cap truncates hard problems. Completions that hit the cap before an answer cannot be verified, and the paper links part of the early degradation to outputs exceeding the cap §3 app. B.
• Multilingual drift. The base model is multilingual. An English instruction in the prompt reduced but did not remove mixed-language output §3 app. B. The authors suggest a language reward or monolingual pre-filtering as untested fixes app. B.
• Costs are estimates and do not include engineering time. The $42 figure is a rental-price estimate on RunPod for 24 hours on four A40 GPUs §3 tab. 3. Baseline costs come from other papers and are not comparable to a re-run.
• Small evaluation sets. AIME24 has 30 problems and AMC23 has 40 app. C tab. 4. One problem changes the score by 3.3 points on AIME24 and 2.5 points on AMC23. The reported differences between checkpoints are within a few problems.
• Average includes benchmarks the default eval script does not run. The Avg. column averages all five benchmarks tab. 1; the committed eval.sh runs three by default eval.sh.
• Dataset subsets for Experiment 2 are not fully specified. The paper gives 3,000 open-s1, 3,000 open-deepscaler and 1,000 easier DeepScaleR problems §3, but not how the 1,000 easy problems were chosen beyond "easier", and the released dataset was not inspected.
• Insights rest on single runs. The paper reports no error bars and no repeated seeds §3. Differences of a few points between Open-RS variants should not be read as settled.

Lineage

• Builds on: DeepSeek-R1, which uses the RL recipe with GRPO and is the source of the base model arXiv:2501.12948 §1.
• Builds on: GRPO, as introduced in DeepSeekMath arXiv:2402.03300 §2.
• Data: the s1 dataset arXiv:2501.19393 §2 and DeepScaleR, cited as a GitHub project rather than an arXiv paper §2 app. D.
• Code base: the Open-R1 reproduction of DeepSeek-R1 by Hugging Face setup.py §3.
• Compared against: Llama-3.1-70B-Instruct, o1-preview, Qwen-2.5-Math-7B-Instruct, rStar-Math-7B, Eurus-2-7B-PRIME, Qwen2.5-7B-SimpleRL, DeepScaleR-1.5B-Preview and Still-3-1.5B-Preview §3 tab. 1.
• Superseded by: not stated in the paper, and no later card in this wiki.