Large reasoning models

Overview

A large reasoning model (LRM) is a language model that produces a long intermediate reasoning trace before its final answer. Unlike chain-of-thought prompting, where the user asks for steps, the reasoning behaviour is part of the model itself. The o1 family is the example the first three papers study: the first is described as building chain-of-thought reasoning into inference 2409.18486 §1.1, the second compares "thinking" models with non-thinking twins 2506.06941 §4.1, and the third is the pre-release system card for o1 and o1-mini 2412.16720 §1.

The next three change or shorten a model's trace. The fourth trains its own small reasoning model with reinforcement learning, which is the one case where the weights are updated from a base model 2503.16219 §1. The fifth takes a student's own long traces, has a teacher prune them, and fine-tunes the student on the result 2505.13975 §3. The sixth needs no teacher: it scores each step of a trace by the model's own uncertainty, masks the most predictable steps, and can train a model to emit the masks itself 2508.03346 §3.2 2508.03346 §3.3.

The seventh is a survey, not a method. It asks whether sudden jumps in LLM ability, including the jumps seen with reasoning models, are true emergence. It defines emergence as an ability that is absent in smaller models and not predicted by a curve fitted to them, and it argues that an abrupt curve alone does not establish it 2503.05788 §2. Its section on reasoning models quotes results from other studies and does not re-run them 2503.05788 §5.

The shared question is whether the extra reasoning helps, where it helps, what it does to safety, and whether it can be shortened. Three of the seven papers are evaluations or a system card, one is a survey, and three report methods for training or shortening traces. So much of this page is about how to measure LRMs and what the measurements show; the rest is about how to make traces shorter and what those shorter traces cost in accuracy. The invariant that matters for users is that the answer is scored, but the trace can be inspected. The paper that looks at traces finds that a correct intermediate answer is not always the one the model commits to 2506.06941 §4.3, and the system card runs a monitor over traces for deception 2412.16720 §4.3.1.

Variants

"Variant" here means a way of testing or building an LRM. The papers do not offer competing models that a reader can choose between. The survey is not a method and has no variant of its own; it is discussed under Overview and Contradictions.

Broad task survey, graded by hand

The authors wrote 27 tasks across reasoning, planning, creation, diagnosis and reflection, then ran o1-preview on them 2409.18486 §1.4. Most tasks are graded by the authors, and the samples are small. Coding is the exception: each problem allows three submissions, and it counts as solved if one passes the judge 2409.18486 §2.1. Card: Evaluation of OpenAI o1: Opportunities and Challenges of AGI.

Controllable puzzles with a simulator

Four rule-based puzzles (Tower of Hanoi, Checker Jumping, River Crossing and Blocks World) each have a difficulty knob N. A simulator checks every move against the rules and the goal state 2506.06941 §3.1 2506.06941 app. A.2. Thinking/non-thinking pairs are compared at matched token budgets, and 25 samples are taken per model, puzzle and size 2506.06941 §4.1. Card: The Illusion of Thinking.

Small-model reinforcement learning on maths

The base model is DeepSeek-R1-Distill-Qwen-1.5B, with no supervised fine-tuning before reinforcement learning 2503.16219 §3. Training uses GRPO, which samples a group of answers per question and scores each against the group average, with rule-based rewards and no learned reward model 2503.16219 §2. The training set is about 7,000 maths problems, run for 24 hours on four A40 GPUs 2503.16219 §3 2503.16219 tab. 2. Gains come early and fade with longer training: AMC23 rose from 63% to 80% within 50 to 100 steps in one run, then declined after 150 to 200 steps 2503.16219 §3. Card: Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't.

Prune and distil the student's own trace

The student model writes a long response first. A teacher model (GPT-4o in the main runs) splits the reasoning into steps, each labelled with a skill. The teacher then keeps, deletes or compresses each step and rewrites the survivors as one trace 2505.13975 §3.1 2505.13975 §3.2. The student is fine-tuned with LoRA on the pruned trace and answer, using 8,000 samples from GSM8K and PRM12K 2505.13975 §3.3 2505.13975 app. B. The paper's argument is that a concise teacher's output can be hard for the student to learn from, and that editing the student's own style avoids much of that loss 2505.13975 §1. Card: DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models.

Entropy-ranked step masking, with optional training

The model writes a full trace first. The thinking is split into steps at blank lines, and each step is scored by the mean entropy of its tokens, using the model's own distribution over the vocabulary at each position 2508.03346 §3.2 2508.03346 eq. 1 2508.03346 eq. 3. The lowest-scoring 80% of steps are replaced by a [SKIP] marker and the rest are kept verbatim 2508.03346 §4.1. On DeepSeek-R1-Distill-Qwen-7B, this static masking cut thinking tokens by 29.7% on MATH500 and 36.3% on AIME 2024, and accuracy moved by under a point on MATH500 but fell by about 7 points on AIME 2024 2508.03346 tab. 1. Across the three models on MATH500, AIME 2024 and AIME 2025 in that table, token cuts run from 27% to 45%; on GSM8K they are only 1% to 2% 2508.03346 tab. 1. The removal direction is the evidence: removing the highest-entropy steps lowers accuracy at once, while removing the lowest-entropy ones does not 2508.03346 §4.1 2508.03346 fig. 1.

The second stage teaches a model to emit [SKIP] without the masking step. It is fine-tuned on masked traces, then trained with GRPO on a reward that combines a correct answer, a share of skips, and penalties for too many skips or overly long answers 2508.03346 §3.3. The trained DeepSeek-R1-7B used 35% to 57% fewer tokens than its full trace across GSM8K, MATH500, AIME 2024 and AIME 2025, with accuracy changes from +0.6 to −6.2 points on the same table 2508.03346 tab. 3. On the full DeepScaleR and OpenR1-Math sets, static masking lost 1.6 and 1.1 points for DeepSeek-R1-7B 2508.03346 tab. 2. Card: Step-entropy compression of chain-of-thought.

Pre-release safety battery (system card)

The authors run a set of safety tests before release: refusals, known jailbreaks, made-up facts, biased answers, conflicts between instruction levels, a monitor on chains of thought, outside red teams, and risk scores in four areas 2412.16720 §4 2412.16720 §5. The training method is named but not specified; deliberative alignment is cited to a separate paper 2412.16720 §5.3. All numbers are author-reported on pre-release checkpoints 2412.16720 §3. Card: OpenAI o1 System Card.

When to use which

Implementation sketch

Three loops are sketched below: an evaluation loop that matches the puzzle protocol, the step-masking rule, and the group-relative update used in the small-model training run. The pruning variant builds training data rather than running a loop, so it is described under Variants. The first loop samples with a fixed token cap, drops answers in the wrong format, and replays every move through the simulator 2506.06941 §4.1 2506.06941 app. A.3. Set the cap and temperature yourself, since they change results.

# Evaluate one model on one puzzle at one size  [arxiv:2506.06941 §4.1]
# Input:  puzzle, size N, model, token cap, samples
# Output: pass rate and first failing move per sample

def evaluate(puzzle, size, model, cap, samples):
    # Build the prompt for this instance  [arxiv:2506.06941 app. A.2]
    prompt = render_prompt(puzzle, size)
    results = []
    # Draw independent samples at the same settings  [arxiv:2506.06941 app. A.3]
    for _ in range(samples):
        answer, trace = model.generate(prompt, max_tokens=cap)
        # Keep only answers in the requested format  [arxiv:2506.06941 §4.1]
        moves = parse_moves(answer)
        if moves is None:
            continue
        # Replay each move and check the goal  [arxiv:2506.06941 app. A.2]
        results.append(simulator.check(puzzle, size, moves))
    return results

The second is the step-masking rule. It uses the entropy of the model's own distribution at each token, averages it within each blank-line step, and masks the lowest-scoring steps 2508.03346 §3.2 2508.03346 eq. 1 2508.03346 eq. 3.

# Mask the lowest-entropy steps of one finished trace  [arxiv:2508.03346 §3.2]
# Input:  steps split at blank lines, per-token distributions, kappa
# Output: the trace with the chosen steps replaced by [SKIP]

def mask_low_entropy_steps(steps, kappa=0.8):
    # Entropy of each token's full-vocabulary distribution, in bits
    # [arxiv:2508.03346 eq. 1]
    # Score each step by the mean of its token entropies
    # [arxiv:2508.03346 eq. 3]
    scores = [mean(token_entropy(t) for t in s.tokens) for s in steps]
    # Number of steps to mask; the paper does not say how this is
    # rounded, so int() floors here  [arxiv:2508.03346 §3.2]
    k = int(kappa * len(steps))
    # Mask the k lowest scores; keep the rest verbatim  [arxiv:2508.03346 §3.2]
    lowest = set(sorted(range(len(steps)), key=lambda i: scores[i])[:k])
    return "".join(
        "[SKIP]" if i in lowest else s.text for i, s in enumerate(steps)
    )

The third is the group-relative update used in the small-model training run. It replaces a learned critic with a score computed within each group of sampled answers 2503.16219 §2 2503.16219 eq. 3.

# Group-relative advantages for one question  [arxiv:2503.16219 §2] [arxiv:2503.16219 eq. 3]
# Input:  G sampled answers to one question, each with a rule-based reward
# Output: one advantage per answer, used to weight its policy-gradient term

def group_advantages(rewards):
    # Group size G is 6 in the paper's runs  [arxiv:2503.16219 §3]
    mean_r = sum(rewards) / len(rewards)
    # Group standard deviation; sample or population form not stated
    # [arxiv:2503.16219 eq. 3]
    std_r = std(rewards)
    # Standardise each reward within its own group
    return [(r - mean_r) / std_r for r in rewards]

Points that vary by paper and need stating in any write-up:

• The token cap and temperature. The main runs use a 64,000-token cap and temperature 1.0; a temperature-0 ablation changes the collapse point 2506.06941 app. A.3 2506.06941 app. A.6.
• Format filtering. Discarded samples are not counted, and the paper does not report how many were dropped 2506.06941 §4.1.
• For coding, whether later attempts see the judge's feedback. The paper does not say 2409.18486 §4.2.
• For training, the clipping range and the KL weight. Neither value is stated in the paper 2503.16219 §2, so a reproduction has to choose them.
• For training, the reward for long answers. The paper describes a cosine length reward only in words, with no formula 2503.16219 §2. Its length cap also truncates hard problems before an answer is reached 2503.16219 §3 2503.16219 app. B.
• For training, output language. The base model is multilingual, and an English instruction reduced but did not remove mixed-language output 2503.16219 §3 2503.16219 app. B.
• For masking, the entropy definition. The paper defines it over the full vocabulary 2508.03346 eq. 1. The released code is reported to use a different quantity, which can rank steps differently; see the paper card before reproducing.
• For masking, the step boundary and the rounding. Steps are split at blank lines 2508.03346 §3.2, and the paper does not say how the count of masked steps is rounded 2508.03346 §3.2.
• For masking, the [SKIP] token. The paper treats it as one token 2508.03346 app. E, but does not say how the tokenizer is set up for it, so token counts may differ between stages.
• For masking training, the group size and KL weight. The appendix gives 14 samples per prompt and a KL coefficient of 0.04 2508.03346 app. C. The skip-count penalty is −1.0 in the text 2508.03346 §3.3.
• For pruning, the token cutoff. The paper uses a 12k-token cutoff when counting tokens, but does not say whether longer generations are truncated, counted at 12k, or dropped from the average 2505.13975 §4.6 2505.13975 app. A.
• For pruning, the answer check. The described pipeline has no filter that drops pairs with a wrong final answer; the teacher may revise the answer to match the pruned trace 2505.13975 §3.2.
• For pruning, the loss and optimiser. The loss is written as a sum over target tokens, and whether the trainer averages it is not stated 2505.13975 §3.3. LoRA rank, alpha and dropout are also not stated 2505.13975 app. B.
• For pruning, the evaluation runs. The paper reports Pass@1 averaged over five runs but does not describe the sampling that would make the runs differ 2505.13975 §4.6.
• For pruning, the teacher's terms. The paper uses OpenAI and Gemini outputs as training targets and does not discuss whether their terms allow training other models on them 2505.13975 §9. Check this before any commercial use.
• For safety monitoring, the judge prompt. The chain-of-thought monitor is a GPT-4o judge whose prompt is not given 2412.16720 §4.3.1, and several safety scores use model-based graders 2412.16720 §4.1.1.

Contradictions

• Boat capacity in River Crossing. The main text and appendix give capacity 2 for N ≤ 3 and 3 above that 2506.06941 §3.1 2506.06941 app. A.2.3. The rebuttal gives capacity 4 for N ≥ 6 and restricts the analysis to N < 6 2506.06941 app. A.1. A reimplementation has to pick one.
• Failure position for Hanoi at N = 8. The rebuttal puts the first failure near move 40, another appendix near move 50, and §4.4 gives more than 100 moves for N = 10 2506.06941 app. A.1 2506.06941 app. A.6 2506.06941 §4.4. Treat the exact figures as approximate.
• Strength of the o1 claims. The abstract of the first paper describes o1-preview as outperforming other models on radiology reports, but its table shows gpt-4-turbo with higher ROUGE-2 2409.18486 tab. 3. Report the table, not the abstract.
• Teacher claim against the teacher table. The pruning paper says every teacher improves both accuracy and token count over the base model 2505.13975 §6.3. Its teacher table has ChatGPT below the base on GSM8K (91.2% against 91.7%) and Gemini, ChatGPT and DeepSeek-V3 below it on MATH500 2505.13975 tab. 3.
• Training-trace length. The pruning paper gives an average of 330 tokens for its training traces and 186 for direct distillation 2505.13975 §6.2, and reports 328 tokens for DRP on GSM8K at evaluation 2505.13975 tab. 1. The paper does not say whether these measure the same thing.
• Baseline tokens within the masking paper. The full-trace GSM8K token count for DeepSeek-R1-7B is 298.33 in its main table 2508.03346 tab. 1 and 296.09 in its reward ablation 2508.03346 tab. 5. The gap is small, but the paper does not explain it.
• o1 on AIME 2024, across papers. The survey quotes 83.3% for o1 and 13.4% for GPT-4o, taken from other studies 2503.05788 §5. The RL paper quotes 44.6% for o1-preview, also from another paper 2503.16219 tab. 1. These are different models and sources, so they do not directly conflict, but they should not appear in one table without naming the model.
• Across the papers. The first reports strong results on high-school and competition maths 2409.18486 §4.14 2409.18486 §2.1. The third reports 46.7% on AIME24 for a 1.5B model 2503.16219 tab. 1, the second reports collapse on hard puzzles 2506.06941 §4.2, the fifth reports 15 of 30 on AIME24 for a 7B model after pruning 2505.13975 tab. 1, and the sixth reports 63.33% on AIME 2024 for the full trace of a 7B model and 56.67% after static masking 2508.03346 tab. 1. These use different tasks, models, sample counts and metrics, so they do not directly conflict. Do not combine them into one accuracy claim.
• Low flag rate against high scheming rate. The system card flags 0.17% of chains of thought as deceptive 2412.16720 §4.3.1, yet in the Apollo scheming tests o1 deceived on follow-up in 0.98 of runs where it took the scheming action 2412.16720 tab. 10. The two measure different things, so they are not a formal conflict, but neither figure estimates how often o1 deceives in agentic work. The o1-preview rate in the same table is 0.00.
• The o1-preview baseline in the third paper. Its 44.6% on AIME24 is taken from another paper and was not re-run 2503.16219 tab. 1. It is not a measurement from the first paper, which reports its own o1-preview results on different tasks 2409.18486 tab. 2.

Papers

• Evaluation of OpenAI o1: Opportunities and Challenges of AGI: a broad, author-graded evaluation of o1-preview on 27 tasks, with a coding protocol and radiology ROUGE scores.
• OpenAI o1 System Card: pre-release safety report on o1 and o1-mini, covering refusals, jailbreaks, hallucination, bias, instruction hierarchy, a deception monitor, red teaming and risk scores.
• The Illusion of Thinking: matched thinking/non-thinking comparison on four controllable puzzles, with a simulator and an analysis of intermediate solutions.
• Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't: GRPO with rule-based rewards on a 1.5B reasoning model, with the gains and the late-training instability reported.
• DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models: a teacher splits, prunes and rewrites a student's long traces, then the student is fine-tuned with LoRA, with token savings on maths benchmarks.
• Step-entropy compression of chain-of-thought: scores blank-line steps by token entropy and masks the lowest 80%, with static results on DeepSeek-R1 and Qwen3 and an SFT-plus-GRPO model that learns to skip steps itself.
• Emergent Abilities in Large Language Models: A Survey: a survey of what "emergence" means for LLMs, whether sudden jumps survive metric changes, and what reasoning models and agents add to the question.