--- kind: concept slug: reinforcement-learning-verifiable-rewards title: Reinforcement learning with verifiable rewards (RLVR) summary: >- Fine-tuning by sampling answers and scoring each with a rule that can be checked automatically (IoU, class match, boxed maths answer, solver objective, kernel speed, simulated lap error, final answer of a worker chain, success in a 3D game task), then favouring answers that do better than their peers. Author-reported gains. Needs a checkable output. tags: [reinforcement-learning, post-training, multimodal, video, math-reasoning, diffusion-llm, code-generation, gpu-kernels, multi-agent, orchestration, embodied-ai, knowledge-graph] papers: ["2503.01785", "2504.06958", "2605.10889", "2503.16219", "2507.08838", "2510.02245", "2505.11792", "2512.02551", "2512.04388", "2505.03238", "2512.04797", "2505.15062"] related: [policy-gradient, rlhf, large-reasoning-models, multimodal-language-model, knowledge-distillation, on-policy-distillation, chain-of-thought-prompting, experience-replay, importance-weighting, discrete-diffusion, dataset-curation, self-consistency, tool-use, low-precision-arithmetic, multi-agent-systems, model-routing, model-predictive-control, low-rank-adaptation, vision-language-action-model, llm-agent, knowledge-graph-reasoning] status: draft updated_at: 2026-11-01 --- ## Overview The model answers a prompt several times. Each answer is scored by one or more rules that do not need a learned judge: an IoU for a box, an exact class match, a boxed final answer for maths, a cell-by-cell score for a puzzle, the objective a solver returns for an optimisation model, the measured speed of a GPU kernel, the tracking error of a simulated car lap, whether a game task was completed, or whether a knowledge-graph answer matches the gold answer. Most variants then compare each answer with the others sampled for the same prompt, so the ones that beat their siblings get pushed up and the rest down [arxiv:2503.01785 §3.1] [arxiv:2504.06958 §3.1] [arxiv:2503.16219 §2]. The shared idea is that a check runs on the output, rather than a learned judge deciding the score. VideoChat-R1's caption reward is an exception, since an LLM judge computes it [arxiv:2504.06958 §3.3], and so is SIMA 2's self-improvement loop, where a Gemini model scores each trajectory [arxiv:2512.04797 §4.5]. Most variants use GRPO or a close relative; SIRL uses REINFORCE++ instead [arxiv:2505.11792 §3.2], and SIMA 2 does not name its algorithm [arxiv:2512.04797 §3.3.3]. The method papers are not carded here. Papers differ on two training details: whether the group spread is divided out of the advantage, and whether a KL penalty keeps the policy near a reference model. Both are covered under Implementation sketch. Three of the papers keep a KL penalty [arxiv:2503.01785 §3.1] [arxiv:2504.06958 §3.1] [arxiv:2503.16219 §2], and SAKE's objective has one too, with its weight not stated in the paper [arxiv:2505.15062 eq. 8]. wd1 sets its weight to zero [arxiv:2507.08838 tab. 6], ExGRPO's objective has none [arxiv:2510.02245 §4.2], and the Conductor sets its KL weight to zero [arxiv:2512.04388 app. A]. SIRL keeps KL only on its model and code sections, not on its reasoning [arxiv:2505.11792 eq. 5]. SIMA 2 does not mention a KL term, and its RL algorithm is not stated [arxiv:2512.04797 §3.3.3]. The vision and video cards cover vision-language and video models. The maths and diffusion cards cover text-only reasoning. The two code cards cover output that a solver or a GPU runs: SIRL writes optimisation models, and CUDA-L2 writes matrix-multiply kernels. The RobotxR1 card covers a robot controller: the output is a parameter set for a model predictive controller, and the reward comes from a simulated lap. The SAKE card covers a model that calls two knowledge-graph tools during its rollout, and the reward reads its final answer tag. The Conductor card applies the same kind of check one level up: the scored output is a plan for other models, and the reward reads the final answer of the chain that the plan runs. The SIMA 2 card covers a game agent whose RL tasks use verification functions. Its self-improvement reward is a model's rubric score, not a rule. All numbers are author-reported. ## Variants ### Rule rewards for vision-language perception (Visual-RFT) Qwen2-VL models are trained with GRPO. For detection, the reward adds an IoU term, a confidence term and a format term; each predicted box is matched to the best unused ground-truth box [arxiv:2503.01785 §3.2.1] [eq. 5] [eq. 6] [eq. 7] [eq. 8]. For classification, the reward is an accuracy term (1 for the right class, else 0) plus the format term [eq. 9]. Card: [Visual-RFT: Visual Reinforcement Fine-Tuning](/papers/2503.01785). Reported results: one-shot fine-grained classification averages 80.3, against 51.7 for supervised fine-tuning and 56.0 for the base model [arxiv:2503.01785 tab. 2]. Two-shot COCO detection reaches 41.5 mAP against 19.6 for the base [arxiv:2503.01785 tab. 3]. On LVIS open-vocabulary detection the 2B model goes from 2.7 to 20.7 mAP [arxiv:2503.01785 tab. 8]. ### Per-task reward mixes for video (VideoChat-R1) Qwen2.5-VL-7B is trained with GRPO, and each task gets its own sum of reward terms: temporal grounding and tracking use format plus IoU; multiple-choice QA uses format plus accuracy; grounded QA uses format, IoU and accuracy; and captioning uses format plus a recall reward that an LLM judge computes [arxiv:2504.06958 §3.2] [arxiv:2504.06958 §3.3] [eq. 2] [eq. 3] [eq. 4] [eq. 5]. Card: [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](/papers/2504.06958). The joint model trains on 18,031 samples from three sets: Charades-STA, GoT-10k and NExTGQA [arxiv:2504.06958 §3.3]. Author-reported: Charades-STA mIoU rises from 29.0 to 60.8 and GoT-10k average overlap from 12.6 to 43.8 over the base model [arxiv:2504.06958 tab. 1]. With one epoch, supervised fine-tuning on Charades-STA reaches 46.3 mIoU and the GRPO run with thinking reaches 59.3 [arxiv:2504.06958 tab. 3]. Three epochs of supervised fine-tuning lower ActivityNet mIoU from 20.6 to 17.3 [arxiv:2504.06958 tab. 3]. ### Small-model maths recipe (Open-RS) A 1.5B reasoning model (DeepSeek-R1-Distill-Qwen-1.5B) is trained with GRPO directly, without supervised fine-tuning. The data is 7,000 curated maths problems, and the rewards are answer correctness, a format bonus and, in the third run, a cosine length-scaled reward [arxiv:2503.16219 §2] [§3]. Card: [Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't](/papers/2503.16219). The run used four A40 GPUs for 24 hours. The authors estimate about USD 42 of rental cost [arxiv:2503.16219 §3] [tab. 3]. Author-reported: AIME24 rises from 28.8 for the start model to 46.7 for the third run at 50 steps [arxiv:2503.16219 tab. 1]. The other two runs score 30.0 on AIME24. The main caution is that the early gains fade: quality peaks within 50 to 100 steps and degrades after about 150 to 200 steps [arxiv:2503.16219 §3]. AIME24 has 30 problems, so one answer moves the score by 3.3 points [arxiv:2503.16219 app. C]. ### Experience replay for on-policy RLVR (ExGRPO) Correct rollouts are stored per question, questions the model always solves are retired, and about half of each batch replays a stored answer. The replayed answer is the one with the lowest mean token entropy under the current policy. Its importance ratio is shaped by f(w) = w / (w + 0.1) [arxiv:2510.02245 §4.1] [§4.2]. Card: [ExGRPO: Learning to Reason from Experience](/papers/2510.02245). Author-reported: the math average on Qwen2.5-Math-7B is 48.3, against 45.3 for on-policy RLVR with the same setup [arxiv:2510.02245 tab. 1]. On Llama3.1-8B, the on-policy run collapses while ExGRPO trains stably. Its math average is 6.1 against 3.4 [arxiv:2510.02245 §5.2] [app. E.3]. The replay starts only after the batch pass rate passes 35% [arxiv:2510.02245 §4.2]. On eight GPUs the average step takes 19% longer [arxiv:2510.02245 app. E.5]. ### Ratio-free weighted loss for diffusion LLMs (wd1) Masked diffusion LLMs define a likelihood only through an iterative denoising process, so the probability of a whole answer cannot be computed exactly. wd1 uses one masked-pass estimate per completion (the d1 estimator), and it needs neither an importance ratio nor a reference model. The loss raises the likelihood of high-advantage completions and lowers it for low-advantage ones, with two softmax weights over the group [arxiv:2507.08838 §3.2] [eq. 8] [eq. 9]. Card: [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](/papers/2507.08838). On LLaDA-8B-Instruct trained with LoRA, author-reported results are: Sudoku 76.4 against 17.6 for the authors' reproduction of d1 at 256 tokens; GSM8K 80.8 against 78.2 at 256 tokens; MATH500 34.4 against 34.4 at 256 tokens and 39.0 against 38.0 at 512 [arxiv:2507.08838 tab. 1]. Each reported number is a single seed [arxiv:2507.08838 tab. 6]. The paper's extended variant, wd1++, uses intermediate denoising steps. Its code is not in the authors' repository [arxiv:2507.08838 §3.3]. ### Solver-checked reward for optimisation models (SIRL) Qwen2.5 7B and 32B instruct models are trained with RL directly, without supervised fine-tuning [arxiv:2505.11792 §3.2] [§4]. Each response has three tagged parts: reasoning, a concise mathematical model, and solver code. The code is run, and the stage-1 reward has three terms: 0.5 for the tags in the right order, 1 if the code runs, and 2 if the objective is within 0.01 of the ground truth [arxiv:2505.11792 §3.3] [eq. 6] [§10]. Stage 2 adds a bonus term, and its weight is not stated [§10]. Card: [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](/papers/2505.11792). Training problems are synthesised from seed cases, augmented, and checked by an LLM judge. The answer is picked by a vote over ten roles, scored on features of each role's solver model [arxiv:2505.11792 §3.1] [eq. 2] [eq. 3]. Problems the 32B baseline solves in 8 of 10 tries are dropped as too easy [§3.1] [§11]. Author-reported macro average over five benchmark columns is 60.6 for the 7B model and 68.3 for the 32B model [arxiv:2505.11792 tab. 1]. In the same table, DeepSeek-V3.1 scores 65.0, so the 7B model trails it on the macro average [tab. 1]. On MAMO Complex, the partial KL reaches 51.7, against 48.3 with full KL and 47.3 with none [arxiv:2505.11792 tab. 3]. Each stage took about 24 hours on 8 H100 GPUs [§11]. ### Speed-measured reward for GPU kernels (CUDA-L2) CUDA-L2 trains an LLM to write half-precision matrix-multiply kernels [arxiv:2512.02551 §1] [§3]. The reward is the mean speedup over timed runs against a reference kernel, minus a penalty for numerical error from an FP32 CPU result, minus a penalty for code length [arxiv:2512.02551 eq. 3] [§3.2.3]. The penalty coefficients are not stated [§3.2.3]. Training has three stages: continued pretraining on CUDA code, RL on about 1,000 general kernels, and RL on matrix multiply alone [arxiv:2512.02551 §3.2]. The base model is DeepSeek 671B [§3.2.1]. Card: [CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning](/papers/2512.02551). Author-reported, on an A100 over 1,000 matrix shapes and offline timing: a mean speedup of +22.0% over torch.matmul and +11.4% over the auto-tuned cuBLASLt baseline, in its best layout [arxiv:2512.02551 tab. 1]. The speedup is a mean of per-run time ratios minus one [§2.4] [eq. 2]. The gain falls with problem size, from about 1.4× on small problems to about 1.0× on the largest, against cuBLASLt auto-tuning [§4.3]. The paper's exact correctness test covers only outputs below 2048; larger outputs are bounded indirectly [§2.3]. ### Simulated-lap reward for a robot controller (RobotxR1) Qwen2.5 1.5B and 3B models are trained first with SFT, then with GRPO on LoRA adapters [arxiv:2505.03238 §3.3] [§4.1]. For the controller module, the model gets an instruction such as "reverse the car" and writes a dictionary of parameters for a model predictive controller (MPC) on a 1:10 scale racing car. The dictionary is applied, the car drives a lap in a simulator, and the driving term is the relative improvement of the tracking error over the default MPC, clipped below at -4. A format term and a parameter term are added to it [arxiv:2505.03238 eq. 3]. Card: [RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning](/papers/2505.03238). Author-reported, on one evaluation track with 20 paraphrased prompts and five runs [arxiv:2505.03238 app. C] [tab. 2]: the 3B model goes from 50.4% after SFT to 63.3% after SFT and GRPO, against 58.5% for GPT-4o. Without SFT, GRPO alone takes the 3B model from 34.7% to 57.2%. The 1.5B model goes from 11.3% to 41.5% with SFT and GRPO. The SFT-only 3B figure leaves out one extraction failure [tab. 2]. The same paper trains a yes or no decision module with GRPO. Its 3B accuracy goes from 83.78% after SFT to 86.82% after SFT and GRPO [arxiv:2505.03238 tab. 1]. The GRPO run on the 3B model used about 11 GB of GPU memory and about 24 hours [§4.1]. The paper evaluates on a different track from the one it trains on, so the transfer is part of the claim [app. B]. It reports no confidence intervals [tab. 2]. ### Plan-level reward for multi-model workflows (Conductor) A 7B planner, starting from Qwen2.5-7B, is trained with GRPO to write a plan for a pool of worker LLMs [arxiv:2512.04388 app. A] [§4.1]. Each plan has three equal-length lists: the worker ids, a natural-language subtask for each step, and an access list that says which earlier steps that worker may see [arxiv:2512.04388 §3.1] [app. E]. The reward is 1 if the final answer of the executed chain is correct, 0.5 if it is wrong but the plan parses, and 0 if the three lists cannot be parsed [§3.1]. Card: [Learning to Orchestrate Agents in Natural Language with the Conductor](/papers/2512.04388). Author-reported, unconstrained: on LiveCodeBench v6 the Conductor scores 83.93, against 82.90 for GPT 5, the best single worker on that task [arxiv:2512.04388 tab. 1]. Under a shared cap of 4,096 output tokens and minimal reasoning budgets, it averages 72.35 over four training tasks, against 64.14 for the best single model [tab. 7]. The paper reports about three steps per question on average [§4.3], against eight calls for Mixture-of-Agents [app. A]. The unconstrained baselines are each model's best across the authors' runs and online leaderboards, so they are a reported frontier and not a controlled comparison [§4.2]. Allowing the planner to call itself raises the BigCodeBench score from 37.8 to 40.0 on one run [tab. 2]. The recursive training questions were already seen in training [§4.4], so this does not measure generalisation to new questions. ### Knowledge-graph tool calls inside the rollout (SAKE) Qwen2.5 3B and 7B instruct models are trained with GRPO, with no supervised stage [arxiv:2505.15062 §3.4] [§3.5]. Each rollout has three model turns and two deterministic tool calls in between. The first turn names the entities in the question. Tool 1 returns, for each entity, a group made of that entity and its most similar knowledge-graph concepts. The second turn keeps some of these groups. Tool 2 returns the triplets that link two kept groups, and the third turn reasons by analogy over them and gives an answer [arxiv:2505.15062 §3.1] [§3.2] [alg. 1]. Tool output is masked out of the loss [alg. 1]. Card: [SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning](/papers/2505.15062). The reward is a weighted sum of a format term and an exact-match term on the answer tag. The format term needs all four closing tags to appear. The weights are 0.3 and 0.7 at 3B, and 0.2 and 0.8 at 7B [arxiv:2505.15062 eq. 9] [eq. 10] [eq. 11] [§3.5]. A rollout can therefore earn the format share without any accuracy, so the format term alone is partial credit [eq. 9]. Author-reported: on the UMLS question sets, the 7B model reaches a 75.4 weighted average, against 70.1 for GPT-3.5-Turbo with GIVE [arxiv:2505.15062 tab. 1]. On CommonsenseQA over ConceptNet it reaches 81.3 [tab. 1]. The UMLS test set has 224 questions [§4.1], so one question moves the weighted average by about 0.45 points. The authors report token use per question at 639 to 1,156 on UMLS, against 7,010 to 19,461 for ToG and GIVE, a reduction of about 90% or more [arxiv:2505.15062 tab. 2] [§4.2]. The paper runs ablations on the 7B model: removing the filtering turn costs 2.5 to 5.0 points on the three UMLS sets [§4.3] [fig. 2]. The 70.1 GIVE figure uses GPT-3.5-Turbo, not the Qwen backbones. The other baselines run on the same backbone as SAKE [arxiv:2505.15062 §4.1] [tab. 1]. ### Verifier-checked task reward in 3D games (SIMA 2) SIMA 2 is a Gemini Flash-Lite agent that plays 3D games from screen pixels and keyboard-and-mouse actions [arxiv:2512.04797 §3.2] [§3.3]. After supervised fine-tuning, it is trained with RL on tasks that each carry a verification function. A task is an initial game state, a text instruction and a verifier. The reward is task completion or a correct answer to a grounded question, and some tasks add shaped rewards. Tasks come from three sources, and a task is kept only if a human could finish it within a time limit [arxiv:2512.04797 §3.3.3]. The RL algorithm, its settings and its reward shaping are not stated [§3.3.3]. Card: [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](/papers/2512.04797). Author-reported: average success on the training environments is about twice SIMA 1's [arxiv:2512.04797 §4.2.1] [fig. 6], and the held-out ASKA and MineDojo results are more than 10% higher in each environment [§4.2.2]. The per-environment values appear only in bar charts [fig. 8] [fig. 10]. Against the base Gemini model, the AIME change is -25.5% after SFT and -15.4% after SFT+RL. The GPQA Diamond change goes from -16.3% to -19.5% [arxiv:2512.04797 tab. 1]. ### Theory and diagnostic (on-policy distillation) [arxiv:2605.10889] is a distillation paper, not a reinforcement-learning training recipe. It runs no training, and its experiments score distillation gradients on Qwen3 students. It is carded here because it links the two settings: it argues that Dr. GRPO, GRPO without the length normalisation, gives in expectation a gradient proportional to an ideal per-node gradient of expected success [arxiv:2605.10889 §2.2] [app. B.2]. That derivation takes the KL weight as zero. Use it to understand the gradient, not to choose RLVR settings. Card: [distillation diagnostic](/papers/2605.10889). ## When to use which | Situation | Use | Why | |---|---|---| | Output is a checkable structure (box, class label) and only a few labelled examples exist | Rule rewards with group-relative RL, as in Visual-RFT | Author-reported gains over supervised fine-tuning at one to sixteen shots [arxiv:2503.01785 tab. 2] | | Output has several checkable parts (a time range plus an answer) | A reward sum per task, with a format term | The paper gives a reward mix for each task type [arxiv:2504.06958 §3.3] | | One model must handle several tasks | Joint training on all tasks | Single-task tracking training left Charades-STA mIoU at 28.7, against 29.0 for the base; joint training reached 60.8 [arxiv:2504.06958 tab. 2] | | Reward needs to judge free text (captions) | An LLM-judged recall reward | In one ablation, two judges gave nearly the same caption F1 (38.2 and 37.9) [arxiv:2504.06958 tab. 7] | | Long videos | Expect small or mixed gains | The authors' training videos are under one minute, and their long-video gains are small or mixed [arxiv:2504.06958 §4] [arxiv:2504.06958 tab. 8] | | Text-only maths with a boxed final answer, a small model, and a day on a few GPUs | The small-model recipe, with checkpoints taken early | AIME24 reaches 46.7 at 50 steps; the paper reports degradation after 150 to 200 steps [arxiv:2503.16219 tab. 1] [§3] | | On-policy RLVR stalls or collapses, and you can store rollouts | Experience replay, as in ExGRPO | Author-reported +3.0 math average over on-policy RLVR on Qwen2.5-Math-7B [arxiv:2510.02245 tab. 1]. The Llama gain uses a simplified prompt and no delayed start, so it is not like-for-like [arxiv:2510.02245 app. E.1] | | The model is a masked diffusion LLM, so exact likelihood is unavailable | Ratio-free weighted loss, as in wd1 | Needs one likelihood estimate per completion, and an RL step took 81 s against 104 s for d1 on four A100s [arxiv:2507.08838 tab. 2] | | The output is a program whose result a solver can check, and you can run the solver for every rollout | Solver-checked reward, as in SIRL | Author-reported macro average 60.6 (7B) and 68.3 (32B) over five columns, against 65.0 for DeepSeek-V3.1 [arxiv:2505.11792 tab. 1]. Training needed 8 H100 GPUs for about 24 hours per stage [§11] | | Speed of a fixed kernel on a fixed GPU is the goal | Measured speed reward, as in CUDA-L2 | Reported gains are for A100 only [arxiv:2512.02551 §1] [§7]. The timing protocol alone needs at least 40 seconds per shape and mode [§2.4] | | Several models can answer, the task has a checkable final answer, and the number of model calls must stay low | Plan-level reward, as in the Conductor | Author-reported 72.35 against 64.14 for the best single model under a shared cap, at about three steps per question [arxiv:2512.04388 tab. 7] [§4.3]. Every training rollout runs the worker chain, so the pool must be callable during training [§3.1] | | A robot controller exposes parameters, and a simulator can score a full run | Closed-loop reward on simulated runs, as in RobotxR1 | Author-reported 63.3% for the 3B model after SFT and GRPO, against 58.5% for GPT-4o, on one track [arxiv:2505.03238 tab. 2]. Every completion needs a simulator run [§3.3] | | The model will run on-board in a quantized format | Measure the task after quantization, and do not reuse the FP16 score | The Q5 build of the 3B model falls from 63.3% to 47.9% on control [arxiv:2505.03238 tab. 5]. The paper does not discuss this drop [tab. 5] | | No timing harness or anti-cheating checks exist yet | Build those first; do not reuse a speed reward without them | The paper adds rules against stream injection and lazy evaluation [§2.4.1], and its reward is hackable without them [§3.2.3] | | An agent acts in an interactive environment, and a program or verifier can check task completion | Verifier-checked task RL, as in SIMA 2 | Author-reported: about twice SIMA 1's success on training environments [arxiv:2512.04797 fig. 6]. The RL algorithm and settings are not stated, so the row cannot be rebuilt from the paper [§3.3.3] | | A small model must answer questions over a knowledge graph, and the per-question token budget matters | Tool-calling rollout with a closed answer tag, as in SAKE | Author-reported 75.4 weighted average on UMLS for 7B, against 70.1 for GPT-3.5-Turbo with GIVE [arxiv:2505.15062 tab. 1]. Each question needs two tool calls at no gradient [§3.2]. The authors measure token savings against ToG and GIVE as they implement them [§4.2] | | A general-purpose model is trained with RL on agent tasks and must keep its text reasoning | Measure general benchmarks after RL, not only agent tasks | SIMA 2 reports negative changes on AIME and GPQA Diamond after SFT and after SFT+RL [arxiv:2512.04797 tab. 1] | | Output is open-ended with no checkable answer | Not covered by these cards | None of the cards gives a reward for that case. See [RLHF](/concepts/rlhf) for learned rewards | Reasoning before the answer helps some tasks and not others. VideoChat-R1 reports that reasoning output does not help temporal grounding and does help QA [arxiv:2504.06958 §4]. Choose the output mode per task rather than assuming it helps everywhere. ## Implementation sketch Language-agnostic pseudocode of the shared core. The reward functions are the variant-specific part. ```pseudocode # One RLVR update for one prompt [arxiv:2503.01785 §3.1] [arxiv:2504.06958 §3.1] # Input: prompt, policy (trainable), reference (frozen), reward terms # Output: scalar loss def rlvr_loss(prompt, policy, reference, reward_terms, G, beta): # Sample a group of G responses; each has its reasoning and its answer # [arxiv:2503.01785 §3.2] responses = sample_group(policy, prompt, group_size=G) # Sum the verifiable reward terms for each response [arxiv:2504.06958 §3.3] rewards = [sum(term(r) for term in reward_terms) for r in responses] # Normalise inside the group [arxiv:2503.01785 eq. 4] [arxiv:2504.06958 eq. 1] # The equations have no epsilon; add a small one to avoid division by zero advantages = (rewards - mean(rewards)) / (std(rewards) + eps) # Per-token log-probabilities under the policy and the reference [§3.1] logp = token_logprobs(policy, responses) ref_logp = token_logprobs(reference, responses) # KL estimate against the reference, per token [arxiv:2504.06958 §3.1] kl = exp(ref_logp - logp) - (ref_logp - logp) - 1 # Advantage-weighted term minus beta times KL; no PPO-style clip # [arxiv:2504.06958 §3.1] [arxiv:2503.01785 §3.1] per_token = -(advantages * logp - beta * kl) # Average over each response's tokens, then over responses [§3.1] return mean(sum(per_token * mask) / sum(mask)) ``` Notes on where variants differ and where the cards disagree with the equations: - **Epsilon.** Both papers' normalisation equations have no epsilon. The released code of both adds 1e-4 to the standard deviation. - **Std scaling.** The sketch divides by the group standard deviation, as Visual-RFT and VideoChat-R1 do. Open-RS does too [arxiv:2503.16219 eq. 3], and so does the Conductor [arxiv:2512.04388 eq. 2]. The Conductor's paper does not say whether the deviation is the population or sample form, or what a group with equal rewards does [app. A]. ExGRPO turns the scaling off [arxiv:2510.02245 §3.1], and wd1 subtracts the group mean only [arxiv:2507.08838 §2.2]. SIRL normalises across the mini-batch instead, as REINFORCE++ does [arxiv:2505.11792 eq. 4] [§3.2]. SAKE's paper states no normalisation rule or group size [arxiv:2505.15062 §3.4]. Choose one and test it, since small groups make the standard deviation noisy. - **KL term.** The sketch keeps the KL penalty. wd1 sets its coefficient to zero and needs no reference model [arxiv:2507.08838 tab. 6]. ExGRPO's objective has no KL term [arxiv:2510.02245 §4.2]. The Conductor also sets its KL weight to zero, and its training has no reference-model sync [arxiv:2512.04388 app. A]. SIRL applies its KL only to the model and code sections [arxiv:2505.11792 eq. 5]. SAKE's objective includes a KL term to a fixed reference, but the paper gives no weight for it [arxiv:2505.15062 eq. 8]. - **Zero-variance groups.** If every response in a group earns the same reward, all advantages are zero and only the KL term acts. This is likely when the model is always right or always wrong, which is more common in few-shot settings (an inference from the setup, not a result the papers report). - **Replayed answers need their old log-probabilities.** In ExGRPO the replayed answer's ratio uses the policy that wrote it [arxiv:2510.02245 §4.2]. Record those log-probabilities when the answer is generated. A missing value must not fall back silently to the current policy. - **IoU matching.** The paper does not state the IoU threshold for matching a predicted box to a ground-truth box [arxiv:2503.01785 §3.2.1]. - **Classification match.** The paper defines accuracy as exact agreement [arxiv:2503.01785 §3.2.1]. Implementations that match loosely change the reward, so test the matcher on real outputs. - **Partial credit.** Reward terms are summed [arxiv:2504.06958 §3.3]. In the released code, a malformed answer can keep its IoU and lose only the format point, so the total is not all-or-nothing. SIRL's terms also add up: a response with correct tags and a crashing program earns 0.5 [§10]. The Conductor gives 0.5 to a parsable plan with a wrong answer and 0 to an unparsable one [arxiv:2512.04388 §3.1]. The generic recipe in its §2 uses other values, so use the §3.1 values for this variant. - **Empty IoU union.** The paper does not say how to score an empty union. The released code scores it as 1. Make the choice explicit. - **Unparseable gold answers.** The Open-RS code gives full credit when the reference cannot be parsed, so those examples reward any output. The paper does not mention this [arxiv:2503.16219 §2]. - **Solver tolerance.** SIRL's accuracy check uses an absolute tolerance of 0.01 in §3.3, but a relative error of 1e-6 in §11 [arxiv:2505.11792 §3.3] [§11]. The two differ, so pick one and state it in the implementation. - **Speed reward.** The CUDA-L2 reward averages timed speedups and subtracts the numerical and length penalties [arxiv:2512.02551 eq. 3]. The penalty coefficients are not stated [§3.2.3]. Timing runs for at least 30 seconds after 10 seconds of warm-up, and the order of the two kernels is shuffled in each run [§2.4]. - **Simulator reward.** The RobotxR1 reward is relative to a baseline run, so the value given to an unparseable parameter set changes its scale. The paper does not state that value [arxiv:2505.03238 §3.3]. Set it explicitly: a large error can clip at the lower bound of -4 [arxiv:2505.03238 eq. 3]. - **Model-scored reward.** SIMA 2's self-improvement round scores each trajectory from 0 to 100 with a Gemini rubric, and 50 or more counts as success [arxiv:2512.04797 §4.5]. The rubric text is not given, and the paper does not say how scored data becomes a training update. Treat this score as a learned judge, not as a rule. ## Contradictions - **Reasoning before the answer.** Visual-RFT trains a think-then-answer format and reports gains on perception tasks [arxiv:2503.01785 §3.2.2] [arxiv:2503.01785 tab. 2]. VideoChat-R1 reports that reasoning output does not help temporal grounding: on Charades-STA the reasoning-at-test output gets 59.9 mIoU, against 60.8 for the direct answer. It also reports that chain-of-thought outputs were sometimes worse than direct output on perception tasks [arxiv:2504.06958 tab. 1] [arxiv:2504.06958 §4]. The two papers study different tasks, so the disagreement is not settled here. - **Stability.** Open-RS reports that a 1.5B model's gains fade after 150 to 200 steps, with mixed-language output returning [arxiv:2503.16219 §3]. ExGRPO reports on-policy RLVR collapsing on Llama3.1-8B, and that replay trains stably there [arxiv:2510.02245 §5.2]. The models, data and recipes differ. Neither paper tests the other's remedy, so the results do not conflict directly, and neither shows whether replay would stop Open-RS's drift. - **KL placement.** SIRL's ablation puts partial KL ahead of both full KL and no KL on all three accuracy columns it reports [arxiv:2505.11792 tab. 3]. ExGRPO and wd1 use no KL term [arxiv:2510.02245 §4.2] [arxiv:2507.08838 tab. 6]. The other papers that keep one include SAKE [arxiv:2505.15062 eq. 8], whose weight the paper does not state. The ablation is a single paper's, on different models, so it does not settle the choice. - **Reward stages.** SIRL's two-stage reward is best on MAMO Complex, at 51.7 against 46.8 for stage 1 alone and 49.3 for stage 2 alone. On OptMATH, stage 2 alone scores 33.1 against 30.5 for the two-stage reward [arxiv:2505.11792 tab. 4]. - **Headline gain in RobotxR1.** The abstract gives the 1.5B gain over SFT as 20.2 points. Table 2 gives 11.3% for SFT and 41.5% for SFT and GRPO, a difference of 30.2 points [arxiv:2505.03238 tab. 2]. This page quotes the table. - **Within one paper.** On-policy distillation paper [arxiv:2605.10889] calls Dr. GRPO an unbiased estimator of the ideal gradient in its introduction [§1], but its own derivation shows only proportionality, with the KL weight set to zero [§2.2] [app. B.2]. Read the derivation, not the introduction. ## Papers - [Visual-RFT: Visual Reinforcement Fine-Tuning](/papers/2503.01785): GRPO with rule rewards for IoU, confidence and class match on Qwen2-VL, against supervised fine-tuning on few-shot classification and detection. - [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](/papers/2504.06958): GRPO with per-task reward mixes for video grounding, tracking and grounded QA on Qwen2.5-VL-7B. - [Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't](/papers/2503.16219): GRPO with rule rewards directly on a 1.5B distilled model, with maths data on four A40 GPUs for 24 hours. - [ExGRPO: Learning to Reason from Experience](/papers/2510.02245): experience replay for RLVR, with bucketed storage, a retired set, entropy-based answer selection and a shaped importance ratio, on Qwen2.5-Math-7B and other backbones. - [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](/papers/2507.08838): a ratio-free weighted log-likelihood loss with rule rewards for masked diffusion LLMs, on LLaDA-8B-Instruct. - [SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning](/papers/2505.15062): GRPO over a three-turn rollout with two knowledge-graph tool calls, rewarded on format and exact match of the answer tag, on Qwen2.5 3B and 7B. - [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](/papers/2505.11792): REINFORCE++ with a solver-checked reward and a partial KL term, on Qwen2.5 7B and 32B instruct models. - [CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning](/papers/2512.02551): staged RL with a measured speed reward, for half-precision matrix-multiply kernels on A100. - [Learning to Orchestrate Agents in Natural Language with the Conductor](/papers/2512.04388): GRPO trains a 7B planner to write three-list workflows for a pool of worker LLMs, rewarded on the final answer of the chain it runs. - [RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning](/papers/2505.03238): GRPO with LoRA on small Qwen2.5 models that write MPC parameters for a simulated racing car, rewarded on relative tracking error, after SFT. - [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](/papers/2512.04797): a Gemini Flash-Lite agent for 3D games, trained with SFT and then RL on verifier-checked tasks, with self-improvement scored by a Gemini reward model. - [Distillation diagnostic](/papers/2605.10889) (arXiv:2605.10889): a training-free diagnostic of distillation gradients, which also links Dr. GRPO to a per-node ideal gradient.