Reinforcement learning with verifiable rewards (RLVR)

Overview

The model answers a prompt several times. Each answer is scored by one or more rules that do not need a learned judge: an IoU for a box, an exact class match, a boxed final answer for maths, a cell-by-cell score for a puzzle, the objective a solver returns for an optimisation model, the measured speed of a GPU kernel, the tracking error of a simulated car lap, whether a game task was completed, or whether a knowledge-graph answer matches the gold answer. Most variants then compare each answer with the others sampled for the same prompt, so the ones that beat their siblings get pushed up and the rest down 2503.01785 §3.1 2504.06958 §3.1 2503.16219 §2. The shared idea is that a check runs on the output, rather than a learned judge deciding the score. VideoChat-R1's caption reward is an exception, since an LLM judge computes it 2504.06958 §3.3, and so is SIMA 2's self-improvement loop, where a Gemini model scores each trajectory 2512.04797 §4.5. Most variants use GRPO or a close relative; SIRL uses REINFORCE++ instead 2505.11792 §3.2, and SIMA 2 does not name its algorithm 2512.04797 §3.3.3. The method papers are not carded here.

Papers differ on two training details: whether the group spread is divided out of the advantage, and whether a KL penalty keeps the policy near a reference model. Both are covered under Implementation sketch. Three of the papers keep a KL penalty 2503.01785 §3.1 2504.06958 §3.1 2503.16219 §2, and SAKE's objective has one too, with its weight not stated in the paper 2505.15062 eq. 8. wd1 sets its weight to zero [arxiv:2507.08838 tab. 6], ExGRPO's objective has none 2510.02245 §4.2, and the Conductor sets its KL weight to zero 2512.04388 app. A. SIRL keeps KL only on its model and code sections, not on its reasoning 2505.11792 eq. 5. SIMA 2 does not mention a KL term, and its RL algorithm is not stated 2512.04797 §3.3.3.

The vision and video cards cover vision-language and video models. The maths and diffusion cards cover text-only reasoning. The two code cards cover output that a solver or a GPU runs: SIRL writes optimisation models, and CUDA-L2 writes matrix-multiply kernels. The RobotxR1 card covers a robot controller: the output is a parameter set for a model predictive controller, and the reward comes from a simulated lap. The SAKE card covers a model that calls two knowledge-graph tools during its rollout, and the reward reads its final answer tag. The Conductor card applies the same kind of check one level up: the scored output is a plan for other models, and the reward reads the final answer of the chain that the plan runs. The SIMA 2 card covers a game agent whose RL tasks use verification functions. Its self-improvement reward is a model's rubric score, not a rule. All numbers are author-reported.

Variants

Rule rewards for vision-language perception (Visual-RFT)

Qwen2-VL models are trained with GRPO. For detection, the reward adds an IoU term, a confidence term and a format term; each predicted box is matched to the best unused ground-truth box 2503.01785 §3.2.1 eq. 5 eq. 6 eq. 7 eq. 8. For classification, the reward is an accuracy term (1 for the right class, else 0) plus the format term eq. 9. Card: Visual-RFT: Visual Reinforcement Fine-Tuning.

Reported results: one-shot fine-grained classification averages 80.3, against 51.7 for supervised fine-tuning and 56.0 for the base model [arxiv:2503.01785 tab. 2]. Two-shot COCO detection reaches 41.5 mAP against 19.6 for the base 2503.01785 tab. 3. On LVIS open-vocabulary detection the 2B model goes from 2.7 to 20.7 mAP 2503.01785 tab. 8.

Per-task reward mixes for video (VideoChat-R1)

Qwen2.5-VL-7B is trained with GRPO, and each task gets its own sum of reward terms: temporal grounding and tracking use format plus IoU; multiple-choice QA uses format plus accuracy; grounded QA uses format, IoU and accuracy; and captioning uses format plus a recall reward that an LLM judge computes 2504.06958 §3.2 2504.06958 §3.3 eq. 2 eq. 3 eq. 4 eq. 5. Card: VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning.

The joint model trains on 18,031 samples from three sets: Charades-STA, GoT-10k and NExTGQA 2504.06958 §3.3. Author-reported: Charades-STA mIoU rises from 29.0 to 60.8 and GoT-10k average overlap from 12.6 to 43.8 over the base model 2504.06958 tab. 1. With one epoch, supervised fine-tuning on Charades-STA reaches 46.3 mIoU and the GRPO run with thinking reaches 59.3 2504.06958 tab. 3. Three epochs of supervised fine-tuning lower ActivityNet mIoU from 20.6 to 17.3 2504.06958 tab. 3.

Small-model maths recipe (Open-RS)

A 1.5B reasoning model (DeepSeek-R1-Distill-Qwen-1.5B) is trained with GRPO directly, without supervised fine-tuning. The data is 7,000 curated maths problems, and the rewards are answer correctness, a format bonus and, in the third run, a cosine length-scaled reward 2503.16219 §2 §3. Card: Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't.

The run used four A40 GPUs for 24 hours. The authors estimate about USD 42 of rental cost 2503.16219 §3 tab. 3. Author-reported: AIME24 rises from 28.8 for the start model to 46.7 for the third run at 50 steps [arxiv:2503.16219 tab. 1]. The other two runs score 30.0 on AIME24. The main caution is that the early gains fade: quality peaks within 50 to 100 steps and degrades after about 150 to 200 steps 2503.16219 §3. AIME24 has 30 problems, so one answer moves the score by 3.3 points 2503.16219 app. C.

Experience replay for on-policy RLVR (ExGRPO)

Correct rollouts are stored per question, questions the model always solves are retired, and about half of each batch replays a stored answer. The replayed answer is the one with the lowest mean token entropy under the current policy. Its importance ratio is shaped by f(w) = w / (w + 0.1) 2510.02245 §4.1 §4.2. Card: ExGRPO: Learning to Reason from Experience.

Author-reported: the math average on Qwen2.5-Math-7B is 48.3, against 45.3 for on-policy RLVR with the same setup 2510.02245 tab. 1. On Llama3.1-8B, the on-policy run collapses while ExGRPO trains stably. Its math average is 6.1 against 3.4 2510.02245 §5.2 app. E.3. The replay starts only after the batch pass rate passes 35% 2510.02245 §4.2. On eight GPUs the average step takes 19% longer 2510.02245 app. E.5.

Ratio-free weighted loss for diffusion LLMs (wd1)

Masked diffusion LLMs define a likelihood only through an iterative denoising process, so the probability of a whole answer cannot be computed exactly. wd1 uses one masked-pass estimate per completion (the d1 estimator), and it needs neither an importance ratio nor a reference model. The loss raises the likelihood of high-advantage completions and lowers it for low-advantage ones, with two softmax weights over the group 2507.08838 §3.2 eq. 8 eq. 9. Card: wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models.

On LLaDA-8B-Instruct trained with LoRA, author-reported results are: Sudoku 76.4 against 17.6 for the authors' reproduction of d1 at 256 tokens; GSM8K 80.8 against 78.2 at 256 tokens; MATH500 34.4 against 34.4 at 256 tokens and 39.0 against 38.0 at 512 2507.08838 tab. 1. Each reported number is a single seed 2507.08838 tab. 6. The paper's extended variant, wd1++, uses intermediate denoising steps. Its code is not in the authors' repository 2507.08838 §3.3.

Solver-checked reward for optimisation models (SIRL)

Qwen2.5 7B and 32B instruct models are trained with RL directly, without supervised fine-tuning 2505.11792 §3.2 §4. Each response has three tagged parts: reasoning, a concise mathematical model, and solver code. The code is run, and the stage-1 reward has three terms: 0.5 for the tags in the right order, 1 if the code runs, and 2 if the objective is within 0.01 of the ground truth 2505.11792 §3.3 eq. 6 §10. Stage 2 adds a bonus term, and its weight is not stated §10. Card: Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling.

Training problems are synthesised from seed cases, augmented, and checked by an LLM judge. The answer is picked by a vote over ten roles, scored on features of each role's solver model 2505.11792 §3.1 eq. 2 eq. 3. Problems the 32B baseline solves in 8 of 10 tries are dropped as too easy §3.1 §11.

Author-reported macro average over five benchmark columns is 60.6 for the 7B model and 68.3 for the 32B model 2505.11792 tab. 1. In the same table, DeepSeek-V3.1 scores 65.0, so the 7B model trails it on the macro average tab. 1. On MAMO Complex, the partial KL reaches 51.7, against 48.3 with full KL and 47.3 with none 2505.11792 tab. 3. Each stage took about 24 hours on 8 H100 GPUs §11.

Speed-measured reward for GPU kernels (CUDA-L2)

CUDA-L2 trains an LLM to write half-precision matrix-multiply kernels 2512.02551 §1 §3. The reward is the mean speedup over timed runs against a reference kernel, minus a penalty for numerical error from an FP32 CPU result, minus a penalty for code length 2512.02551 eq. 3 §3.2.3. The penalty coefficients are not stated §3.2.3. Training has three stages: continued pretraining on CUDA code, RL on about 1,000 general kernels, and RL on matrix multiply alone 2512.02551 §3.2. The base model is DeepSeek 671B §3.2.1. Card: CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning.

Author-reported, on an A100 over 1,000 matrix shapes and offline timing: a mean speedup of +22.0% over torch.matmul and +11.4% over the auto-tuned cuBLASLt baseline, in its best layout 2512.02551 tab. 1. The speedup is a mean of per-run time ratios minus one §2.4 eq. 2. The gain falls with problem size, from about 1.4× on small problems to about 1.0× on the largest, against cuBLASLt auto-tuning §4.3. The paper's exact correctness test covers only outputs below 2048; larger outputs are bounded indirectly §2.3.

Simulated-lap reward for a robot controller (RobotxR1)

Qwen2.5 1.5B and 3B models are trained first with SFT, then with GRPO on LoRA adapters 2505.03238 §3.3 §4.1. For the controller module, the model gets an instruction such as "reverse the car" and writes a dictionary of parameters for a model predictive controller (MPC) on a 1:10 scale racing car. The dictionary is applied, the car drives a lap in a simulator, and the driving term is the relative improvement of the tracking error over the default MPC, clipped below at -4. A format term and a parameter term are added to it 2505.03238 eq. 3. Card: RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning.

Author-reported, on one evaluation track with 20 paraphrased prompts and five runs 2505.03238 app. C tab. 2: the 3B model goes from 50.4% after SFT to 63.3% after SFT and GRPO, against 58.5% for GPT-4o. Without SFT, GRPO alone takes the 3B model from 34.7% to 57.2%. The 1.5B model goes from 11.3% to 41.5% with SFT and GRPO. The SFT-only 3B figure leaves out one extraction failure tab. 2. The same paper trains a yes or no decision module with GRPO. Its 3B accuracy goes from 83.78% after SFT to 86.82% after SFT and GRPO 2505.03238 tab. 1. The GRPO run on the 3B model used about 11 GB of GPU memory and about 24 hours §4.1.

The paper evaluates on a different track from the one it trains on, so the transfer is part of the claim app. B. It reports no confidence intervals tab. 2.

Plan-level reward for multi-model workflows (Conductor)

A 7B planner, starting from Qwen2.5-7B, is trained with GRPO to write a plan for a pool of worker LLMs 2512.04388 app. A §4.1. Each plan has three equal-length lists: the worker ids, a natural-language subtask for each step, and an access list that says which earlier steps that worker may see 2512.04388 §3.1 app. E. The reward is 1 if the final answer of the executed chain is correct, 0.5 if it is wrong but the plan parses, and 0 if the three lists cannot be parsed §3.1. Card: Learning to Orchestrate Agents in Natural Language with the Conductor.

Author-reported, unconstrained: on LiveCodeBench v6 the Conductor scores 83.93, against 82.90 for GPT 5, the best single worker on that task [arxiv:2512.04388 tab. 1]. Under a shared cap of 4,096 output tokens and minimal reasoning budgets, it averages 72.35 over four training tasks, against 64.14 for the best single model tab. 7. The paper reports about three steps per question on average §4.3, against eight calls for Mixture-of-Agents app. A. The unconstrained baselines are each model's best across the authors' runs and online leaderboards, so they are a reported frontier and not a controlled comparison §4.2. Allowing the planner to call itself raises the BigCodeBench score from 37.8 to 40.0 on one run tab. 2. The recursive training questions were already seen in training §4.4, so this does not measure generalisation to new questions.

Knowledge-graph tool calls inside the rollout (SAKE)

Qwen2.5 3B and 7B instruct models are trained with GRPO, with no supervised stage 2505.15062 §3.4 §3.5. Each rollout has three model turns and two deterministic tool calls in between. The first turn names the entities in the question. Tool 1 returns, for each entity, a group made of that entity and its most similar knowledge-graph concepts. The second turn keeps some of these groups. Tool 2 returns the triplets that link two kept groups, and the third turn reasons by analogy over them and gives an answer 2505.15062 §3.1 §3.2 alg. 1. Tool output is masked out of the loss alg. 1. Card: SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning.

The reward is a weighted sum of a format term and an exact-match term on the answer tag. The format term needs all four closing tags to appear. The weights are 0.3 and 0.7 at 3B, and 0.2 and 0.8 at 7B 2505.15062 eq. 9 eq. 10 eq. 11 §3.5. A rollout can therefore earn the format share without any accuracy, so the format term alone is partial credit eq. 9.

Author-reported: on the UMLS question sets, the 7B model reaches a 75.4 weighted average, against 70.1 for GPT-3.5-Turbo with GIVE 2505.15062 tab. 1. On CommonsenseQA over ConceptNet it reaches 81.3 tab. 1. The UMLS test set has 224 questions §4.1, so one question moves the weighted average by about 0.45 points. The authors report token use per question at 639 to 1,156 on UMLS, against 7,010 to 19,461 for ToG and GIVE, a reduction of about 90% or more 2505.15062 tab. 2 §4.2. The paper runs ablations on the 7B model: removing the filtering turn costs 2.5 to 5.0 points on the three UMLS sets §4.3 fig. 2.

The 70.1 GIVE figure uses GPT-3.5-Turbo, not the Qwen backbones. The other baselines run on the same backbone as SAKE 2505.15062 §4.1 tab. 1.

Verifier-checked task reward in 3D games (SIMA 2)

SIMA 2 is a Gemini Flash-Lite agent that plays 3D games from screen pixels and keyboard-and-mouse actions 2512.04797 §3.2 §3.3. After supervised fine-tuning, it is trained with RL on tasks that each carry a verification function. A task is an initial game state, a text instruction and a verifier. The reward is task completion or a correct answer to a grounded question, and some tasks add shaped rewards. Tasks come from three sources, and a task is kept only if a human could finish it within a time limit [arxiv:2512.04797 §3.3.3]. The RL algorithm, its settings and its reward shaping are not stated §3.3.3. Card: SIMA 2: A Generalist Embodied Agent for Virtual Worlds.

Author-reported: average success on the training environments is about twice SIMA 1's 2512.04797 §4.2.1 fig. 6, and the held-out ASKA and MineDojo results are more than 10% higher in each environment §4.2.2. The per-environment values appear only in bar charts fig. 8 fig. 10. Against the base Gemini model, the AIME change is -25.5% after SFT and -15.4% after SFT+RL. The GPQA Diamond change goes from -16.3% to -19.5% [arxiv:2512.04797 tab. 1].

Theory and diagnostic (on-policy distillation)

2605.10889 is a distillation paper, not a reinforcement-learning training recipe. It runs no training, and its experiments score distillation gradients on Qwen3 students. It is carded here because it links the two settings: it argues that Dr. GRPO, GRPO without the length normalisation, gives in expectation a gradient proportional to an ideal per-node gradient of expected success 2605.10889 §2.2 app. B.2. That derivation takes the KL weight as zero. Use it to understand the gradient, not to choose RLVR settings. Card: distillation diagnostic.

When to use which

Reasoning before the answer helps some tasks and not others. VideoChat-R1 reports that reasoning output does not help temporal grounding and does help QA 2504.06958 §4. Choose the output mode per task rather than assuming it helps everywhere.

Implementation sketch

Language-agnostic pseudocode of the shared core. The reward functions are the variant-specific part.

# One RLVR update for one prompt  [arxiv:2503.01785 §3.1] [arxiv:2504.06958 §3.1]
# Input:  prompt, policy (trainable), reference (frozen), reward terms
# Output: scalar loss

def rlvr_loss(prompt, policy, reference, reward_terms, G, beta):
    # Sample a group of G responses; each has its reasoning and its answer
    # [arxiv:2503.01785 §3.2]
    responses = sample_group(policy, prompt, group_size=G)
    # Sum the verifiable reward terms for each response  [arxiv:2504.06958 §3.3]
    rewards = [sum(term(r) for term in reward_terms) for r in responses]
    # Normalise inside the group  [arxiv:2503.01785 eq. 4] [arxiv:2504.06958 eq. 1]
    # The equations have no epsilon; add a small one to avoid division by zero
    advantages = (rewards - mean(rewards)) / (std(rewards) + eps)
    # Per-token log-probabilities under the policy and the reference  [§3.1]
    logp = token_logprobs(policy, responses)
    ref_logp = token_logprobs(reference, responses)
    # KL estimate against the reference, per token  [arxiv:2504.06958 §3.1]
    kl = exp(ref_logp - logp) - (ref_logp - logp) - 1
    # Advantage-weighted term minus beta times KL; no PPO-style clip
    # [arxiv:2504.06958 §3.1] [arxiv:2503.01785 §3.1]
    per_token = -(advantages * logp - beta * kl)
    # Average over each response's tokens, then over responses  [§3.1]
    return mean(sum(per_token * mask) / sum(mask))

Notes on where variants differ and where the cards disagree with the equations:

• Epsilon. Both papers' normalisation equations have no epsilon. The released code of both adds 1e-4 to the standard deviation.
• Std scaling. The sketch divides by the group standard deviation, as Visual-RFT and VideoChat-R1 do. Open-RS does too 2503.16219 eq. 3, and so does the Conductor 2512.04388 eq. 2. The Conductor's paper does not say whether the deviation is the population or sample form, or what a group with equal rewards does app. A. ExGRPO turns the scaling off 2510.02245 §3.1, and wd1 subtracts the group mean only 2507.08838 §2.2. SIRL normalises across the mini-batch instead, as REINFORCE++ does 2505.11792 eq. 4 §3.2. SAKE's paper states no normalisation rule or group size [arxiv:2505.15062 §3.4]. Choose one and test it, since small groups make the standard deviation noisy.
• KL term. The sketch keeps the KL penalty. wd1 sets its coefficient to zero and needs no reference model 2507.08838 tab. 6. ExGRPO's objective has no KL term 2510.02245 §4.2. The Conductor also sets its KL weight to zero, and its training has no reference-model sync [arxiv:2512.04388 app. A]. SIRL applies its KL only to the model and code sections 2505.11792 eq. 5. SAKE's objective includes a KL term to a fixed reference, but the paper gives no weight for it 2505.15062 eq. 8.
• Zero-variance groups. If every response in a group earns the same reward, all advantages are zero and only the KL term acts. This is likely when the model is always right or always wrong, which is more common in few-shot settings (an inference from the setup, not a result the papers report).
• Replayed answers need their old log-probabilities. In ExGRPO the replayed answer's ratio uses the policy that wrote it 2510.02245 §4.2. Record those log-probabilities when the answer is generated. A missing value must not fall back silently to the current policy.
• IoU matching. The paper does not state the IoU threshold for matching a predicted box to a ground-truth box 2503.01785 §3.2.1.
• Classification match. The paper defines accuracy as exact agreement 2503.01785 §3.2.1. Implementations that match loosely change the reward, so test the matcher on real outputs.
• Partial credit. Reward terms are summed 2504.06958 §3.3. In the released code, a malformed answer can keep its IoU and lose only the format point, so the total is not all-or-nothing. SIRL's terms also add up: a response with correct tags and a crashing program earns 0.5 §10. The Conductor gives 0.5 to a parsable plan with a wrong answer and 0 to an unparsable one 2512.04388 §3.1. The generic recipe in its §2 uses other values, so use the §3.1 values for this variant.
• Empty IoU union. The paper does not say how to score an empty union. The released code scores it as 1. Make the choice explicit.
• Unparseable gold answers. The Open-RS code gives full credit when the reference cannot be parsed, so those examples reward any output. The paper does not mention this 2503.16219 §2.
• Solver tolerance. SIRL's accuracy check uses an absolute tolerance of 0.01 in §3.3, but a relative error of 1e-6 in §11 2505.11792 §3.3 §11. The two differ, so pick one and state it in the implementation.
• Speed reward. The CUDA-L2 reward averages timed speedups and subtracts the numerical and length penalties 2512.02551 eq. 3. The penalty coefficients are not stated §3.2.3. Timing runs for at least 30 seconds after 10 seconds of warm-up, and the order of the two kernels is shuffled in each run §2.4.
• Simulator reward. The RobotxR1 reward is relative to a baseline run, so the value given to an unparseable parameter set changes its scale. The paper does not state that value 2505.03238 §3.3. Set it explicitly: a large error can clip at the lower bound of -4 2505.03238 eq. 3.
• Model-scored reward. SIMA 2's self-improvement round scores each trajectory from 0 to 100 with a Gemini rubric, and 50 or more counts as success 2512.04797 §4.5. The rubric text is not given, and the paper does not say how scored data becomes a training update. Treat this score as a learned judge, not as a rule.

Contradictions

• Reasoning before the answer. Visual-RFT trains a think-then-answer format and reports gains on perception tasks 2503.01785 §3.2.2 [arxiv:2503.01785 tab. 2]. VideoChat-R1 reports that reasoning output does not help temporal grounding: on Charades-STA the reasoning-at-test output gets 59.9 mIoU, against 60.8 for the direct answer. It also reports that chain-of-thought outputs were sometimes worse than direct output on perception tasks 2504.06958 tab. 1 2504.06958 §4. The two papers study different tasks, so the disagreement is not settled here.
• Stability. Open-RS reports that a 1.5B model's gains fade after 150 to 200 steps, with mixed-language output returning 2503.16219 §3. ExGRPO reports on-policy RLVR collapsing on Llama3.1-8B, and that replay trains stably there 2510.02245 §5.2. The models, data and recipes differ. Neither paper tests the other's remedy, so the results do not conflict directly, and neither shows whether replay would stop Open-RS's drift.
• KL placement. SIRL's ablation puts partial KL ahead of both full KL and no KL on all three accuracy columns it reports 2505.11792 tab. 3. ExGRPO and wd1 use no KL term 2510.02245 §4.2 2507.08838 tab. 6. The other papers that keep one include SAKE 2505.15062 eq. 8, whose weight the paper does not state. The ablation is a single paper's, on different models, so it does not settle the choice.
• Reward stages. SIRL's two-stage reward is best on MAMO Complex, at 51.7 against 46.8 for stage 1 alone and 49.3 for stage 2 alone. On OptMATH, stage 2 alone scores 33.1 against 30.5 for the two-stage reward [arxiv:2505.11792 tab. 4].
• Headline gain in RobotxR1. The abstract gives the 1.5B gain over SFT as 20.2 points. Table 2 gives 11.3% for SFT and 41.5% for SFT and GRPO, a difference of 30.2 points 2505.03238 tab. 2. This page quotes the table.
• Within one paper. On-policy distillation paper 2605.10889 calls Dr. GRPO an unbiased estimator of the ideal gradient in its introduction §1, but its own derivation shows only proportionality, with the KL weight set to zero §2.2 app. B.2. Read the derivation, not the introduction.

Papers

• Visual-RFT: Visual Reinforcement Fine-Tuning: GRPO with rule rewards for IoU, confidence and class match on Qwen2-VL, against supervised fine-tuning on few-shot classification and detection.
• VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning: GRPO with per-task reward mixes for video grounding, tracking and grounded QA on Qwen2.5-VL-7B.
• Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't: GRPO with rule rewards directly on a 1.5B distilled model, with maths data on four A40 GPUs for 24 hours.
• ExGRPO: Learning to Reason from Experience: experience replay for RLVR, with bucketed storage, a retired set, entropy-based answer selection and a shaped importance ratio, on Qwen2.5-Math-7B and other backbones.
• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models: a ratio-free weighted log-likelihood loss with rule rewards for masked diffusion LLMs, on LLaDA-8B-Instruct.
• SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning: GRPO over a three-turn rollout with two knowledge-graph tool calls, rewarded on format and exact match of the answer tag, on Qwen2.5 3B and 7B.
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling: REINFORCE++ with a solver-checked reward and a partial KL term, on Qwen2.5 7B and 32B instruct models.
• CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning: staged RL with a measured speed reward, for half-precision matrix-multiply kernels on A100.
• Learning to Orchestrate Agents in Natural Language with the Conductor: GRPO trains a 7B planner to write three-list workflows for a pool of worker LLMs, rewarded on the final answer of the chain it runs.
• RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning: GRPO with LoRA on small Qwen2.5 models that write MPC parameters for a simulated racing car, rewarded on relative tracking error, after SFT.
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds: a Gemini Flash-Lite agent for 3D games, trained with SFT and then RL on verifier-checked tasks, with self-improvement scored by a Gemini reward model.
• Distillation diagnostic (arXiv:2605.10889): a training-free diagnostic of distillation gradients, which also links Dr. GRPO to a per-node ideal gradient.