Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics

In plain words

A robot arm needs to know where its gripper is and where it should go next. A vision-language model can look at a picture of the workspace and talk about it, but out of the box it is poor at turning that into precise numbers for movement. The usual fix is to fine-tune it on question-and-answer pairs written by people, and the authors argue that those pairs are often not tuned to the job and can make the model forget things it knew before.

This paper trains the model differently. Each training question shows a scene and offers four possible answers, one correct and three made up. The model writes out its reasoning and then picks an answer. It is rewarded for a well-formed reply and for naming the right option, and it learns from groups of its own attempts at the same question, favouring the ones that did better than the others. Two extra question types ask it to name the current position and the direction of the next move, which the authors say helps.

The authors also built a new test with open-ended questions about robot scenes, graded by GPT-4o against human-written answers. On the low-level questions their 7B model scores above several commercial models, and it does better than its own base model and a supervised baseline on several tasks. These numbers are the authors' own and come from simulated scenes plus a few real-robot trials.

TL;DR

• Trains Qwen2.5-VL-7B-Instruct with GRPO on multiple-choice questions about the next keypoint state of a robot gripper, with a format reward plus an answer reward and no reasoning labels §3.3 §4.1.
• Adds two auxiliary question types, current-state and movement prediction, built from expert demonstrations and rule-based labels §3.2.
• Compares against Direct SFT and CoT SFT on the same questions; both SFT baselines score near zero on EmbodiedBench Manipulation [tab:EmbodiedBench_result].
• Author-reported: 11.68% vs 8.92% for the base model on EmbodiedBench Manipulation [tab:EmbodiedBench_result], and 1.51 vs 1.40 on Robot-R1 Bench spatial reasoning §4.2 tab:LowLevelControl_result. Planning drops (1.44 vs 1.66) §4.2 tab:OurBench_highlevel_result.

Problem

Vision-language models often miss the spatial detail that low-level robot control needs, and they are weak at high-level plans as well §1. Supervised fine-tuning on heuristically built embodied QA data is the usual remedy, but the authors say that data is not optimised for predicting robot actions, and that SFT causes catastrophic forgetting and poor generalisation §1, §5.

The paper assumes:

• expert demonstrations with keypoints and robot states exist, and waypoints can be extracted from them with the ARM repository §3.2 §4.1;
• a rule-checkable reward is available, namely whether the chosen option matches the labelled next state §3.3;
• the target is a single robot, a Franka Panda, controlled by end-effector position only §3.1.

The paper does not test real robots except for a small pick-and-place study used with a diffusion action head app. F.

Method

Robot state and keypoints §3.1 §3.2

The robot state is the end-effector position (x, y, z). Orientation and gripper state are left out on purpose §3.1. The appendix lists this as a limitation: rotation and gripper actions are not reasoned about app. A.

A demonstration is a sequence of frames, each with an observation and a state. Keypoints are a subset of frame indices that mark changes in direction or subgoals §3.2. The next keypoint after time t is the earliest keypoint index greater than t §3.2.

Metadata §3.2

Every question is conditioned on task metadata M. The text lists three parts: reference points in the scene (for example the centre of a table), the orientation of the three axes, and the size of a constant object such as the gripper §3.2. The paper does not give example metadata text or its exact format.

Question types §3.2

Waypoint prediction. The correct option is the state of the next keypoint. Three distractor states are sampled from the valid state space and the four options are shuffled §3.2.

Current-state prediction. Given the image and metadata, pick the current state from four options: the true state plus three sampled distractors §3.2.

Movement prediction. The label is a short phrase such as "move up" or "slightly move backward", computed from the change between the current state and the next keypoint state §3.2. The rules are:

• an axis with a non-zero change adds a direction word; the paper gives "move forward" for a positive x change, and does not state the words for y or z §3.2;
• the adverb "slightly" is added when the change along an axis is no more than half of the largest change over all axes §3.2;
• the distractors are three adjacent movement labels §3.2.

The paper calls these labels "primary direction" but does not say how a multi-axis move is phrased beyond the rules above.

Learning objective §3.1 §3.3

The policy is an LLM, written π_θ, that answers a question q. A reward model scores each answer. The paper uses GRPO §3.1. For each question, G responses are sampled from the old policy. Each gets a reward r_i. The advantage is the reward standardised within the group:

A_i = (r_i - mean(r)) / std(r)

The objective maximises a clipped importance-weighted advantage term, minus β times the KL divergence from a reference policy, averaged over the group §3.1. The equation is unnumbered in the paper. ε and β are described as hyperparameters but their values are not stated §3.1.

The reward is R = r_f + r_a. The format reward r_f gives credit when the reply has a <think> block followed by an <answer> block. The answer reward r_a gives rule-based credit when the content of the answer block exactly matches the correct option §3.3. The paper does not state the numeric values of r_f or r_a.

Algorithm §3.3 §4.1

Training uses GRPO on the mixed multiple-choice set from the three question types §3.2 §4.1. The pseudocode below follows the text. Hyperparameters are in the table further down.

# Build one multiple-choice training question  [§3.2]
def make_waypoint_question(demo, t, metadata, rng):
    # Index of the first keypoint after time t  [§3.2]
    keypoint = next_keypoint_after(demo.keypoints, t)
    correct = demo.states[keypoint]
    # Three distractor states from the valid state space  [§3.2]
    distractors = sample_valid_states(rng, count=3)
    # Shuffle the correct state among the distractors  [§3.2]
    options = shuffle([correct] + distractors, rng)
    question = prompt(metadata, demo.observations[t], options)
    return question, correct

# One GRPO update over a batch of questions  [§3.1] [§4.1]
def grpo_update(policy, old_policy, reference, batch, G=5):
    total = 0
    for question, answer in batch:
        # Sample G responses from the old policy at temperature 1.0  [§4.1]
        responses = [old_policy.sample(question) for _ in range(G)]
        # Reward R = r_f + r_a: format plus correct answer  [§3.3]
        rewards = [format_reward(o) + answer_reward(o, answer)
                   for o in responses]
        # Group-standardised advantage  [§3.1]
        advantages = standardise(rewards)
        # Clipped surrogate minus beta times KL to the reference  [§3.1]
        total += clipped_objective(policy, old_policy, reference,
                                   responses, advantages)
    return -total / len(batch)

Movement labels are generated by a rule rather than a model:

# Movement label from the change to the next keypoint  [§3.2]
def movement_label(state_now, state_next):
    # Change along each axis (x, y, z)  [§3.2]
    change = state_next - state_now
    largest = max(abs(change))
    words = []
    for axis, delta in zip(["x", "y", "z"], change):
        # Skip axes with no change  [§3.2]
        if delta == 0:
            continue
        # Direction word; only "forward" for +x is given  [§3.2]
        word = direction_word(axis, delta)
        # Add "slightly" when the change is at most half the largest  [§3.2]
        if abs(delta) <= largest / 2:
            word = "slightly " + word
        words.append(word)
    return words

Robot-R1 Bench §3.4 §4.2

A benchmark of 215 open-ended questions about 50 images from 10 RLBench tasks, five frames per task §3.4. There are four types: planning, high-level action, movement and spatial reasoning §3.4. The split is 65 spatial and 50 for each of the other three types §3.4. Answers are scored 0 to 3 by GPT-4o as a judge against a human reference answer and a per-type rubric §3.4 app. B.

Hyperparameters

Values are as stated in the paper unless marked "code" or "not stated".

Results

Author-reported. Setup: Qwen2.5-VL-7B-Instruct, trained on RLBench demonstrations. Baselines are the base model, Direct SFT, CoT SFT, and commercial models queried in several versions. The authors chose temperature 0 for the evaluated models §4.2.

Robot-R1 Bench, low-level control (movement and spatial, scale 0 to 3) §4.2 tab:LowLevelControl_result

Robot-R1 Bench, high-level control §4.2 tab:OurBench_highlevel_result

EmbodiedBench Manipulation, success rate %, average over 5 instruction groups §4.3 tab:EmbodiedBench_result

SpatialRGPT-Bench, accuracy %, quantitative and qualitative averages §4.4 tab:SpatialRPGPT

Ablation on the question design (Robot-R1 Bench) §4.4 tab:ablation_result

WP = waypoint, SP = current state, MP = movement prediction §4.4 tab:ablation_result.

Other reported results

• RL algorithm comparison, Robot-R1 Bench §4.4 tab:diff_algorithm: GRPO 1.44 / 1.30 / 0.76 / 1.51 (planning / high-level / movement / spatial); RLOO 1.56 / 1.54 / 0.68 / 1.52; REINFORCE++ 1.40 / 1.08 / 0.62 / 1.38.
• Three seeds §4.4 tab:robot_r1_seed: averages 1.57 / 1.42 / 0.65 / 1.59, standard deviations 0.13 / 0.11 / 0.11 / 0.08.
• Cold start, SFT on GPT-4.1-mini CoT data followed by Robot-R1 app. C tab:cold_start: planning 1.22 vs 0.98 for SFT alone; high-level 1.44 vs 1.34; movement 0.88 both; spatial 1.43 vs 1.40.
• Robot-R1 Bridge Bench on BridgeV2 real-robot images, 100 images app. C tab:r1bench_bridge: planning 1.38 vs 1.20 base; high-level 1.10 vs 0.89; movement 0.45 vs 0.46; spatial 1.56 vs 1.53.
• VLABench app. C tab:VLABench: M&T 46.58 vs 39.02; spatial 40.33 vs 35.83; physical law 25.33 vs 39.20; complex reasoning 33.87 vs 38.51.
• LIBERO, a vision-language-action model with a diffusion head app. F tab:libero: average 71.8 vs 69.0 for the base model. Goal 73.8 vs 61.6; Objective 80.4 vs 83.4.
• Real robot, four pick-and-place tasks app. F tab:real_exp: average success 23.96% vs 16.67% for the base model.
• Judge agreement with human experts §4.4 tab:pearson: Pearson correlation 0.3315 for planning; 0.8974, 0.8931 and 0.8961 for high-level action, movement and spatial. Computed on 80 responses, with the human score as the median of 8 annotators.

Paper vs code

No official code. The paper and the arXiv abstract page name no repository or project page. Searches: the paper's text and sources link only third-party frameworks used for training (EasyR1, Qwen2-VL-Finetune) and the ARM waypoint repository §4.1. GitHub repository search for "robot-r1" and related terms returned no repository owned by the authors. The authors' GitHub account for Younggyo Seo was checked and contains no Robot-R1 repository. Nothing was cloned.

Pitfalls

• The question format does most of the work. The gain depends on the MCQA framing. The ablation shows that an open-ended reward with waypoint prediction alone reaches a higher high-level average (1.43) than the final model (1.37) [tab:ablation_result]. The paper treats this as a reason to prefer MCQA, but the open-ended variant is better on that one measure.
• Planning gets worse. The base model's planning score is 1.66 and the trained model's is 1.44 [tab:OurBench_highlevel_result]. The authors say this is expected because the objective targets next-keypoint prediction, not long-horizon plans §4.2.
• Distractor sampling is underspecified. The distractors are sampled "from the valid state space" for waypoint and current-state questions §3.2. How the space is bounded, and how close distractors are to the correct state, is not stated. Distractors that are easy to rule out make the reward trivially easy to earn.
• Movement words are only partly defined. Only the +x direction wording is given §3.2. The naming for y and z, and how multiple moving axes combine, are not stated. The "slightly" rule uses ≤ half of the largest change §3.2.
• Reward values and clipping are not stated. The numeric values for r_f and r_a, the clip ε and the KL weight β are missing §3.1 §3.3. An implementer must choose them, and the advantage normalisation (the standard deviation of the group) can be unstable when all group rewards are equal.
• Batch size is ambiguous. "A batch size of 128" §4.1 does not say whether it counts prompts or responses, and the rollout batch of 512 is not tied to either.
• Epoch counts conflict in places. Experiments use 5 epochs §4.1. The Robot-R1 Bench appendix results were trained for one epoch app. C. Compare numbers only across the same epoch setting.
• The "28%" claim in the introduction is not reproduced by the tables. The introduction says low-level control improves by over 28% §1. From the table averages, spatial goes from 1.40 to 1.51 (about +8%) and movement from 0.58 to 0.76 (about +31%) [tab:LowLevelControl_result].
• Base model naming differs. §4 names the base model "Qwen2.5-7b-VL-Ins" and §4.3 writes "Qwen2.5-7B-Instruct", while the tables say "Qwen2.5-VL-7B-Ins". The model is presumably the vision-language variant; the text and tables should be read together.
• Seed count is unclear. The text describes "two additional experimental runs" §4.4, but the seed table has three rows, one of which matches the main result. Treat the standard deviations as based on three runs and check against the paper's appendix before citing them.
• Judge reliability is weak on planning. Pearson correlation is 0.33 for planning, so planning scores from the GPT-4o judge are the least trustworthy §4.4 tab:pearson.
• Single simulator and few real trials. Training uses five RLBench tasks at 224 × 224 §4.1. Real-robot evidence is four tasks with 96 trials total, using a separate diffusion policy per task app. F. Low-level improvements on the Bridge Bench are small, which the authors attribute to camera and coordinate differences app. C.
• Cost and scale. Training took about 12 hours on four A100 80GB GPUs for 5 epochs over about 7.5K prompts app. B. The SFT baselines use a different learning rate (1e-5) from the GRPO run (1e-6), so the comparison is not learning-rate-matched §4.1.
• Frameworks, not the authors' code. The GRPO settings come from EasyR1 and the SFT settings from Qwen2-VL-Finetune §4.1. Those are third-party repositories, not the paper's own code; no code is cited here.
• Limitation the abstract hides. Only end-effector position is modelled. Rotation and gripper actions are excluded and the authors say these are needed for complex tasks app. A.

Lineage

• Builds on: GRPO, from DeepSeekMath, which the paper cites as the policy-optimisation algorithm §3.1; DeepSeek-R1-Zero's explicit think-then-answer format §3.3.
• Compared against: SFT with direct answers and with a hand-written chain of thought, following the robotic reasoning setup of Zawalski et al. §4.1; RLOO and REINFORCE++ as alternative RL algorithms §4.4; EmbodiedBench for evaluation §4.3.
• Superseded by: not known from content already in this wiki.