RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning

In plain words

A self-driving robot car needs a brain that can run on the car itself, because a cloud connection cannot be relied on. Small language models are cheap enough to run there, but they are weak at reasoning about the car. The usual fix is to copy answers from a much larger model. This paper asks whether a small model can learn from experience instead: it tries a setting in a simulator, gets scored on how well the car followed the instruction, and adjusts.

The model's job is to turn a plain instruction such as "reverse the car" into new settings for the car's steering controller. A reward compares how the car drove with those settings against how it drove with the default settings. The training uses the same trial-and-score style that recent math-reasoning work used with static problem sets, but here each trial runs a closed loop with a simulated car.

The authors report that a 3-billion-parameter model trained this way follows instructions better than the much larger GPT-4o did in their test. The result comes from one track and a set of paraphrased instructions, so treat it as a proof of concept.

TL;DR

The paper trains Qwen2.5 1.5B and 3B models first with supervised fine-tuning on examples, then with GRPO reinforcement learning, where each completion's reward comes from a driving simulation §3.3. The output is a parameter dictionary for a model predictive controller (MPC). On the evaluation track the 3B model scores 63.3% control adaptability against 58.5% for GPT-4o, both author-reported tab. 2. The 1.5B model's gain over SFT is written as 20.2 points, but the table values give 30.2. Use the table, and see Pitfalls.

Problem

Deployed robots need on-board reasoning without a cloud link §1. Reasoning models such as DeepSeek-R1 were trained with reinforcement learning on static math or code data, and the 7B model that the authors compare against was not trained with GRPO at all §1 §4.1. Distilling a large model into a small one with SFT copies behaviour but gives no closed-loop feedback §1 §4.3. The paper's target is a 1:10 scale racing car, and the authors say adapting it to other robots would need system-specific work §6.

Compute is a stated constraint. The authors say their GRPO runs fit on consumer GPUs (RTX 4070 Ti, RTX 3090) and that models above Qwen2.5-3B could not be trained with GRPO, which needed about 11 GB of VRAM §4.1.

Method

The system has two LLM modules around a classical MPC racing-line tracker §3.

1. Base controller. A kinematic MPC follows a precomputed minimum-curvature racing line in curvilinear coordinates. Its state holds longitudinal and lateral position, heading error, steering angle and speed. Its control input is a steering-rate change and an acceleration §3.1. The cost in eq. 1 penalises lateral error, speed error and heading error, with control-change weights. Its weights, the velocity bounds and the track-boundary inflation factor can be set online §3.1.
2. Retrieval. Both LLM modules read prompts that include retrieved memories from a retrieval-augmented setup inherited from the baseline framework. Five memories are retrieved per query §3. The memory texts are in app. E.
3. DecisionxR1 (decision module, §3.2). The LLM reads a window of robot states and answers whether the car follows the behaviour the human asked for. Training has two stages: SFT on examples from the baseline framework, then GRPO on a static dataset of labelled robot-state windows §3.2. The static labels are set by programmed rules per driving style, such as "reverse" meaning negative speed §3.2.
4. MPCxR1 (control module, §3.3). The LLM is given an instruction and writes a dictionary new_mpc_params = {...} of MPC parameters. The dictionary is applied to the MPC, the car drives in simulation, and the reward is computed from the tracking error over the lap §3.3 fig. 3. Training uses a circle map and evaluation uses a racing track app. B.
5. Rewards. The decision reward is an accuracy term plus a format term eq. 2. The MPC reward is a driving term plus a format term plus a parameter term eq. 3. Driving reward is the relative RMSE improvement over the default MPC, clipped below at -4 eq. 3. Each driving style has its own RMSE definition, given in app. C.
6. Training. GRPO with LoRA adapters, 750 steps for the MPC module §4.1. The MPC module's base model and LoRA settings are in Hyperparameters.

# MPCxR1 reward for one sampled completion  [§3.3] [eq. 3]
# Input:  completion (text), baseline_rmse (float), eval_fn (callable)
# Output: scalar reward

def mpc_reward(completion, baseline_rmse, eval_fn):
    # Parse the parameter dictionary  [code: llm_mpc.py#L236-L274]
    params = parse_mpc_params(completion)
    if params is None:
        # Unparseable output is scored as a large error; see Pitfalls
        # [code: train/rl_mpc_train.py#L228-L231]
        rmse = 69.0
    else:
        # Apply params, drive one lap in simulation, measure the behaviour RMSE
        # [code: train/rl_mpc_train.py#L233-L268]
        rmse = eval_fn(apply_and_drive(params))
    # Relative improvement over the default MPC, lower bound -4  [eq. 3]
    # [code: train/rl_mpc_train.py#L328-L338]
    improvement = (baseline_rmse - rmse) / baseline_rmse
    return max(improvement, -4.0)

# Decision reward for one sampled completion  [§3.2] [eq. 2]
# Input:  completion (text), gt (bool label from programmed rules)
# Output: scalar reward

def decision_reward(completion, gt):
    # Accuracy term: 1 when the answer line matches the label, else 0
    # [code: train/rl_decision_train.py#L169-L188]
    predicted = parse_adherence_line(completion)
    accuracy = 1.0 if predicted == gt else 0.0
    # Format term: share of the three tag pairs present exactly once
    # [code: train/rl_decision_train.py#L142-L157]
    tags = ["reasoning", "explanation", "answer"]
    format_score = tag_fraction(completion, tags)
    return accuracy + format_score

Hyperparameters

Values marked "paper" come from the text or tables. Values marked "code" are set in the official repository and are not stated in the paper. Where the paper gives no value, the table says "not stated".

Results

Author-reported. Setup: a 1:10 scale car in an F1TENTH simulator, with a training set on a circle track and evaluation on "The Grand Tour" track app. B. Metrics are relative to the default MPC, with values in parentheses showing the gain over that baseline tab. 2.

Decision-making accuracy tab. 1. Test set of 1600 prompt–trajectory pairs with the same RAG context for all models §4.2.

Control adaptability tab. 2, evaluated on The Grand Tour. Columns are centreline error E_C [m], velocity error E_V, reverse error E_R and smoothness E_S, all in m/s or m/s², lower is better. Ext. Fail is the count of unparseable MPC outputs. † marks averages computed without extraction failures.

Computation on the Jetson Orin AGX and RTX 3090 tab. 3. Each measurement repeated 10 times with the same prompt. Latency is the average per inference.

Quantized decision accuracy tab. 4: 3B drops from 86.82% (FP16) to 79.88% (Q5); 1.5B from 82.83% to 79.63%; 7B SFT-only from 87.32% to 86.95% app. F.

Quantized control adaptability tab. 5 on The Grand Tour, Q5 versus FP16: 3B SFT+RLVR 47.9% (Q5, no extraction failures) versus 63.3% (FP16); 1.5B SFT+RLVR 49.1%† with 3 failures versus 41.5%; 7B SFT-only 46.4% versus 58.0%.

Physical car §4.4 fig. 4: one qualitative run with a Q5 3B model on the car's Jetson Orin AGX. There are no quantitative numbers for the physical run.

Paper vs code

Official repository: ForzaETH/LLMxRobot, linked from the paper's footnote §1. Code at commit 4d896e72cf7b26908195fbb172142b7111a53358, Apache-2.0. We checked the training scripts, configs, reward functions, evaluation functions and the inference wrappers.

• Reward terms. Paper: the MPC reward has three terms eq. 3. Code: the GRPO trainer receives four reward functions: driving, format, parameter extraction and a parameter-name match. The name-match reward is not in the paper rl_mpc_train.py rl_mpc_train.py. The repo sets no reward weights.
• MPC format reward. Paper: "similar to" the decision format reward, which names three tags §3.2 §3.3. Code: the MPC format reward checks only the reasoning and answer tags, averaged over two rl_mpc_train.py. The decision format reward checks reasoning, explanation and answer rl_decision_train.py.
• Baseline RMSE values. Paper app. C: centreline 1.23, velocity 2.1, reversing 4.93, smoothness 0.38, on the Circle map. Code defaults: 1.201, 2.01, 2.84, 2.749 for centreline, velocity, reversing and smoothness, plus 0.11 for racing line, which the paper does not list eval_driving.py. The code values are used for the relative reward and are not the paper's Circle-map numbers.
• Smoothness metric. Paper: deviation of acceleration from 0 m/s² app. C. Code: root mean square of longitudinal and lateral acceleration together eval_driving.py.
• Velocity target. Paper uses 1.83 m/s for the velocity case app. C. Code default is also 1.83 eval_driving.py.
• Decision dataset and steps. Paper: eight driving styles in the static decision dataset §3.2. Code: the training raw-state folder has six files (centreline, oscillate, left wall, dont move, racing line, reversing) . The test folder has eight . Paper and code do not say which folder the 8-style dataset came from. The decision run is 800 steps in code rl_decision_train.yaml; the paper does not state it.
• Decision output format. The correctness reward reads a line "Adhering to Human: True/False" and scores a missing line as wrong rl_decision_train.py. The paper does not describe this output format.
• Inference decoding. Paper: not stated. Code: the Hugging Face pipeline samples with temperature 1.0 and top-p 0.8 inf_pipeline.py; the GGUF wrapper, which uses the llama.cpp route the paper describes for deployment, uses temperature 0.0 and top-k 1 with the llama-2 chat format inf_gguf.py inf_gguf.py.
• Default MPC parameters. Paper: "empirically obtained" §3.1. Code defaults are in llm_mpc.py; v_max is 12 there.
• SFT values. Paper: SFT "follows the procedure" of the baseline framework §3.2. The repo's SFT config targets Qwen2.5-7B with LoRA rank 16, alpha 16, learning rate 2e-4 and 150 steps sft_train.yaml sft_train.yaml. Those are defaults in the repo, not values the paper states for the 1.5B and 3B models.

No discrepancies were found in the reward clipping, the GRPO step count (750) or the choice of five retrieved memories.

Pitfalls

• Headline gain. The abstract and §4.3 give the 1.5B gain over SFT as 20.2 points. Table 2 gives 11.3% (SFT) and 41.5% (SFT+RLVR), a difference of 30.2 points tab. 2. The 3B gain of 12.9 points (50.4% to 63.3%) matches the table. Quote the table values.
• Latency. §4.4 gives an average latency of about 8.3 s for the deployed Q5 3B model at 38.78 tokens/s. Table 3 gives 3.74 s latency for that same model and hardware tab. 3. The appendix's full-stack timings are 9.12 s, 5.84 s and 9.93 s app. D. §4 states a 7.5× latency reduction for the 1.5B model. The table values, 14.88 s and 1.82 s, give about 8.2×.
• Quantization cost in control. The text emphasises 0 extraction failures for Q5 3B, but the control score falls from 63.3% to 47.9% tab. 5. The paper does not discuss that drop.
• Table 2 text. §4.3 gives 9.5% for the 1.5B RLVR-only improvement, while Table 2 gives 9.6%. Use the table.
• Source table. A commented-out copy of Table 2 in the LaTeX differs in places, e.g. the 1.5B SFT E_S entry is 4.54 there and 4.01 in the active table. The card uses the active table.
• Reward range. Eq. 3 writes the range of R_drive as "[1, −4]", which is reversed. With RMSE ≥ 0 the maximum is 1. The code's upper clip at 4 is never reached.
• Failure values. A crash or unparseable output gets RMSE 69 in the code rl_mpc_train.py rl_mpc_train.py rl_mpc_train.py. For any baseline ≤ 13.8 that clips to -4, the floor. The paper does not specify this.
• Wall-clock cost. The code waits 2 s before each completion and 5 s after setting the parameters, plus the lap time in the simulator rl_mpc_train.py rl_mpc_train.py. Each completion therefore costs at least 7 s of fixed waiting before simulation time. With 4 completions per step and 750 steps, the fixed waits alone are about 5.8 hours.
• Infrastructure. Training needs a live ROS stack, a simulator and an MPC node. The ROS host IP is hard-coded in the config rl_mpc_train.yaml. Crashed MPC nodes make the loop wait and retry rl_mpc_train.py.
• Sample size. The headline comparison uses one track, 20 paraphrased prompts app. C and five runs tab. 2. The paper gives no confidence intervals.
• The 7B model. No 7B GRPO run is reported. The 7B SFT-only model, at 58.0%, is the strongest SFT-only local model in Table 2 tab. 2.
• Generalisation claim. Training uses a circle map and evaluation a racing track app. B. The paper's claim is about this transfer, so do not extend it to other tracks without a test.
• Real car. One qualitative run, no numbers §4.4 app. D.

Lineage

• Builds on the R1-Zero approach of DeepSeek-R1 for GRPO-style verifiable rewards §1. That paper is not compiled in this wiki.
• Builds on the LLM-driven MPC framework of Baumann et al., which provides the DecisionxLLM and MPCxLLM architectures, the SFT procedure and the RAG setup §3 §3.2. That paper is not compiled in this wiki.
• Compared against: GPT-4o (cloud, no training) tab. 1 tab. 2, and Qwen2.5-7B SFT-only tab. 2.