RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning

In plain words

Robot foundation models are large pretrained policies that can be taught a new task with a modest amount of extra data. How well they do depends heavily on that data. Collecting it by having a person steer the robot is slow, and the person's moves vary in quality and style. That noise hurts most on precise jobs, such as pushing a cable plug into a socket.

The idea is to let a different learner write the practice data. For each task, a reinforcement-learning agent learns by trial and error, guided by a reward that says whether the attempt succeeded. Once it succeeds reliably, its successful runs become the fine-tuning set for the big generalist policy, which then copies that behaviour. Think of a specialist who masters one skill alone and then writes a clean practice log for a student who needs to learn it.

This matters because the generalist keeps its broad abilities while picking up the precision of a specialist. The paper reports that, on several precise tasks, this beat fine-tuning on the same number of human demonstrations. The resulting generalist also handled new connectors and scenes better than the specialist that produced its data.

TL;DR

RLDG trains one reinforcement-learning expert per task on a real robot with HIL-SERL, rolls out the successful episodes, and fine-tunes OpenVLA (with LoRA) or Octo on that data instead of human demonstrations §3. On precise connector insertion and FMB tasks, the authors report 10% to 37% higher success rates than fine-tuning on human demonstrations, and better generalisation than the RL experts §4.2 §4.3. The evidence is small real-robot trial counts (20 or 80 per condition) with no error bars. The code release covers only the RL-expert stage; the generalist data-collection and fine-tuning parts are marked "Coming soon".

Problem

Fine-tuning a robot foundation model needs task data, and human teleoperation is the usual source §1. Human demonstrations vary in execution quality and style, which makes it hard for a generalist to learn robust policies, and the problem is worst for contact-rich, dexterous tasks §1. Human demonstrations can also differ from what the robot policy sees and does, for example in viewpoint, memory and task knowledge §3.

Fine-tuning the foundation model directly with reinforcement learning is, according to the authors, hard in practice: optimisation is unstable, compute is expensive, and pretrained capabilities may be forgotten §1. The method assumes each task has a reward function. Here it is a binary success classifier trained on teleoperated positive and negative examples app. B. The authors name specifying rewards for new tasks as a limitation §6.

Method

RLDG has three stages §3.

1. Train an RL expert per task. Each task is a Markov decision process whose state is the RGB image plus proprioception and whose action is a desired end-effector movement. The expert maximises the expected discounted return §3.1 eq. 1. The authors use HIL-SERL, which adds human interventions to RLPD, and train until the policy reaches 100% success §3.1 app. B.
2. Collect the fine-tuning data. Roll out each converged expert and keep the successful episodes app. B. For tasks with one expert per object, use an equal number of episodes per object §3.2. For a multi-stage task, the expert covers only the critical phase, and human demonstrations cover the rest §3.2.
3. Fine-tune the generalist with supervised learning. The loss is the negative log-likelihood of the recorded actions given the observations §3.3 eq. 2. The generalist is then trained on this dataset only; the paper treats the data as the only channel from RL to the generalist §3.2.

The two generalists differ in their action heads §3.3:

• OpenVLA: a 7B-parameter vision-language-action model built on Llama 2. It predicts 7 action dimensions, each discretised into 256 bins, with a cross-entropy loss over the bins §3.3. The pretrained weights come from Open X-Embodiment training §3.3. Fine-tuning uses LoRA §3.3 app. B.
• Octo: predicts continuous actions with a diffusion head. The transformer reads the tokenised observation and goal and outputs an embedding that conditions a denoiser trained with the standard DDPM objective §3.3. The Octo-Base model is used with its secondary image tokenizer removed and its image goal masked, and the rest of the network is fine-tuned directly §3.3.

Equations as written in the paper:

J(\pi) = \mathbb{E}_{s_0 \sim \rho_0,\; a_t \sim \pi(a_t \mid s_t),\; s_{t+1} \sim P(s_{t+1} \mid s_t, a_t)} \left[ \sum_{t=0}^{T} \gamma^t R(s_t, a_t) \right] eq. 1

Here \rho_0 is the initial robot configuration distribution, P the transition dynamics and R the task reward §3.1.

\mathcal{L}(\theta) = -\mathbb{E}_{(s_t, a_t) \sim \mathcal{D}} \left[ \log \pi_\theta(a_t \mid s_t) \right] eq. 2

The method is independent of the RL algorithm and of the generalist architecture §3. The implementation, however, is HIL-SERL §3.1.

# RLDG: distil per-task RL experts into one generalist  [§3]
# Input:  tasks (each with a success reward), generalist policy pi_0
# Output: fine-tuned generalist policy

def rldg(tasks, generalist, human_phases):
    # Train one RL expert per task with HIL-SERL, from 20 demonstrations,
    # until success is 100%; the reward is a success classifier  [§3.1] [app. B]
    experts = [train_hil_serl(task, initial_demos=20) for task in tasks]

    # Roll out each converged expert and keep only successful episodes
    # [§3.2] [app. B]
    rl_episodes = []
    for expert, task in zip(experts, tasks):
        rl_episodes += rollout_successes(expert, task)

    # Use the same number of episodes for each object  [§3.2]
    rl_episodes = balance_per_object(rl_episodes)

    # Multi-stage tasks: add human demonstrations for the non-RL phases  [§3.2]
    dataset = rl_episodes + human_phases

    # Fine-tune with the supervised action log-likelihood loss  [§3.3] [eq. 2]
    for batch in dataset:
        loss = -log_prob(generalist(batch.observation), batch.action)
        generalist.update(loss)

    return generalist

Generalists receive only the wrist camera image as input; the RL expert also receives end-effector pose, velocity and wrench §4.1.

Hyperparameters

Results

Author-reported. Setup: a Franka Emika Panda arm with a wrist camera fig. 2; OpenVLA and Octo fine-tuned on RL data versus the same number of successful human-demonstration episodes §4.2. Per-bar values appear only in the figures and are not transcribed here.

Headline claims in the abstract and introduction are averages or maxima that the tables above do not fully reproduce; see Pitfalls.

Paper vs code

Repository: generalist-distillation/RLDG at commit 8c4ba163135a82458a952f06302acf6b60072c93 (shallow clone; HEAD dated 2024-12-17). The paper's abstract points to a project website, and its owner is the generalist-distillation organisation, which also hosts that site's repository. This card treats the repository as the authors' code. That is a judgement from the shared organisation name, not a link from the paper. It contains a README, a HIL-SERL experiment folder for connector insertion (config, wrapper, learner and actor launch scripts) and a rollout script README.md.

• Generalist code is missing. OpenVLA and Octo fine-tuning, and data collection, are marked "Coming soon" README.md. All generalist details in this card come from the paper only.
• One connector config, three experts claimed. The paper trains experts on USB, Ethernet and VGA §4.1. The README says three policies were trained README.md, but the repository has a single connector_insert config with one target pose config.py. The per-connector settings are not included.
• Image resolution differs. The paper uses 128 x 128 wrist images §4.1. The code uses a 1280 x 720 camera frame, crops it to the lower middle part, and sets IMAGE_RESOLUTION to 224 config.py config.py. The resize step is not in the repository.
• Successful-only rollouts match. The appendix filters failed trajectories app. B. The rollout script keeps only successful episodes [code: hil-serl/rollout_rl.py#L90-L93]. It samples actions stochastically (argmax=False) rollout_rl.py; the paper does not say how actions are chosen during rollouts.
• Reset randomisation matches. The reset range of plus or minus 5 cm and the 5 cm height above the port agree with the appendix [code: hil-serl/connector_insert/config.py#L33-L35] app. A.
• Values the paper does not state. The code sets a 100-step episode limit and a reward threshold vector config.py config.py. It also sets stiffness, damping, integral and clip parameters for a precision mode, set before reset, and a compliance mode, set after reset config.py wrapper.py wrapper.py. The paper gives only the 1 kHz controller rate §4.1.
• The rollout script is not runnable from this repository alone. It uses rng before any definition in the file rollout_rl.py. It reads FLAGS.learner, FLAGS.save_video and FLAGS.seed [code: hil-serl/rollout_rl.py#L31-L39], which are not defined in this file. It also reads config.encoder_type and config.discount rollout_rl.py, which the repository's TrainConfig does not define config.py. These come from the HIL-SERL framework, which is not included. The learner and actor scripts call train_rlpd.py from HIL-SERL run_learner.sh.
• Paths in the README do not match the repository. The README refers to rldg/hil-serl/... README.md; the files sit at hil-serl/....
• Dataset not examined. The project page links a Hugging Face dataset. It was not opened and no data from it is used here.

No license file exists in the repository, so no code is quoted; code is cited by path and line only.

Pitfalls

• Headline numbers do not line up. The abstract says "up to 40%" higher success. The per-task figures in the text reach 37% (Connector Insertion, Octo). The introduction's "30%" and "50%" averages are not derived in the text. Use the per-task numbers in the table above.
• Percent or points is unspecified. "33% and 23% higher" and the other gains do not say whether they are absolute points or relative gains. The Pick and Place counts give absolute differences (for example 16/20 to 19/20).
• Small trial counts, no uncertainty. Most conditions use 20 trials, the unseen connector comparison uses 80. The text gives no confidence intervals and does not state the number of training seeds.
• Sample-efficiency factor is not derived. The introduction says "6 to 10 times" less data, and elsewhere "6 to 10 times more" human demonstrations. Only the VGA scaling point is given in counts, 45 RL episodes against 300 demonstrations (about 6.7 times). The RL experts' own real-world training (1 to 3 hours per task app. B) is not counted in these comparisons.
• State coverage is not shown. The abstract attributes the gain to both "optimized action distributions" and "improved state coverage". The compiled analysis tests only action quality, by relabelling human states with RL actions on FMB Insertion §5.1. The paper concludes that action quality contributes most, but the state-coverage measurement is absent from the compiled text (a critical-state analysis exists only as commented-out source).
• Matched counts, not matched sources. Both arms use the same number of successful episodes §4.2. The human arm uses successful expert demonstrations app. B; the text does not describe how those demonstrations were collected or how many operators provided them.
• Reward design is a hidden cost. The RL side needs a success classifier built from teleoperated positive and negative examples app. B. The authors name reward specification as the main limitation §6.
• Speed optimisation can hurt distillation. RL favours speed, and on Pick and Place the distilled policy sometimes releases the object too early §5.2 §6.
• Control-rate mismatch. OpenVLA runs at 4 Hz against 10 Hz for the RL expert, which the authors blame for its slower cycle time §4.2.
• Octo does not fit its data. The authors report that the fine-tuned Octo policies could not fit the fine-tuning dataset perfectly, which lowers success and lengthens cycle time §4.2.
• Different adaptation methods across models. OpenVLA is fine-tuned with LoRA (rank 32) and Octo with full fine-tuning app. B. Cross-model comparisons mix this difference with the action head difference.
• Input asymmetry. The RL expert uses proprioception and the wrist image; the generalist uses the wrist image only §4.1. The distilled policy therefore sees less than its teacher, and the paper does not ablate this.
• Missing values for a re-implementation. Learning rates, epochs, step counts, the reward threshold, the episode limit and RL details beyond HIL-SERL are not stated in the paper. The code supplies some of them, for the connector task only.

Lineage

• Builds on: HIL-SERL, arXiv:2410.21845, which supplies the expert training and the success-classifier reward §3.1 app. B. SERL is cited for the underlying real-world RL framework §1. OpenVLA, arXiv:2406.09246, and Octo, arXiv:2405.12213, are the generalists fine-tuned §3.3. LoRA is the parameter-efficient method used for OpenVLA §3.3.
• Compared against: fine-tuning the same generalists on human demonstrations §4.2; the RL experts themselves §4.3.
• Superseded by: not known from content in this wiki.