Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

In plain words

Teaching a robot a new task usually means a person writes the code for each skill, designs the rewards, and wires up the camera-to-motion logic. This paper asks whether a robot can improve that code by practising on its own. The system has a library of small reusable skills, such as pick, place or push, and a written prompt that tells an AI model how to combine them. Neither the AI model nor its weights are changed.

The practice happens in a simulator. The system first picks useful behaviours from a large collection of recorded robot demonstrations and builds practice tasks that resemble them. It then tries those tasks. A second copy of the agent also sees the simulator's hidden state, which tells the system what really went wrong. A video-reviewing step compares the two runs and suggests a change to a skill or to the prompt. Each change is tested on the whole task set, and only changes that help without breaking other tasks are kept. Changes that pass are merged and retested again.

At the end the system is frozen and run on a real robot, with only a common calibration step first. The idea matters because the improved code is shared across tasks, so one repaired skill can help several tasks at once.

TL;DR

• RPG is a practice loop for an LLM-driven robot: a runtime agent runs simulated tasks, a privileged agent and a video analyser diagnose failures, and a coding agent revises shared skills and the prompt §3.3.
• Only revisions that raise mean success without any task losing more than one of five trials are kept, and merged systems are re-tested §3.4 eq. 2.
• Held-out success on 22 simulated manipulation tasks goes from 28.6% after round 1 to 95.0% after round 15, against 75.5% for ASPIRE and 60.0% for CaP-Agent0 with GPT-6 Astra Pro (author-reported) tab. II.
• The frozen system completed 30 of 30 trials on three physical tasks, with calibration shared with the baselines §4.5 tab. V.
• No official code was found; an implementer must rebuild the framework from the description.

Problem

Robot execution systems built on an MLLM plus code-as-policy skills need substantial human effort to write skills, design rewards and integrate perception with control. Reusing a fixed skill set without updating it limits how far a single system generalises across tasks §1. The paper argues that editing the shared skill code and the system prompt, rather than fine-tuning a model, can make the robot better across tasks, and that such edits must be checked across tasks because a fix for one task can break another §1 §3.4.

The paper also notes that the frequent model calls of an agent-as-policy design make low-level control slow, while pure code-as-policy is hard to specify for judgements such as grasp success §1.

Method

RPG has three stages: Reconstruct (build practice tasks from an offline dataset), Practice (iterate on the system) and Go Real (calibrate and freeze for deployment) §3 fig. 2.

Execution system. The system is a pair S_r = (L_r, rho_r): a shared skill library and a system prompt. A multimodal Runtime Agent selects and composes skills to finish a task. Model weights and the agent harness are not changed during practice §3.1 app. Self-Improvement Protocol. A skill takes arguments, runs a sequence of robot actions, then reports whether it succeeded and what it measured §3.1.

Initial library. Nine low-level tools from CaP-X cover state queries, perception, inverse kinematics, motion and gripper control. The Constructor builds six higher-level skills from them, covering grasping, placing, a combined pick-and-place, handing an object between arms with the table as support, moving above a target, and pushing. The initial library L_0 has 15 entries, and S_0 = (L_0, rho_0) is the starting system for the ablations §3.1.

Reconstruct. For each selected source task, the Constructor reviews 3 to 8 trajectories with their videos and descriptions, identifies the manipulation capability, and proposes a simulation practice task. The budget is 7.5 minutes per source task. A practice task can match only a portion of its source task, or a related manipulation, and need not copy the source scene §3.2 tab. I. The Constructor builds a MuJoCo scene with the YAM robot model and writes a check_success function for each task. Humans check that this function measures the intended objective before practice starts. The task set and its evaluators are then frozen §3.2.

Practice round. Each round has the following parts, in order §3.3 app. Self-Improvement Protocol:

• Execution: the Runtime Agent tries each task five times on fixed development seeds. In parallel, a Privileged Agent runs the same initial states with the same model and library, but it also receives simulator state (object names, poses, sizes, velocities and cloth-corner positions). Skills themselves never receive privileged state §3.3.
• Diagnosis: on development tasks, a Video Analyzer compares failed Runtime runs with Privileged runs and, where available, related source videos. It finds the first observed failure, proposes a change and does not edit the system §3.3 fig. 3.
• Candidate revision: the Implementor may propose new skills, changes to skills and prompt changes. A development task below 5 of 5 successes can generate at most one candidate. Every candidate starts from S_r. Each candidate must load, have valid syntax and interfaces, and use only deployable observations §3.3.

Validation and retention. Each candidate is tested with complete Runtime episodes on all 22 tasks, using the same five development seeds per task, so 110 episodes per system §3.4. Mean success is computed as in eq. 1. The candidate passes the gate in eq. 2 when mean success rises and no task's success drops by more than one of five §3.4 app. Self-Improvement Protocol.

The Merger then combines the eligible candidates into S+. Candidates are sorted by their gain in mean development success. The best one starts the merged system, and the rest are added greedily, with at most eight integration attempts per round. Changes to the same file are combined with a three-way merge against the common parent, and a coding agent resolves what cannot be merged mechanically app. Self-Improvement Protocol. S+ is tested again with the same gate against S_r. If it passes and is retained, S_{r+1} = S+; otherwise S_{r+1} = S_r §3.4 fig. 3.

Gate, as printed in the paper:

\widehat q_i(S)=\frac1n\sum_{k=1}^{n}e_i(\tau_{ik}(S)),\qquad \widehat J(S)=\frac1K\sum_{i=1}^{K}\widehat q_i(S) eq. 1

\mathsf{Gate}(\widetilde S;S_r)=\mathbf1\!\left[ \Delta\widehat J>0\ \land\ \min_i\Delta\widehat q_i\geq-0.20 \right] eq. 2

Here n = 5 and K = 22, and e_i is the binary evaluator of task i §3.4 app. Self-Improvement Protocol.

The practice loop as described in the appendix, written as pseudocode. The paper has no algorithm float, so the structure is reconstructed from the text.

# RPG practice round  [§3.3] [§3.4] [app. Self-Improvement Protocol]
# Input:  system S = (library, prompt); tasks W (K = 22 tasks)
# Output: next system, either merged and retained or the previous one

def practice_round(system, tasks):
    # 1. Runtime and Privileged Agents each run 5 dev seeds per task  [§3.3]
    runtime_runs = run_agent(system, tasks, privileged=False, seeds=5)
    privileged_runs = run_agent(system, tasks, privileged=True, seeds=5)

    # 2. Diagnose failed Runtime runs on development tasks  [§3.3]
    diagnoses = analyze_videos(runtime_runs, privileged_runs)

    # 3. At most one candidate per dev task below 5/5 successes  [§3.3]
    candidates = []
    for task in dev_tasks_below_five_of_five(runtime_runs):
        candidates.append(implement(system, diagnoses[task]))

    # 4. Keep candidates that pass the gate against the round start  [§3.4]
    eligible = [c for c in candidates if gate(c, system, tasks)]

    # 5. Greedily merge, best first, at most 8 attempts  [app. Protocol]
    merged = greedy_merge(sort_by_gain(eligible), max_attempts=8)

    # 6. Retain the merged system only if it passes the same gate  [§3.4]
    if merged is not None and gate(merged, system, tasks):
        return merged
    return system


def gate(candidate, system, tasks):
    # 7. Mean success must rise and no task may lose 2 or more of 5  [eq. 2]
    delta_mean = mean_success(candidate, tasks) - mean_success(system, tasks)
    worst_drop = min(
        task_success(candidate, t) - task_success(system, t) for t in tasks
    )
    return delta_mean > 0 and worst_drop >= -0.20

Go Real. The final system and CaP-Agent0 both receive the same coordinate calibration and hardware adaptation, which includes pad offsets and the empty-gripper aperture. There is no object-specific tuning. The physical runs use low reasoning effort. The Privileged Agent and Video Analyzer are not used on the robot §3.5 §4.5.

Hyperparameters

Values are as stated in the paper. Where a value is not stated, the table says so. No code was available, so no code-only values are listed.

Results

Author-reported. Setup: 22 simulated bimanual manipulation tasks (YAM station), 10 held-out initializations per task, 220 episodes per system, round-end systems evaluated retrospectively after the practice run §4.1 §4.2. Compute hardware: not stated. Confidence intervals: not reported in the text read.

Headline numbers:

Overall held-out success by round, in percent, as reported in the appendix table: 29, 38, 43, 74, 78, 78, 82, 83, 83, 84, 87, 87, 92, 91, 95 for rounds 1 to 15 app. Round-by-Round Task Evolution.

Per-task simulation results (successes out of 10 held-out seeds; bold in the paper marks the highest count per row, and it is not repeated here) tab. II:

Table III, five-round ablations from the same 15-entry start, each evaluated on 220 held-out episodes tab. III:

Table IV, library-only comparison with the runtime fixed (Gemini 3.8 Flash, xhigh), ten paired initializations per task tab. IV:

Table V, physical trials, ten per task and method, intermediate and complete results from the same trials tab. V:

Five further physical tasks (open scissors, uncap marker, pull out tissue, erase whiteboard, unscrew bottle cap) were shown qualitatively with no task tuning and no quantitative comparison fig. 6 §4.5.

Paper vs code

No official code.

Searched: the paper text and bibliography have no code link. The project page (https://rpg-robot.github.io, cited in the paper) gives "Code coming soon" on its code button, and the only repository it links is the template for the page, which the page says it adapts. GitHub repository search for "rpg-robot", "agent-as-policy" and "capx robot" returned no repository from the authors, and the organisation vision-language-kinematics holds only its website repository. The baselines' own code (CaP-X, ASPIRE) is not this paper's code and was not used.

Pitfalls

• No code, so every detail is a reconstruction. The implementer must rebuild the Constructor, Runtime Agent, Privileged Agent, Video Analyzer, Implementor and Merger, plus the simulator, the YAM model and the 22 task evaluators. Asset sources, the initial system prompt text (rho_0) and the prompts of the six agents are not stated in the sections read. Build the evaluators first and have humans check them, as the paper does §3.2.
• Privileged state is simulation-only. The Privileged Agent and the failure diagnosis depend on object poses, velocities and cloth-corner positions. Without a simulator that exposes this state, the diagnosis step cannot be reproduced in the same way §3.3.
• Dataset videos matter for diagnosis. The Video Analyzer uses related source videos from the offline dataset when available, and the dataset used here is ABC. The contribution of dataset videos to the gains is not isolated in the paper, except in the sense that removing the analyzer also removes its access to them §4.3.
• Gate noise. Development success is measured on five seeds, so each per-task value moves in steps of 20 percentage points. The gate needs only a positive mean change over 110 episodes, which is a noisy signal. The paper itself reports held-out drops at rounds 6 and 14 that passed the development checks §4.2 app. Self-Improvement Protocol.
• Round-15 is not a checkpoint choice. The held-out curve is computed after the run on saved round-end systems, and the paper says held-out seeds are not used for selection. Still, the 28.6% and 95.0% endpoints are the saved round 1 and round 15 systems on the same 10 seeds per task §4.2.
• Ablation numbers overlap with the main run. The full-RPG row of Table III (78.2%) equals round 5 of the main curve. The text describes the ablations as five-round runs from the same 15-entry initialisation. Whether the full row is the main run's round 5 or a separate run is not stated tab. III §4.3 fig. 4.
• Prompt and library changes are confounded. The largest single gain, from 43.2% to 73.6% between rounds 3 and 4, came from a prompt revision that also changed how the agent spends its observation budget, and no skill was edited in that round. The paper says this gain is the combined effect of the prompt update §4.2.
• Library-only comparison is small. Table IV uses four tasks and ten paired initializations. A scripted privileged caller gives a larger gain (19/40 to 39/40), but the same comparison also loses one success on ABC sorting (6/6 to 5/6 after five extra paired trials). The paper uses this to argue for task-level validation §4.4.
• Towel folding does not improve. The standard towel task peaks at 5/10 in held-out evaluation, and the hard variant (1 cm corner tolerance) stays at 0/10 in every round. Seven of RPG's 11 simulated failures are towel folds. The paper attributes this to deformable state that the rigid-object skills cannot track §4.2 app. Limitation Case: Deformable Manipulation.
• Sim and physical criteria differ. The physical towel criterion (footprint ratio 0.40 to 0.60, parallel sides) is not the simulated one (42 to 58% area, plus three geometric conditions). The paper says it is not intended to match difficulty §4.5 app. Simulation Task Suite.
• Different reasoning efforts. Simulation uses xhigh; physical trials use low to reduce cost and latency. Results at one effort level may not carry to the other §3.5 §4.1.
• Baseline reproduction. CaP-Agent0 and ASPIRE were reproduced from their released code, and RATs was evaluated after 50 iterations of self-directed play. The table caption says RATs reports any-time success. The reproduction details beyond this are not stated tab. II.
• Cost and throughput. The authors name API rate limits as the main limit on parallel rollouts, and note that overlapping revisions to shared skills get harder to integrate as the task set grows §5.
• Non-monotonic progress. Several tasks drop between rounds (for example, Place dish on rack 4/10 in round 10 after 8/10 in round 9, and Nut assembly 3/10 in round 6 after 10/10 in round 5). Do not assume a round is safe to stop at app. Round-by-Round Task Evolution.
• Feedback-held-out transfer is partial. Grasp plate, an extra task outside the main 22, reaches 10/10 in rounds 10 and 11 and falls to 6/10 later. The paper does not claim universal transfer [app. Transfer without Direct Task-Specific Improvement Feedback].
• Physical results are small. Three tasks with ten trials each, with retries allowed inside a trial and a single grasp offset. No physical variance or confidence interval is given. The five extra physical tasks are qualitative only §4.5.

Lineage

• Builds on: the code-as-policy line of work, for example arXiv:2209.07753 §1 and CaP-X, which supplies the nine base tools and the CaP-Agent0 baseline §3.1 tab. II. ASPIRE is arXiv:2607.00272 tab. II.
• Compared against: CaP-Agent0 (CaP-X) with four models, ASPIRE, and RATs (Playful Agentic Robot Learning) tab. II tab. V.
• Data: the offline dataset is ABC, arXiv:2606.27375 §3.
• Superseded by: not stated.