Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence

In plain words

A robot that follows a learned policy can get stuck or drop an object, and the usual fix is to look back at the failed attempt after it ends. This paper asks for something closer to a supervisor that watches while the robot works. The robot's main policy stays fixed. Next to it sits a set of small checks written as code, which look for signs of trouble during the attempt, such as a lost grip or a stalled movement. When a check fires, the robot runs a recovery routine, for example re-approaching the object.

The checks and recoveries are not written by hand. After a batch of attempts, an automated process groups the failures, finds where each one first went wrong, proposes a new check or recovery, and keeps it only if it helps on held-out trials. The authors also built the rollout system that runs many simulated attempts in parallel, since faster rollouts mean faster learning.

TL;DR

• A frozen vision-language-action policy is wrapped by code-based runtime critics and recovery skills that are evolved from rollout failures §2.1.
• Three loops run at different timescales: per-action critics, per-batch diagnosis and repair, and per-iteration gated updates §1 §2.6.
• Author-reported: RoboCasa 18-task macro success 73.56% to 93.56% and LIBERO-Pro (40 task-setting pairs) 32.00% to 71.13% tab. 2 tab. 3.
• The abstract and introduction headline 90.8% on LIBERO-Pro; no table or sentence in the body supports that number (see Pitfalls).
• Relevant if you want to add monitoring and recovery around a frozen robot policy, and can run many simulated rollouts.

Problem

Frozen VLA and world-action policies act open-loop during inference, so a small slip can cascade into failure §2.2. Post-episode reflection can diagnose a failure but cannot intervene while it happens §1. Hand-tuning low-level control to fix one failure instance tends to degrade the policy on other seeds §2.2. Expert-in-the-loop debugging does not scale across long- tail initial conditions §2.2. The authors assume a simulator where a rollout can be reproduced from its seed, which is what makes held-out validation possible §4.1.

Method

The system has two parts: a harness H = {C, R, T} that is evolved, and a fixed policy π with a fixed orchestrator that approves interventions eq. 1.

• Runtime critics C. Monitoring functions that scan the trajectory so far and emit a structured proposal P_t, which holds evidence e_t and a suggested mode eq. 2 §2.1. A critic never executes actions app. C.
• Recovery playbook R and toolset T. Strategies keyed to a failure mechanism, and executable tools that may be generated during evolution §2.1.
• Orchestrator. Sets the final mode σ_t from the proposal, the playbook, the toolset and the task context K. An intervention is allowed only if its evidence is accepted eq. 3 §2.1. The paper names frontier multimodal models as examples of this role §2.1.
• Evolutionary agent. Revises H from failed rollouts, H^(k+1) = A_evo(D_fail, H^(k)) eq. 4 §2.1. The objective is to maximise success J(H) with π and the orchestrator held fixed eq. 5 §2.1.

Evolution uses three phases, each with a named agent §2.3 to §2.6.

1. Phase I, profiling (Loop 1). Run the frozen policy on a development seed set. Keep only valid rollouts: a rollout counts if it has a full sensor trace and video, and infrastructure failures are rerun on the same seed §2.3 eq. 6. Each trajectory is a time series of state, action, milestone-status and physical signals eq. 7. Successful trajectories form a reference index per milestone eq. 8. The first missing milestone m* is the first goal milestone never observed in the trajectory eq. 9 §2.3.
2. Phase II, diagnosis and repair (Loop 2). Failures are clustered by the earliest observable divergence t_EOD, the first step where the state leaves the reference distribution by more than a threshold ε eq. 10 §2.4. Each cluster gets a medoid seed. The diagnosis agent then checks layers in a fixed order: evaluation, critic, state representation, planning/control, recovery, parameters. It stops at the first layer that is a root cause eq. 11 §2.4. The repair agent then patches the harness at that layer only: a new or refined critic, a recovery, or a tool §2.5. Each recovery carries a VLA re-entry predicate Ψ(s_t). Control returns to the VLA only if the failure is cleared and contact is stable eq. 12 §2.5. A patch is validated by a diagnostic replay and then a fresh closed-loop rollout from the start state; it passes only if the goal milestone is reached and every intervention was approved eq. 13 §2.5.
3. Phase III, consolidation (Loop 3). Validated patches for one cluster are merged into one harness H_merged = Consolidate({H_patch,j}) eq. 14. Critics are merged with logical OR over trigger conditions. Recoveries are generalised. New tools may be added §2.6.1. The merged harness is packaged as SKILL.md (milestones, approval rules, Ψ), tools/, plans/ and params/ §2.6.1. It must then resolve all cluster failures (historical regression, 100%) and show a held-out gain ΔSR over the base policy eq. 15 eq. 16 §2.6.2. If held-out evaluation reveals a new failure, that held-out seed is moved to the development set and a fresh set is needed before the iteration closes §2.6.2.

The systems part, Z-Infra, separates agent logic from workers §3. A Control Plane routes requests. Environment workers run simulators in sessions, and rollout workers serve GPU models with batched, first-come-first-served scheduling §3.3. The programming interface offers session, episode and step primitives, including policy_step, which performs observe, infer and step in one call tab. 1 [lst. 1] §3.4.

The critic-governed action loop and one evolution round are shown below as pseudocode. Both are our paraphrase of the paper §2.2 alg. 1 §2.3 to §2.6. In the first, a recovery runs only when the orchestrator approves it.

# Critic-governed action loop  [§2.2] [alg. 1]
# Input:  frozen VLA; runtime critic; recovery library; orchestrator
# Output: task success or a bounded failure record

def run_episode(vla, critic, recoveries, orchestrator, task):
    # Repeat until the official predicate holds or the budget runs out  [alg. 1]
    while episode_is_active():
        # One action chunk from the VLA, or from a recovery  [alg. 1]
        window = run_next_chunk(vla, recoveries)
        # The critic returns a proposal only  [app. C]
        proposal = critic(window)
        if task_predicate_satisfied(task):
            return "success"
        if proposal.reports_normal_progress():
            continue
        # The orchestrator approves each intervention  [§2.1] [eq. 3]
        if orchestrator.approves(proposal, recoveries):
            run_recovery(recoveries, proposal)
    return "bounded failure record"

# One evolution round, Phases I to III  [§2.3] [§2.4] [§2.5] [§2.6]
# Input:  harness; development seeds; held-out seeds
# Output: merged harness, or the previous harness

def evolve_round(harness, dev_seeds, held_seeds):
    # Loop 1: roll out the frozen policy on development seeds  [§2.3]
    corpus = rollout_all(harness, dev_seeds)
    # Keep valid failures only; infrastructure failures are rerun  [§2.3]
    failures = [r for r in corpus if r.valid and not r.success]
    # Cluster by earliest divergence and take one medoid per cluster  [§2.4]
    for cluster in cluster_by_divergence(failures):
        # Diagnose the medoid, checking layers from top to bottom  [§2.4]
        layer = diagnose_top_down(cluster.medoid)
        # Patch the layer that was found  [§2.5]
        patch = repair(harness, layer, cluster.medoid)
        # Validate by a diagnostic replay, then a fresh rollout  [§2.5]
        if not validates(patch, cluster.medoid):
            continue
        # Generalise the validated patches across the cluster  [§2.6.1]
        candidate = consolidate(harness, cluster, patch)
        # Gate: all cluster seeds pass, then a held-out gain  [§2.6.2]
        if historical_regression_passes(candidate, cluster):
            if held_out_gain(candidate, held_seeds) > 0:
                harness = candidate
    return harness

# VLA re-entry predicate  [§2.5] [eq. 12]
# Input:  evidence e_t; contact signals phi_t; threshold gamma
# Output: True if control may return to the VLA

def may_resume_vla(evidence, contact_signals, gamma):
    # The failure evidence must be cleared by the recovery  [§2.5]
    failure_cleared = not evidence.still_present()
    # The paper defines Stability as the magnitude of torque oscillation
    # and grasp force, yet requires Stability > gamma  [§2.5] [eq. 12]
    stable = stability(contact_signals) > gamma
    return failure_cleared and stable

Equation numbers follow the order of the LaTeX source. Figures and tables are numbered in source order. They were not checked against the compiled PDF.

Hyperparameters

Results

Author-reported. Setup: RoboCasa Atomic-Seen, 18 tasks, GR00T N1.5 baseline §4.4 tab. 2. LIBERO-Pro, 10 tasks per suite and perturbation, π0.5 baseline §4.4 tab. 3. Each LIBERO entry is 20 episodes, so one episode is 5 points.

RoboCasa, success rate (%) tab. 2:

Task identities T1 to T18 are in tab. 4. The paper says the gain is consistent across tasks §4.4.

LIBERO-Pro, success rate (%) tab. 3. Task indices are in tab. 5.

Macro averages over the 40 pairs: π0.5 32.00%, Zetta 71.13%. Zetta improves 32 pairs and matches the baseline on 8 §4.4. I checked these averages against the rows and they agree.

Other reported numbers, all author-reported:

Aha values in §4.2 refer to internal checkpoints v0 to v2 of a single failure cluster, not outer rounds. Goal-S6 (5% to 90%) is a checkpoint value and does not match the final Goal-S task 6 value in tab. 3.

Paper vs code

Code: the repository linked from the project page, at commit 21d7a3afd6ca5a34de9d0350f0158252c5c55467. The LaTeX source has a commented-out code link, and the rendered paper has none. We used the link on the project page, https://air-embodied-brain.github.io/zetta, which points to the repository above. Checked: the evolution gates, the critic, the clustering and the campaign store. Not checked: the full runtime, the VLA re-entry predicate Ψ, and the validation of the reported numbers.

• Acceptance gate has a significance test the paper does not state. The paper accepts a candidate on 100% historical regression and ΔSR > 0 §2.6.2 eq. 15 eq. 16. The code's same-seed default is 1.0, which matches gating.py gating.py. The held-out check is a paired one-sided exact McNemar test on discordant seeds, with α = 0.025 and at least one discordant win, applied in two stages of 10 and 50 preregistered seeds gating.py gating.py gating.py. The paper describes neither the test nor the stages.
• Held-out mode can be "test" or "validation". The code records a heldout_mode that changes whether a held-out result can promote a candidate protocol.py store.py. The paper has no such option.
• Campaign phases are not in the paper. The store has a same-seed phase, a time-boxed transition to held-out, and promotion only after a held-out gate store.py. The paper describes one sequence.
• Clustering method differs from the description. The paper partitions failures by earliest divergence plus milestone and object context §2.4. The code uses complete-link clustering over segment similarity (token and vector cosine) clustering.py clustering.py. The medoid is chosen by a minimum clustering.py. Each failure segment stores its earliest divergence step models.py.
• Critics are rule-based temporal predicates. A rule has a feature, a predicate, a dwell window and a consecutive-count condition. A false activation guard clears the history, and a cooldown follows a firing critic.py. The paper describes critics as code, but gives no thresholds.
• Repository license is unknown. pyproject.toml names a LICENSE file pyproject.toml, and the clone contains none. The notices file says some code derives from RPent, which has no license text, and from RLinf, which is Apache-2.0 THIRD_PARTY_NOTICES.md. We have not verified any license for reuse. Do not copy code from this repository into a page without settling this.
• Hardware conflict in the paper. §4.1 says 8 × RTX 4090 §4.1. §4.6 says 8 × A100 §4.6. The repository does not settle which was used.

Pitfalls

• The 90.8% headline is not traceable. The abstract and introduction report 90.8% on LIBERO-Pro, and "34.5% to 90.8%" for scaling [abstract] §1. The body gives 71.13% (40-pair macro) and 92.5% (Goal-T) and 89.0% (Goal-S) per setting, and 31.0% to 92.5% (Goal-T) and 38.0% to 89.0% (Goal-S) for scaling §4.3 tab. 3. Use the body's numbers. Do not reuse 90.8% without a source.
• Latency claims disagree. The introduction says a 91% latency reduction, which is 11.1× §1. §4.6 says per-episode latency falls 11.9× against RPent §4.6. These may measure different things, but the paper does not say so.
• The 20.6× throughput figure mixes conditions. The introduction compares 35.1 episodes per minute (concurrency 64) with RPent's 1.7 §1. The body's like-for-like ratio is 12.8× at concurrency 16 (22.09 against 1.72) §4.6.
• The orchestrator's role is ambiguous. §2.1 has the orchestrator approve every intervention during execution eq. 3 alg. 1. §4.6 says agents are invoked "only during offline Reflection and Evolve phases" §4.6. Which one runs in the evaluations is not stated. The LLM used is not named.
• Goal-task labels conflict. The mapping table names Goal task 2 as "put the wine bottle on top of the cabinet" and task 8 as "put the bowl on the plate" tab. 5. §4.2 and Appendix D call Goal-T2 "wine bottle in bowl", and Goal-T8 "wine bottle on plate" §4.2 app. D.
• The re-entry predicate's sign is unclear. Ψ requires Stability(s_t) > γ, but Stability is defined as the magnitude of torque oscillation and grasp force §2.5 eq. 12. Read literally, large oscillation passes. The likely intent is the opposite. Ask the authors or test both.
• The held-out protocol is underspecified. The paper says a moved held-out seed needs a fresh set §2.6.2, but does not say which numbers in the tables came from a single untouched held-out run. The LIBERO-Pro dev stop rule (≥ 50% on dev) also ties the stop point to the dev seeds §4.1.
• Small samples and no variance in tables. LIBERO-Pro uses 20 episodes per entry. Some figure captions cite Wilson 95% intervals fig. 7 fig. 8, but the tables give no intervals and no seed variance §4.4.
• Ψ thresholds and dwell are missing. γ, ε and critic dwell windows are not stated eq. 10 eq. 12 §2.2. The code sets dwell per rule. Those values are in configuration, not checked here.
• Equation and figure numbers are from source order. Verify against the PDF before citing them in another page.
• Cost of the loop is not reported. Wall-clock time per round, number of candidates tried and orchestrator calls per round are not stated.

Lineage

• Builds on: RPent, the agent framework the code was adapted from, and the evaluation baseline for throughput and latency §4.6 THIRD_PARTY_NOTICES.md. RLinf (Apache-2.0) contributed selected embodied runtime files THIRD_PARTY_NOTICES.md. Ray for distributed execution §3.3.
• Builds on, concepts: SkillOpt and EmbodiSkill for code-space, SGD-like optimisation of skills and reflection §1.
• Compared against: frozen π0.5 and GR00T baselines §4.4; RPent for throughput and latency §4.6; "Ours without Z-Infra" §4.6.
• Superseded by: not known from content in this wiki.