Act2Goal: From World Model To General Goal-conditioned Policy

In plain words

A robot can be told what to do in three ways: by words, by a reward, or by a picture of the finished scene. This paper uses the picture. You show the robot what the table should look like when the job is done, and the robot works out how to get there.

The hard part is the middle of a long task. A policy that only looks at "now" and "the goal" tends to drift once the task has many steps. Act2Goal first imagines a sequence of in-between camera frames that lead from the current view to the goal, using a video model. It then picks a few of those imagined frames at two time scales: many close-up frames for the next moves, and a few far-away frames to keep the whole task on track. An action generator reads these frames and produces the motor commands.

The robot can also get better after it is deployed. It stores what it did, treats the place where it ended up as if that had been the goal, and fine-tunes a small add-on to the model. No reward signal is needed.

TL;DR

Act2Goal is a goal-image robot policy built from a goal-conditioned video world model, a flow-matching action expert and a multi-scale frame selection rule (MSTH). It is trained offline by imitation and can be adapted online with LoRA using relabelled goals. The authors report success rates on Robotwin 2.0 and on three real-robot tasks, and a jump from 0.30 to 0.90 on an out-of-distribution real-robot task after online adaptation. All numbers are author-reported.

Problem

Goal-conditioned policies map the current observation and a goal image straight to actions. The authors argue that this single-step prediction gives no explicit model of task progress, so performance falls on long-horizon and out-of-distribution tasks §1 §2. Planning the whole trajectory is globally consistent but brittle to deviations, while short-horizon control is reactive but loses direction over long tasks §1.

The paper assumes a robot with multi-view cameras, a proprioceptive state, and demonstration data (AgiBot World plus a small proprietary dataset) app. Implementation Details.

Method

The system has three parts. The goal-conditioned world model (GCWM) generates intermediate visual states. MSTH selects which generated frames are kept at which time scale. The action expert, which mirrors the world model's block structure at a smaller width, generates actions through layer-wise cross-attention §3.1 fig. 3. Training has two offline stages §3.3 and an optional online stage with relabelled goals §3.4 alg. 1.

Goal-conditioned world model §3.1. The authors start from the Genie Envisioner architecture, add a goal-image input concatenated along the sequence of hidden states, and remove language conditioning §3.1. Current observation and goal are encoded by a VAE into latents z_t and z_g. A flow-matching network f_θ generates the latent frames between them from noise eq. 1. At inference, the latent is refined over N steps with a fixed step size of 1/N using the learned velocity field eq. 2. The predicted latents are decoded by the VAE into visual states §3.1.

Action expert §3.1. The action expert has the same number of blocks as the world model with reduced width. It predicts an action trajectory by flow matching, conditioned on the proprioceptive state c_p and on per-layer features c_w taken from the world model eq. 3. Actions are refined over N steps in the same way eq. 4.

MSTH frame selection §3.2. Given an imagined horizon K, a proximal horizon P and a visual stride r, the visual states are split in two. The proximal segment takes states at every r steps up to P. The distal segment takes M states whose offsets follow a logarithmic spacing from P to K eq. 5. The action sequence follows the same split: proximal actions exist at every step up to P, and distal actions sit at the distal offsets. At deployment only the proximal actions are executed; distal actions only guide the plan §3.2.

Stage 1 offline §3.3. The pre-trained world model is fine-tuned to generate multi-view frames that follow the MSTH distribution. The visual flow-matching loss is L_v eq. 6 and the action flow-matching loss is L_a eq. 7. They are summed as L_v + λ L_a with λ = 0.1 eq. 8.

Stage 2 offline §3.3. The whole model is fine-tuned end-to-end by behaviour cloning, using only L_a. Gradients from the action loss flow back into the world model §3.3.

Online autonomous improvement §3.4 alg. 1. Only LoRA layers are trained while the base model is frozen app. Implementation Details. For each rollout, every transition (o, c_p, a, o') is stored in a replay buffer. Its goal is relabelled to the observation reached at the end of that inference step, whether or not the goal was reached §3.4. When the buffer reaches a set size, a fixed number of updates runs on it, then the buffer is cleared alg. 1 §3.4. Algorithm 1 writes the update as a squared-error regression of the policy output onto the stored action, while the text says the update follows Stage 2 §3.4 alg. 1. See Pitfalls.

Overall pipeline in pseudocode (inference, one control step):

# Act2Goal inference for one control step  [§3.1] [eq. 1-4] [fig. 3]
# Input:  obs, goal (multi-view images); robot_state (proprioception)
# Output: proximal action chunk to execute

def act2goal_step(obs, goal, robot_state, N):
    # Encode current observation and goal into VAE latents  [§3.1] [eq. 1]
    z_t = vae_encode(obs)
    z_g = vae_encode(goal)
    # Denoise imagined future latents with the world-model flow  [eq. 2]
    z = sample_noise(predicted_latent_shape)
    for n in range(N):
        z = z + (1 / N) * world_velocity(z, z_t, z_g)
    # Keep per-layer world-model features as the action conditioning  [§3.1]
    c_w = world_layer_features(z, z_t, z_g)
    # Denoise an action chunk with the action flow  [eq. 3] [eq. 4]
    a = sample_noise(action_shape)
    for n in range(N):
        a = a + (1 / N) * action_velocity(a, c_w, robot_state)
    # Execute only the proximal actions; distal actions stay as guidance  [§3.2]
    return a[:executed_steps]

MSTH distal offsets as pseudocode:

# MSTH distal offsets  [§3.2] [eq. 5]
# Input:  K (imagined length), P (proximal horizon), M (distal count)
# Output: M future time offsets, growing in gap with the horizon

def distal_offsets(K, P, M):
    offsets = []
    for m in range(1, M + 1):
        # Logarithmic spacing from P toward K  [eq. 5]
        offset = P + floor((K - P) / log(M + 1) * log(m + 1))
        offsets.append(offset)
    return offsets

Stage 1 and Stage 2 losses:

# Two-stage offline training  [§3.3] [eq. 6-8]
# Stage 1: joint world-model and action flow matching  [eq. 8]
def stage1_loss(batch, lam):
    # Flow-matching loss on the visual latents  [eq. 6]
    loss_v = flow_loss(video_model, batch.latents, batch.goal_latents)
    # Flow-matching loss on the action chunk  [eq. 7]
    loss_a = flow_loss(action_model, batch.actions, world_features)
    # Weighted sum, lambda = 0.1 in the paper  [§3.3] [eq. 8]
    return loss_v + lam * loss_a

# Stage 2: behaviour cloning, action loss only  [§3.3]
def stage2_loss(batch):
    # Gradients reach the world model through the action path  [§3.3]
    return flow_loss(action_model, batch.actions, world_features)

Online improvement, following Algorithm 1 alg. 1:

# Online autonomous improvement with goal relabelling  [alg. 1] [§3.4]
# Trainable: LoRA parameters phi (rank 64); base model frozen  [app.]
def online_loop(policy, lora, buffer, capacity, updates):
    while not converged:
        # Run one episode and store every transition  [alg. 1, step 1]
        episode = run_episode(policy, goal)
        for (obs, prop, act, obs_next) in episode:
            buffer.append((obs, prop, act, obs_next))
        if len(buffer) >= capacity:
            for k in range(updates):
                batch = buffer.sample()
                # Relabel: the achieved observation becomes the goal  [§3.4]
                goal_new = batch.obs_next
                loss = mse(policy(batch.obs, batch.prop, goal_new), batch.act)
                lora = lora - alpha * grad(loss, lora)
            buffer.clear()

Hyperparameters

Values are as stated in the paper. Anything not listed is "not stated".

Results

Author-reported. Setup: Robotwin 2.0 (four tasks, Easy and Hard modes) and three AgiBot Genie-01 real-robot tasks, compared against DP-GC, HyperGoalNet and π0.5-GC §4.1. Success rates, no online adaptation tab. 1 tab. 2.

MSTH ablation on whiteboard writing, by word length (short ≤ 3 letters, medium 4–6, long ≥ 7) §4.3 tab. 3:

Online adaptation §4.2:

• Robotwin hard-mode scenarios, multi-round online training: success improves over about three rounds before converging; the authors report up to 8× over the pre-trained baseline fig. 6 §4.2. Per-task values are only in the figure.
• Rollout selection: using all rollouts gave the best result; failed-only rollouts still improved the policy fig. 6 §4.2.
• Real-robot drawing of an unseen pattern improved within 15 minutes of online fine-tuning fig. 7 §4.2.
• Plug-In OOD task: success rises from 0.30 (offline, tab. 2) to 0.90 with online training §4.2 §1. The number of rounds is not stated.

Paper vs code

No official code.

The paper links only its project page, https://act2goal.github.io/ §Abstract. The page fetched without body text, so no links could be read from it. A GitHub search for "act2goal" returned two repositories, js5056/Act2Goal.github.io and Act2Goal/Act2Goal.github.io. Both are project-page sites with no description and no license, not method code, so neither was used. An organisation lookup for Act2Goal returned 404, and a combined search for "MSTH" with the method and robot names returned nothing. No repository was cloned.

Pitfalls

• Equation numbers. The source labels equations eq9–eq16, which are stale. This card numbers the eight displayed equations in method order as they render: 1 world-model flow, 2 world-model update, 3 action flow, 4 action update, 5 MSTH offsets, 6 visual loss, 7 action loss, 8 stage-1 loss. Check against the compiled PDF before relying on a number.
• Online loss is ambiguous. The text says the online update "follows the same approach as Stage 2", which is the action flow-matching loss §3.4 §3.3. Algorithm 1 writes a mean-squared regression of the policy output onto the stored action alg. 1. An implementer has to pick one, and the paper does not say which was used.
• Flow-matching details are missing. The interpolation paths φ_t and ψ_t, and the distributions of z_0, z_1, a_0, a_1, are not stated §3.3 eq. 6 eq. 7. Only the target (z_1 − z_0) is given. The inference update in eq. 2 and eq. 4 integrates in the direction that matches this target, but the number of steps N is not stated.
• Log base in MSTH is irrelevant, but the index rule is not. The ratio log(m+1) / log(M+1) in eq. 5 does not depend on the log base. The paper does not say whether m starts at 0 or 1 in the code, how the floor is applied to ties, or what K, P, r and M are.
• Robotwin Hard table contradicts text. The text says Act2Goal has an "overwhelming advantage" in Hard mode §4.1. tab. 1 shows π0.5-GC at 0.42 against Act2Goal's 0.13 on Hard Move Can. The caption's claim about three Hard tasks is consistent with the table.
• Reported rates do not match the stated rollout counts. Real tasks use 40 rollouts and simulation uses 90 app. Testing Metric. Several reported rates are not integer counts over those sizes, for example 0.93 real (37.2 of 40), 0.48 real (19.2 of 40), 0.62 simulation (55.8 of 90) and 0.13 simulation (11.7 of 90). Either episodes were dropped or the rates were rounded or averaged; the paper does not say. Expect small differences if you reproduce.
• Cross-references point to the wrong sections. The text says the four Robotwin tasks are "described in Section 4.2" and cites "Task 1 in Sec. 4.3", but those tasks are described in §4.1 §4.1 §4.2 §4.3. Follow the content, not the references.
• Training data is partly private. Stage 1 and 2 use the public AgiBot World dataset plus "a small proprietary dataset" app. Implementation Details. The proprietary part is not described, so exact reproduction is not possible from the paper.
• Heavy compute. Stage 1 takes 7×24 h and Stage 2 takes 48 h on 16×A800 GPUs app. Implementation Details. The online stage runs on an RTX 4090 with about 5 min per round app. Implementation Details.
• Online scope is LoRA only. Only LoRA layers are updated online and the base model is frozen app. Implementation Details §3.4. Whether the world model is used during online updates is not stated; Algorithm 1 uses only the action policy with a relabelled goal alg. 1.
• Abstract vs body wording. The abstract says "within minutes"; §4.2 says 15 minutes for the drawing task; the appendix gives 5 min per round §4.2 app. Implementation Details.
• Baseline setup is thin. Baseline training budgets, input resolution and hyperparameters are not given. π0.5-GC is described as using a fixed language instruction and goal image §4.1.
• Third-party base model. The world model starts from Genie Envisioner §3.1; its weights and license are outside this card.

Lineage

• Builds on: Genie Envisioner, a video world model whose architecture and weights the paper adapts §3.1; Hindsight Experience Replay, used for goal relabelling §3.4; LoRA, used for online fine-tuning §3.4.
• Compared against: DP-GC, HyperGoalNet and π0.5-GC, all goal-conditioned policies §4.1.
• Superseded by: not stated.