Training-Time Action Conditioning for Efficient Real-Time Chunking

In plain words

A robot that acts in real time cannot wait for its model to finish thinking. So the model plans a short block of future moves, called a chunk, while the previous block is still running. The new block has to join the old one smoothly, which means it must start from the moves the robot has already committed to.

The existing way to do this adds work to every step of the model's answer generation, and that extra work makes the robot slower to react. This paper moves the work into training. During training the model is shown the first few moves as already fixed, as if the computer had been late by a few steps, and it learns to fill in the rest. At run time the committed moves are simply given to the model, so nothing extra is computed and the robot software does not change.

The authors report that this matches the old method's success and speed on two real tasks, box building and espresso making, at a lower compute cost, and that it does better in simulation when the delay is long. The change to the model is small: it needs a separate time signal for each future move.

TL;DR

• Training-time RTC simulates inference delay during training and conditions the policy on ground-truth action prefixes, instead of inpainting them at inference §1 §4.
• It adds no inference-time computation and needs no change to the robot runtime, only a per-action-timestep flow timestep in the network §1 §4 fig. 2.
• In simulated Kinetix, it does better than inference-time RTC once the delay reaches 2 steps, and its advantage grows with the delay §5.1 fig. 3.
• On box building and espresso with the pi0.6 VLA, it keeps task success and speed roughly at parity with inference-time RTC, at lower compute §5.2 fig. 5.
• Hard prefix only: it cannot softly use the overlapping actions beyond the prefix, and the training delay distribution must be chosen to match expected latency §6.

Problem

Real-time chunking (RTC) lets a flow-matching policy generate the next action chunk while the current one executes, and keeps the two chunks continuous by conditioning the new chunk on the frozen prefix of previously predicted actions §1 §2. The inference delay d is the number of controller steps between starting a generation and its result being available; the first d actions of the new chunk cannot be executed in time, so they are taken from the previous chunk §3 fig. 1. This is valid when d <= H - s, where H is the prediction horizon and s the execution horizon §3 fig. 1.

Inference-time RTC enforces the prefix with pseudoinverse guidance, which needs a vector-Jacobian product, that is backpropagation, at every denoising step §4. The authors say this overhead raises latency, and that inference-time inpainting handles high inference delays poorly §1 §5.1.

Method

Core idea. Simulate the delay during training. Learn p(A_{t+d:H} | o_t, A_{t:t+d}): the postfix is generated given the observation and the prefix, where both come from the same ground-truth chunk §4. The authors argue that this is cheaper than inference-time inpainting, though less flexible §4 §6.

Three changes to a standard flow-matching policy §4:

1. Per-action flow timestep. Give each action timestep its own flow-matching timestep. In a diffusion-transformer with adaLN-zero conditioning, let the scale, shift and gate differ between tokens. This adds no learnable parameters §4 fig. 2.
2. Clean prefix. Put the ground-truth, noise-free prefix actions into the input and set their flow timesteps to 1. The postfix is left unchanged §4.
3. Postfix-only loss. Compute the loss only on the outputs for the postfix §4 alg. 1.

Sampling delay. The delay is unknown in advance and can vary at run time, so d is sampled at random during training §4. The simulated experiments use delays in {0, 1, 2, 3, 4} with exponentially decreasing weights §5.1. The real-world experiments use delays drawn uniformly between 0 and 10 §5.2.

Inference. The sampler has the same interface as the action generation of inference-time RTC, taking the prefix and the delay and returning the postfix, so it is a drop-in replacement §4. At each denoising step the prefix positions are overwritten with the committed actions and given timestep 1 alg. 1.

Loss and sampler, as listed in the paper (JAX-style) alg. 1; the red lines mark the differences from standard flow matching in the original listing.

# Training loss with a simulated delay  [alg. 1]  [§4]
# Input:  model, observation, action_chunk (B, H, D), max_delay
# Output: scalar loss

def training_loss(model, observation, action_chunk, max_delay):
    # Sample Gaussian noise and one flow timestep per example  [alg. 1]
    noise = sample_standard_normal_like(action_chunk)  # (B, H, D)
    time = sample_uniform_01(batch_size)               # (B,)
    # Sample a simulated inference delay per example  [§4] [alg. 1]
    delay = sample_delay(batch_size, max_delay)        # (B,)
    # The first `delay` steps form the action prefix  [§3] [alg. 1]
    prefix_mask = arange(H)[None, :] < delay[:, None]  # (B, H)
    # Prefix steps get flow timestep 1, the rest keep the sampled time  [§4]
    time = where(prefix_mask, 1.0, time[:, None])      # (B, H)
    # Interpolate between noise and the chunk, per action timestep  [eq. 1]
    noisy_actions = (
        time[:, :, None] * action_chunk
        + (1 - time[:, :, None]) * noise
    )
    # Predict the velocity for every timestep  [§4] [alg. 1]
    predicted_velocity = model(observation, noisy_actions, time)
    # Regression target is chunk minus noise, as in the listing  [alg. 1]
    target = action_chunk - noise
    squared_error = (predicted_velocity - target) ** 2
    # Average the loss over the postfix steps only  [§4] [alg. 1]
    postfix_mask = logical_not(prefix_mask)[:, :, None]
    total = sum(squared_error * postfix_mask)
    return total / (sum(postfix_mask) + 1e-8)

# Sampling with a known action prefix  [alg. 1]  [§4]
# Input:  model, observation, prefix (B, H, D), delay, num_steps
# Output: chunk (B, H, D); the postfix is chunk[:, delay:]

def sample_actions(model, observation, prefix, delay, num_steps):
    # Start from Gaussian noise for the whole chunk  [alg. 1]
    noisy_actions = sample_standard_normal_like(prefix)  # (B, H, D)
    flow_time = 0.0
    step_size = 1.0 / num_steps
    prefix_mask = arange(H)[None, :] < delay           # (B, H)
    for _ in range(num_steps):
        # Overwrite prefix steps with the committed actions  [alg. 1]
        noisy_actions = where(
            prefix_mask[:, :, None], prefix, noisy_actions
        )
        # Prefix steps are clean, so their timestep is 1  [alg. 1]
        time = where(prefix_mask, 1.0, flow_time)
        velocity = model(observation, noisy_actions, time)
        # Euler step from noise towards the data  [§3] [alg. 1]
        noisy_actions = noisy_actions + step_size * velocity
        flow_time = flow_time + step_size
    return noisy_actions

Hyperparameters

Results

Author-reported. Setup: simulated Kinetix (the dynamic benchmark from RTC), H = 8, 4-layer MLP-Mixer, 2048 rollouts per point §5.1; real-world box building and espresso making with pi0.6, compared against a synchronous baseline and inference-time RTC §5.2. The paper states no exact success numbers in the text; the values are only in the figures and were not transcribed here.

Paper vs code

Official code: Physical-Intelligence/real-time-chunking-kinetix at code_commit 9296f31. Its README cites this paper and gives the training-time recipe README.md. The repository is owned by the authors' lab (Physical Intelligence). It contains the simulated experiments only; the real-world experiments are not in the code.

Found by: the abstract and paper have no code link (arXiv page checked), no project page is linked, and a GitHub search for the method and the authors' organisation returned this repository.

Discrepancies and differences:

• Regression target sign. Eq. 2 writes the target as (eps - A) eq. 2, but Algorithm 1 and the code use A - eps alg. 1 model.py. The sampler integrates from noise (τ = 0) to data (τ = 1) alg. 1 model.py, which requires A - eps. Implement the algorithm's sign.
• Delay distribution in sim. The text says "exponentially decreasing weights" §5.1; the code uses weights proportional to exp(-k) for delay k in {0..4} model.py. The base is not stated in the paper. Algorithm 1 uses a uniform draw alg. 1, which is the real-world setting.
• Fine-tuning from epoch 24. The text says "resume training" §5.1. The code loads only the policy weights from the epoch-24 checkpoint and creates a fresh AdamW optimizer with a new 1000-step warmup train_flow.py train_flow.py. Optimizer state is not carried over.
• Figure 3 execution horizon. The paper plots s = max(d, 1) fig. 3. The code sweeps all execution horizons from max(1, d) to 8 - d eval_flow.py, and the plot's setting is one of them.
• Real-world (delay 0 to 10, 8,000 steps, pi0.6, H100 latency): not in the code.

Pitfalls

• Eq. 2 sign. As written, eq. 2 has the opposite sign to Algorithm 1 and the code. Training on the equation's target would give a velocity with the opposite sign to the one the sampler integrates.
• Delay range boundary. The real-world text says "between 0 and 10" §5.2, while the listing draws randint(0, max_delay), which excludes max_delay alg. 1. The value of max_delay for the real robots is not stated, and the 200 ms maximum is the paper's own figure at 50 Hz §5.2.
• Delays outside the training range. A run-time delay larger than the maximum sampled during training is not covered by the paper. The authors say the real-world model supports latencies up to 200 ms §5.2.
• Choosing the delay distribution. The authors say the delay distribution must be chosen carefully to match the expected latency §6; they give no rule for doing so. In sim, higher delays needed less training supervision §5.1.
• Prefix positions after the last step. Algorithm 1 overwrites the prefix positions at the start of each step but applies the last Euler update to them as well, so the returned chunk's first delay entries are not exactly the committed prefix alg. 1. The code's evaluation loop discards those entries and executes the old chunk for the prefix eval_flow.py. The paper does not discuss this.
• Per-token timestep needs per-token conditioning. The method relies on a time signal that differs per action token §4 fig. 2. For architectures without per-token modulation, the paper gives no recipe.
• Hard prefix only. Unlike inference-time RTC, the model cannot use the overlapping actions beyond the prefix with decreasing weights §4 §6.
• Extra training. The gain costs training compute: 8 epochs of fine-tuning in sim §5.1, and 8,000 steps per task on the real robots §5.2. The authors note that a separate checkpoint per delay might do better §5.1.
• Real-world evidence is small. Two tasks, one base model, and results only in a figure. Its error bars are 68% Wilson intervals for success and ±1 SEM for duration fig. 5.
• Code is simulation-only. The code gives the exact simulated optimizer and loss, but nothing for the VLA model.py.

Lineage

• Builds on: Real-time chunking (RTC), arXiv:2506.07339, whose inference-time inpainting this paper replaces §1 §4. The arXiv id comes from the code README README.md. Also builds on the π0.6 VLA base model §5.2. No card for either paper is in this wiki yet.
• Compared against: naive synchronous and asynchronous baselines and inference-time RTC §5 §5.1 §5.2.
• Related work named by the paper: SmolVLA's asynchronous execution, which does not solve the inter-chunk discontinuity §2; A2C2 (correction head) and VLASH (conditions on a single future action), both concurrent work. This paper conditions on a full prefix instead §2.
• Superseded by: not known.