FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction

In plain words

A robot that follows a language instruction, such as "put the carrot on the plate", can still fail in the middle of a task. It might close its gripper on nothing, or drop an object on the wrong spot, and most such policies never notice.

FPC-VLA adds a second, slower model that acts as a checker. The robot's main model proposes its next move as usual. When the gripper is about to open or close, which is a moment where a mistake is costly, a vision-language model looks at the camera image and answers in plain words whether the move looks right. If it does not, it names a small correction, such as "move left, small", and the robot adjusts its move before acting.

The paper also smooths the robot's moves. Instead of trusting only the latest prediction, it blends the recent predictions, giving more weight to the newest ones and to those that agree with the latest movement. The checker is trained on examples that were generated automatically from existing robot data, so no one has to label failures by hand.

TL;DR

• A VLA policy proposes 7-D actions; a Qwen2.5-VL 7B supervisor checks them at gripper-state changes and answers Yes or No with a directional correction §3.1 §3.5 §4.1.
• The supervisor is fine-tuned with LoRA on failure-correction QA pairs generated from robot data (synthetic) plus annotated data (MuJoCo and teleoperation) §3.4 §4.1.
• A dual-stream fusion averages past predictions: pose by cosine similarity and time decay, gripper by time decay alone §3.3 alg. 1.
• Author-reported: highest average in each comparison table that reports one (SIMPLER, LIBERO, two real robots); ablations attribute gains to the supervisor and the fusion §4.2 §4.3 §4.6.
• No official code was found; the card is written from the paper alone.

Problem

Imitation-trained VLA policies execute predicted actions without checking them and rarely anticipate or recover from failure §1 §2.3. Failure datasets are usually labelled after the fact, which limits early prediction §1. Many policies also run only part of each predicted action chunk, and averaging predictions over several plausible action modes can give an action that fits none of them §1.

Method

Inputs and outputs §3.1. The input is an RGB image I_t and an instruction l at step t. The output is a refined 7-D action a'_t, six pose increments and a gripper state §3.5, eq. 16. The pipeline has three parts: a dataset generator for failure correction §3.4, a VLM supervisor §3.5, and the dual-stream fusion §3.3. The fusion runs on the policy's output before the supervisor sees it §3.5, alg. 2.

VLA model §3.2. The model has three parts.

1. Vision: a DINO encoder and a SigLIP encoder run on the image. Their features are concatenated and linearly projected into the language model's embedding space §3.2, eq. 1.
2. Language: a Llama2 backbone receives a begin-of-sequence token, the vision tokens and the instruction tokens §3.2, eq. 2. Its output cognition token Z conditions the action module §3.2.
3. Action: a Diffusion Transformer predicts the noise that was added to a ground-truth action sequence, conditioned on Z, with AdaLN for the timestep and Z §3.2, eq. 3, eq. 4. At inference it starts from Gaussian noise and denoises iteratively §3.2. The number of denoising steps is not stated.

# Dual-stream action fusion  [alg. 1] [§3.3]
# Input:  past_actions (N, 7); row k is a_t^(k), predicted from O_{t-k}
# Output: fused action (7,)

def fuse_actions(past_actions, alpha, lam, beta):
    # Split each prediction into 6 pose values and 1 gripper value  [§3.3]
    poses = past_actions[:, :6]                  # (N, 6)
    grippers = past_actions[:, 6]                # (N,)
    # Cosine similarity of each pose to the latest pose  [eq. 7]
    sims = cosine(poses, poses[0])               # (N,)
    # Scale the similarity and squash it with a sigmoid  [eq. 8]
    sim_weights = sigmoid(alpha * sims)          # (N,)
    # Exponential decay that favours recent predictions  [eq. 9]
    decays = exp(-lam * arange(N))               # (N,)
    # Pose weight: similarity times decay, regularised by beta  [eq. 10]
    raw = maximum(0, (1 - beta) * sim_weights * decays)
    pose_weights = raw / sum(raw)
    # Weighted average of the poses  [eq. 11]
    fused_pose = pose_weights @ poses            # (6,)
    # Gripper: decay alone, normalised to sum to one  [eq. 12]
    gripper_weights = decays / sum(decays)
    fused_gripper = gripper_weights @ grippers   # scalar
    # Concatenate pose and gripper into the output action  [§3.3]
    return concat(fused_pose, fused_gripper)     # (7,)

Dual-stream fusion §3.3. For each time t, the fusion collects the predictions a_t^(k) for k = 0 to N-1, where a_t^(k) is the policy's prediction for step t made from the observation at t-k §3.3, eq. 5, eq. 6. The pose and gripper are fused separately. The paper's reason is that the gripper often moves independently of the arm, for example when it holds an object still while the arm moves, so a pose-based similarity says nothing about it §3.3.

• Pose: sim^(k) is the cosine similarity between p^(k) and the latest pose p^(0), with a small epsilon in the denominator §3.3, eq. 7. s^(k) = sigmoid(alpha · sim^(k)) eq. 8. d^(k) = exp(-lambda · k) eq. 9. The weight w^(k) is max(0, (1 - beta) · s^(k) · d^(k)), normalised over k eq. 10. The fused pose is the w-weighted sum of the poses eq. 11.
• Gripper: the fused value is the decay-weighted average of past gripper values, with d normalised over k eq. 12. No similarity weight is applied.
• Algorithm 1 gives the same steps in loop form alg. 1.

Failure-correction dataset §3.4. Input data is RLDS-format robot episodes §3.4 §4.1. The dataset is built in these steps.

1. Gripper events. A change timestep c is one where the normalised gripper state moves by more than delta_g; the paper sets delta_g = 0.5 §3.4, eq. 15.
2. Context frames. For each event, the retained frames are c-k through c, where k = 3 in the paper's implementation, clipped to the episode §3.4.
3. Correction target. For a retained frame t, c* is the next change at or after t, and the correction is Delta a = a_{c*} - a_t §3.4.
4. Text. Each component of the correction, x, y, z and yaw only, becomes a direction and a magnitude using two thresholds delta_small and delta_large; a component below delta_small is dropped §3.4. Directions are given in the robot base frame §3.4 tab. 1.
5. Prompt. A prompt has three parts: the task, the answer rules and example answers. tab. 2 shows one example. The answer must start with "Yes" or "No"; a "No" must include at most one movement direction per axis pair and one rotation direction, each with a magnitude of small or large tab. 2.

Synthetic pairs are generated from RLDS robot data; a smaller set is annotated from MuJoCo simulation and real-robot teleoperation. The annotation procedure is not described §3.4 §4.1.

Supervisor §3.5, alg. 2. The supervisor runs only when the gripper state changes by more than delta_g, compared with the previous step's gripper state, which starts at 0 §3.5, eq. 17 alg. 2. It then queries the VLM with the current image and the prompt P_t §3.5, eq. 18. A reply starting with "No" gives a correction: the three translation components and the yaw component are added to the fused action, and the other components are left unchanged [§3.5, eq. 19]. Otherwise the fused action runs unchanged alg. 2.

# Supervisor loop  [alg. 2] [§3.5]
# Input:  fused actions (7,) per step, camera images, VLM, prompt, delta_g
# Output: the actions that are executed

def run_episode(steps, vlm, prompt, delta_g):
    # The gripper state before the first step is taken as 0  [alg. 2]
    prev_gripper = 0.0
    for action, image in steps:
        # Check only when the gripper state changes  [eq. 17] [§3.5]
        if abs(action[6] - prev_gripper) > delta_g:
            # Ask the VLM for a Yes/No answer and a correction  [eq. 18]
            reply = vlm(image, prompt)
            if reply.startswith("No"):
                # Read the translation and yaw directions  [§3.4] [alg. 2]
                shift, yaw = parse_correction(reply)
                # Add the correction; only x, y, z and yaw change  [eq. 19]
                action = action + correction(shift, yaw)
        execute(action)
        # Remember this step's gripper state for the next check  [alg. 2]
        prev_gripper = action[6]

Key equations, as written:

\hat{\mathbf{a}}_t = [\Delta x_t, \Delta y_t, \Delta z_t, \Delta r_{x,t}, \Delta r_{y,t}, \Delta r_{z,t}, g_t]^\top §3.5, eq. 16

\mathbf{a}'_t = \hat{\mathbf{a}}_t + [\Delta x, \Delta y, \Delta z, 0, 0, \Delta r_z, 0] §3.5, eq. 19

Hyperparameters

Values are as stated in the paper. "not stated" means the paper gives no value.

Results

Author-reported. Setups: SIMPLER (WidowX, Google Robot) §4.2; LIBERO (Franka) §4.2; real Xiaomi robot and ALOHA §4.3. Baseline numbers are those in the paper's tables; whether they were re-run or copied is not stated.

SIMPLER, WidowX (average over four tasks) tab. 3. "Fine-tuned" marks in-domain fine-tuning. FPC-VLA is not fine-tuned.

SIMPLER, Google Robot (average) tab. 4. Visual Matching and Variant Aggregation averages.

SpatialVLA and ThinkAct report per-task numbers but no average for these columns, so they are left out of the averages above tab. 4.

LIBERO, Franka (average over four suites, 500 trials each) tab. 5.

FPC-VLA is not the best on Goal or Spatial in this table tab. 5.

Real robots, grasp and task success (50 trials per task) tab. 6.

Ablation, four tasks (WidowX: spoon, eggplant; Xiaomi: stack; ALOHA: cup) tab. 7. Average task success.

Other results. Cross-platform: a supervisor trained only on Google Robot data is evaluated on WidowX; the success rates are shown only in fig. 7, and the text calls them competitive §4.4. Failure-prediction accuracy against DeepSeek-VL-7B, InternVL3-8B, LLaVA-1.5-7B and the base Qwen2.5-VL-7B is shown only in fig. 8; the text says the fine-tuned supervisor is best on all three robots §4.5.

Paper vs code

No official code. Searched: the abstract and the LaTeX source name only the project page https://fpcvla.github.io/, which has no code link (its "Code" button points to #). GitHub repository search for "FPC-VLA", "fpcvla", "failure correction vla supervisor" and "fpc-vla org:Xiaomi" found no code repository. The one match, fpcvla/fpcvla.github.io, holds only the project website (an index.html and figs/, no license file) and is not model or training code. Nothing was cloned.

Pitfalls

• Gripper state is continuous after fusion but binary before it. The fused gripper is a decay-weighted average eq. 12, yet the trigger assumes g_t in {0, 1} §3.5, eq. 16. The paper does not say how a fused value is turned into the state the trigger compares, nor whether the trigger uses the raw or the fused gripper, although Algorithm 2 receives the fused action alg. 2.
• The similarity weighting is nearly flat with the stated alpha. Because sigmoid(0.1 · x) for x between -1 and 1 lies between about 0.475 and 0.525 (our arithmetic from eq. 8 and §4.1), the similarity factor changes the weights by about 10% at most. With lambda = 0.01 the decay is also gentle. The fusion is therefore close to a plain average, and the ablation gains tab. 7 should be checked with these exact constants.
• Fusion cost is not stated. Algorithm 1 calls the VLA on each of the N past observations at every step alg. 1, and the paper does not say whether these calls are cached. Whether the reported 0.176 s per non-keyframe §4.1 includes them is not stated.
• Chunk indexing is implicit. a_t^(k) is the prediction for step t made from O_{t-k} §3.3, which needs the chunk from O_{t-k} to reach k steps ahead. The paper does not say which chunk element is used, nor how many of the 15 predicted steps are executed §4.1. N is not stated.
• The zero-change frame gets a zero target. The retained frames include the change frame c itself §3.4. For that frame c* = c, so Delta a = 0 and every component is "None". The paper does not say what answer such a frame gets.
• The same thresholds appear as label cut-offs and as step sizes. delta_small and delta_large (0.01 and 0.1) define the dataset labels §3.4. Algorithm 2 maps magnitudes to "step sizes delta_small, delta_large" alg. 2. The paper does not say that the executed step equals these numbers, and does not give the units of the action values.
• Only yaw is corrected. Roll and pitch corrections are excluded in the labels §3.4 and in the update eq. 19. The supervisor's answers are parsed by a "No" prefix check alg. 2; the parse format and the handling of malformed replies are not stated.
• "Zero-shot" needs care. The paper uses "zero-shot" for "no fine-tuning on the evaluation setting" §4.2. The VLA is pretrained on OXE and LIBERO §4.1, and the Franka supervisor is trained on LIBERO-derived synthetic data §4.1. The LIBERO results therefore do not come from held-out data. The WidowX supervisor is trained on BridgeV2-derived data §4.1, which the VLA pretraining mixture may also contain; the paper does not say.
• The ablation percentages do not match Table 7. The text gives a drop of 31.3% without the supervisor and 16% with it §4.6. From tab. 7, the task average falls 74.4 to 50.8 without the supervisor (23.6 points, about 31.7% relative) and 82.1 to 66.1 with it (16.0 points, about 19.5% relative). The disturbance distribution (range [0.01, 0.1]) is given without units or distribution §4.6.
• Citation error for SigLIP. The paper cites the DINOv2 reference (oquab2023dinov2) for both DINO and SigLIP §3.2. The SigLIP citation should be checked against the paper's bibliography before it is relied on.
• Missing training details. Optimiser, schedules, epochs, the VLA's initialisation and its size are not stated. The VLA's denoising step count T and noise schedule are also not stated §3.2.
• Baselines. Baseline rows come from the paper's tables; whether they were re-run under the same protocol is not stated.
• Supervisor only sees gripper changes. Failures after a successful grasp, such as slips, are not covered; the paper says so §6.
• Attribution. The arXiv metadata author list (eleven names) omits Chenyang Zhao, who appears in the LaTeX author block [title block]. The list above follows the metadata.

Lineage

• Builds on: the VLA family it cites, including RT-1, RT-2, RT-X, Octo, OpenVLA, SpatialVLA, RoboVLMs and CogACT §2.3; the DiT action head §3.2; Llama2 §3.2; and Qwen2.5-VL as the supervisor's base §4.1. The paper does not name the VLA base checkpoint.
• Related failure work: AHA, RoboReflect, RACER, RoboFAC, EvolvingGrasp and Phoenix are discussed in §2.4. The paper's stated difference is that it anticipates failures before they happen rather than correcting them after §2.4.
• Compared against: RT-1, RT-1-X, RT-2-X, Octo, OpenVLA, RoboVLMs, CogACT, SpatialVLA, ThinkAct, Cot-VLA, GRAPE, TraceVLA, ACT, Diffusion, π0 and π0-Fast §4 tab. 3–7.
• Superseded by: not stated.