VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models

In plain words

Robot policies that are built from large vision-and-language models can suddenly do something odd when the scene looks unfamiliar, and a failure usually shows up only after the robot has already made a mess. This paper asks how to notice that early, in real time, without first collecting examples of failures.

The authors combine two cheap signals. The first asks whether the model's internal picture of the scene looks like the training data, measured with a standard statistical distance. The second asks whether the robot's own plan keeps changing from one moment to the next: the robot re-plans every few steps, and if the new plan does not agree with the part of the old plan it has not yet carried out, something is probably wrong. Either signal can raise an alarm. The paper also proposes a way to score detectors that counts both how many failures are caught and how early.

TL;DR

• Finetuned VLA policies fail unpredictably out of distribution. VLA-FAIL is a runtime detector that needs no failure rollouts and no extra model §1.
• It combines LLMD (token-wise last-layer Mahalanobis distance under a fixed prior noise sample) and ACC (velocity-normalized consistency between overlapping action chunks) with an OR of two thresholds §3.3.
• Author-reported: comparable to or better than action-sampling baselines on most tasks, with about 1–2% latency overhead on an RTX 5090 §5 tab. 3.
• A new metric, AUCPDT, integrates precision against penalized detection time without fixing a threshold §4.1.

Problem

A supervised detector needs failure rollouts, which are costly and can be unsafe to collect. Novelty detection alone is insufficient, since out-of-distribution states do not always cause failures and some failures come from internally inconsistent action generation with no visual anomaly §1. Prior detectors for generative policies either sample many actions per step (for example, entropy or diffusion-loss estimates with 32 samples in the authors' runs), rely on VLMs or video models, or need an extra trained model §2 §4. The authors state that their target is minimal per-step overhead, since real-time VLA inference is already expensive §1.

Method

VLA-FAIL is one detector with two scores. Each score is computed per environment step, and the alarm fires when either exceeds its threshold §3.3.

Assumed model structure §3.1. The VLA is split into a feature extractor f that maps an observation to H last-layer feature tokens of size F, and a final linear layer that maps each token to an action. Action chunks have length H, and only the first R actions are executed before the policy is queried again (receding-horizon control) §3.

LLMD, last-layer token-wise Mahalanobis distance §3.1.

1. Fix one prior noise sample a0* drawn from a standard normal, once, and use it for all observations. Evaluate the features at flow time t = 0 with this noise, giving f*(o) = f(o, t = 0, a0*). This removes feature variation caused by the stochastic action sampling. One forward pass is needed, and it can run in parallel with action sampling §3.1.
2. After finetuning, compute for each token position s a mean and a centered covariance over the finetuning dataset, using the same features f*(o_i). This is a single pass with no gradients §3.1.
3. Regularize each covariance with lambda times the identity, lambda much smaller than 1 §3.1, set to 1e-6 in the experiments app. F.
4. At run time, compute the squared Mahalanobis distance per token and take the maximum over the H tokens. The maximum means one unlikely token is enough to trigger the score. No temporal smoothing is applied §3.1.

ACC, action chunk consistency §3.2.

1. Under receding-horizon control, the unexecuted suffix of the previous chunk (K = H minus R actions) overlaps the prefix of the new chunk §3.2 fig. 2.
2. For each action dimension d, compute the velocity as the range of the new chunk's overlapping actions in that dimension, floored at v_min §3.2.
3. Compute the mean absolute difference between the two overlapping sequences, divided by the velocity in each dimension, averaged over the K steps and D dimensions §3.2.
4. Smooth with an exponential moving average. The paper writes the update with weight alpha on the previous smoothed score §3.2.
5. In practice D = 3, using the predicted absolute end-effector position. Only one action sample per step is needed §3.2.

VLA-FAIL alarm §3.3. Thresholds tau_ACC and tau_LLMD are set per detector on calibration data, using a time-constant conformal band. The alarm is ACC >= tau_ACC or LLMD >= tau_LLMD. The authors do not use time-dependent thresholds because episode lengths vary, for example in the Drawer task §3.3 app. D.

# LLMD score for one step  [§3.1]
# Input:  feats (H, F) from the fixed prior-noise pass f*(o)
# Stats:  mean (H, F), precision (H, F, F), fitted once after finetuning
# Output: scalar, higher means more likely failure

def llmd_score(feats, mean, precision):
    # Squared Mahalanobis distance for each token position  [§3.1]
    deviations = feats - mean                             # (H, F)
    distances = per_token_quadratic(deviations, precision)  # (H,)
    # Max over tokens: one unlikely token triggers the score  [§3.1]
    return max(distances)

# Token-wise statistics, computed once after finetuning  [§3.1]
# Input:  train_feats (N, H, F), features f*(o_i) over the finetuning data
# Output: mean (H, F), precision (H, F, F)

def fit_llmd_statistics(train_feats, lam):
    # Token-wise mean over the dataset  [§3.1]
    mean = mean_over_samples(train_feats)                   # (H, F)
    # Token-wise centered covariance  [§3.1]
    centered = train_feats - mean                           # (N, H, F)
    covariance = covariance_per_token(centered)             # (H, F, F)
    # Regularize only to keep the matrix invertible  [app. F]
    precision = inverse(covariance + lam * identity(F))     # (H, F, F)
    return mean, precision

# ACC score for one step  [§3.2]
# Input:  prev_chunk (H, D), curr_chunk (H, D), with D = 3 EEF positions
#         executed = R, v_min, alpha, prev_smoothed (scalar)
# Output: smoothed scalar, higher means more likely failure

def acc_score(prev_chunk, curr_chunk, executed, v_min, alpha, prev_smoothed):
    # Unexecuted suffix of the previous chunk vs. prefix of the new one  [§3.2]
    overlap_count = len(prev_chunk) - executed        # K = H - R
    overlap_prev = prev_chunk[executed:]              # (K, D)
    overlap_curr = curr_chunk[:overlap_count]         # (K, D)
    # Velocity per dimension: range of the new chunk, floored at v_min  [§3.2]
    spread = max(overlap_curr, axis=0) - min(overlap_curr, axis=0)  # (D,)
    velocity = max(spread, v_min)                     # (D,)
    # Velocity-normalized mean absolute difference  [§3.2]
    errors = abs(overlap_prev - overlap_curr) / velocity  # (K, D)
    raw_score = mean(errors)                          # mean over K and D
    # Exponential moving average over steps  [§3.2]
    return alpha * prev_smoothed + (1 - alpha) * raw_score

# Combined alarm  [§3.3]
# Input:  acc_now, llmd_now (current scores), tau_acc, tau_llmd (thresholds)
# Output: True when the policy should be flagged as failing

def vla_fail_alarm(acc_now, llmd_now, tau_acc, tau_llmd):
    # Either detector crossing its own threshold raises the alarm  [§3.3]
    return acc_now >= tau_acc or llmd_now >= tau_llmd

Hyperparameters

Finetuning, shared across tasks app. F, table "Model-specific finetuning parameters":

Detector settings on the real-world tasks app. F, table "Model-specific hyperparameters ... real world":

Detector settings on Libero-Plus app. F, table "Model-specific hyperparameters ... Libero-Plus":

Per-task rollout settings, real world app. G, tables for pi0.5 and X-VLA:

Other settings:

Results

Author-reported. Setup: six real-world tasks on two VLAs (pi0.5, 3.6B parameters; X-VLA, 0.9B parameters), roughly 80 rollouts per task, threshold-independent metrics (PR and PDT areas, higher PR and lower PDT are better) §4 §5 app. G. Real-world numbers are point estimates; 95% confidence intervals are in tab. 1.

Real-world, threshold-independent tab. 1:

Real-world averages across four tasks (Drawer, Kitchen, Stack T, Mixer), from the RAPT comparison table app. C: FAIL reaches PR 0.94 and PDT 0.19 on X-VLA, and PR 0.77 and PDT 0.31 on pi0.5. The RAPT baseline in that table reaches PR 0.85 and PDT 0.28 (X-VLA) and PR 0.71 and PDT 0.38 (pi0.5). The table has no average over all six tasks.

Libero-Plus, threshold-independent tab. 2 (point estimates; PR higher is better, PDT lower is better):

Headline reading, from the authors' text: FAIL is the only method in the top three on nearly all real and simulated tasks §5. On the Libero-Plus average, FAIL has the lowest PDT for X-VLA and the third-lowest for pi0.5 tab. 2. On the Libero-Plus average for pi0.5, ACC has the best PR (0.75) tab. 2. ACC alone is strong on jerky-motion failures, LLMD alone detects earlier but less accurately §5.

Overhead, NVIDIA RTX 5090, action sampling and feature extraction running in parallel tab. 3:

Preprocessing for LLMD is dominated by feature extraction: 60 min (pi0.5) and 67 min (X-VLA) on Stack T, plus about 3 min for statistics §5.

Threshold-dependent results (conformal band, significance 0.05, 20 calibration rollouts, real world and Libero-Plus) are in app. D. The authors call these less reliable for method comparison than the threshold-independent results.

Figures: fig. 1 shows the pipeline (a) and per-step sampling time of detectors (b). fig. 2 shows ACC on a successful and a failed pi0.5 Blocks episode.

Paper vs code

No official code.

Search performed: the paper's arXiv metadata and LaTeX source contain no code link [raw/src main.tex]. The only link is the project page https://vla-fail.github.io/vla-fail-2026, which the paper's abstract names. Fetching that page showed it to be an academic project-page template with no link to method code. A GitHub repository search for "vla-fail" returned only a website repository under the user vla-fail (JavaScript, one repository) and several unrelated third-party projects. The KIT ALR lab GitHub account showed no repository matching vla, fail or failure. The GitHub API itself returned HTTP 403 to WebFetch, so the search was done through the web interface only.

Pitfalls

• ACC smoothing alpha has two different values in the paper. The method text says strong smoothing alpha = 0.9 "performs well ... and is required" §3.2. The ablation text says "We use a fixed value of alpha = 0.1 across all experiments" app. B, and the caption of the smoothing figure says the same app. B. The hyperparameter tables list alpha = 0.9 for both VLAs and both settings app. F. The hyperparameter tables are the most specific source, but the conflict is unresolved in the text. An implementer should treat alpha as a tunable value and check both settings on calibration data. Also note that the text says lower alpha improves AUCPR but delays detection (AUCPDT) app. B.
• Overlap in the real-world table does not match H minus R for X-VLA. With chunk length H = 30 app. F and R = 10 executed actions, the ACC overlap should be H minus R = 20. The per-task table gives overlap 10 for five X-VLA tasks and 20 only for Cups app. G. For pi0.5 with H = 20, the overlap equals H minus R in every task. Either X-VLA's effective H differs in the real world or the overlap is defined differently. The paper does not say which.
• Chunk length is stated two ways. The finetuning table gives H = 20 (pi0.5) and 30 (X-VLA) app. F. The Libero-Plus detail table gives chunk length 5 (pi0.5) and 20 (X-VLA) app. G. Use the setting in the table for the environment you run.
• LLMD preprocessing time differs between sections. The results text gives 60 min and 67 min on Stack T §5. The baselines appendix gives 70 min for LLMD on the real-world setup app. C. Expect preprocessing in the order of an hour on the authors' hardware.
• The covariance dataset is underspecified. The paper defines statistics over a dataset of observation-action pairs §3.1 but does not say whether it covers every frame of every finetuning demonstration or a subset. It also does not state the seed of the fixed prior noise a0*. The authors evaluate three seeds and report confidence intervals partly because of this noise §4 app. G.
• The EMA start value and reset rule for ACC are not stated. The first step's smoothed score and whether the score resets between episodes are not given.
• The velocity floor matters. v_min is tuned per setting, not per task. Real-world results drop sharply below 0.01, where the velocity scaling amplifies sensor noise on a stationary robot app. B. Simulation works with lower values app. F.
• Two fusion rules. For the threshold-independent AUC, the authors rank-transform LLMD and ACC and take the minimum §4.1. The deployed detector is an OR of two calibrated thresholds §3.3. Do not expect the same numbers from both.
• Thresholds need calibration rollouts. The threshold-dependent results use 20 successful calibration rollouts, which were removed from the test set app. D. A single outlier in calibration can raise the threshold sharply and drop recall, as the authors note app. D. The threshold-independent results use no calibration, but ACC's action-grid bounds were tuned on 20 of the test rollouts. The authors say this slightly favors ACC app. G.
• Episodes are not stopped at success. During evaluation, every episode runs to the horizon even after success, because otherwise episode length would leak the outcome. The authors also note that a ground-truth success signal would not be available in deployment app. G. Plan the deployment loop accordingly.
• PDT is normalized to episode time. Episode time is normalized to [0, 1], and undetected failures count as 1. The AUCPDT integration starts at the failure rate and uses step-wise (right-hand) sums over Pareto-optimal thresholds §4.1 app. A. Implementing it without this detail gives different numbers.
• Baseline latency numbers come from replays. ACE, STAC and Diff are too slow to run live, so they were evaluated on stored observations, not in closed loop §4 app. G. The latency comparison in fig. 1 concerns the sampling cost, not closed-loop performance.
• Camera inputs differ by VLA. The real-world pi0.5 used the gripper and right overhead views only, because the public checkpoint accepts two camera images. X-VLA used three app. G.
• Typo in the baselines table. The LLMD entry for pi0.5 PR on Kitchen is printed as "00.62" app. C. It matches the value 0.62 in tab. 1, so read it that way.
• The paper claims no formal safety guarantee. The authors state that VLA-FAIL is empirical and cannot guarantee safety. It misses failures whose features and actions both look normal, for example a policy that stops with confidence or ignores the instruction §6.
• Only two VLAs and one robot setup are tested. The real-world setup and the Libero-Plus suites are the only evaluation settings. Transfer to other VLAs is untested. The paper says LLMD also applies to diffusion and discrete VLAs but gives no experiments for them §3.1.
• Costs the abstract hides. LLMD needs the finetuning dataset for its preprocessing pass, which is impossible when that data is unavailable §6. ACC needs receding-horizon control with sufficient overlap and does not apply to fully open-loop chunks §6 app. B.

Lineage

• Builds on: last-layer Mahalanobis OOD detection for classifiers, Lee et al. §2 (no arXiv id in the card's bibliography). Last-layer Laplace approximation §2.
• Builds on: the flow-matching action expert and the pi0.5 VLA, arXiv:2504.16054 §4.
• Compared against: ACE, a feature-entropy detector for generative policies, in the FIPER line of work §4 (no arXiv id in the bibliography). STAC, action-distribution divergence between overlapping chunks §4. Diff, a diffusion-loss detector adapted to flow matching, from Diff-dagger §4 (no arXiv id). RAPT, a recurrent proprioception-prediction model app. C.
• Concurrent work: Rewind-IL, arXiv:2604.16683, also uses the discrepancy between overlapping action chunks as a failure signal §2.
• Superseded by: not known from content already in this wiki.