Failure monitors for vision-language-action (VLA) policies are trained offline and then frozen. After deployment, lighting, objects and cameras drift, and the policy itself is updated or moved to new tasks. On recorded real-robot runs, after a switch of task or policy version, a frozen monitor fit on the old task ranks failures little better than chance. We present Streaming Failure Forecasting (SFF), a monitor that keeps learning from the one label every episode provides: whether it succeeded. SFF keeps the offline monitor's network fixed and refits only its last linear layer after each episode, at 2.18 ms per policy call on a Jetson CPU core. Its alarm threshold moves only after successes, which in simulation kept false alarms near target except when two drifts came at once. With every setting locked before testing, SFF ranks failures better than the same monitor kept frozen on 4 of 5 simulated test suites under unannounced drift, and above resetting on detected drift on every suite. Carried with its locked learner settings to 434 recorded runs of four VLAs on real SO-101 arms, replayed across unannounced switches of task or policy version, it ranks failures better than the frozen monitor for every VLA and, in an exploratory comparison, above a full reset at the true switch. A deployed failure monitor need not stay frozen: refitting its readout from outcomes is cheap and helps after the changes we tested.
Figures & tables
Figure 1: A failure monitor that keeps learning after deployment. (a) On recorded runs of four VLAs on real SO-101 arms [ 1 ] , the task or the policy version switches without warning; the frozen monitor, fit on the old task, stays near chance, while SFF learns from each outcome (within-task AUROC, mean of the VLAs, exploratory; each VLA in App. Fig. 13 ). (b) Runs of hero/vla after a switch (ArmnetBench). (c) Only the readout learns, from each run’s outcome.
Figure 2: SFF keeps the offline monitor’s network fixed and refits only its last linear layer after each episode. At every policy call (top), the frozen VLA’s internal features pass through hidden layers trained offline and then fixed, and the current readout θ (blue, the only part that learns) scores the call. The alarm fires once the episode’s peak score exceeds τ . After each episode (bottom), its outcome refits θ in closed form, pulled toward the offline head θˉ , and τ moves only if the episode succeeded. Frame: LIBERO.
Figure 3: Where SFF was tested. (a) A LIBERO-Goal scene under each simulated condition, with the failure rate of SmolVLA’s recorded episodes (all suites in App. B ); worn fingers do not show in the image. (b) A simulated and a real stream. Monitors are never told where a block starts. (c) The real ArmnetBench tasks [ 1 ] and which VLA was fine-tuned for which. Simulated frames are rendered from LIBERO [ 15 ] .
Figure 4: SFF ranks above monitors that react to change; of its parts, the online updates matter most, and the anchor matters on real runs. SFF’s within-task AUROC minus each alternative’s or each variant’s. Left: one mark per simulation suite (square: the discovery suite, not a test; filled: specified in advance; bars: 95% interval where a paired interval was logged). Right: real runs pooled over the four VLAs with 95% intervals (exploratory). Right of 0 favours SFF.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: What was fixed before which data. Each mark is a commit or a first opening of a result.
Evidence
Comparison and criterion
Fixed in advance?
Outcome
LIBERO-10 (primary)
SFF vs. frozen, one-sided Holm p<0.05
yes, before any evaluation run
passed, p=0.026 ; interval reaches 0
LIBERO-Goal, LIBERO-Plus, Spatial new seeds, Spatial our policy
SFF vs. frozen and vs. detect-and-reset
yes; Goal before its data existed
ahead of frozen on all but our fine-tuned policy; ahead of detect-and-reset on all
Real: π0.5 , π0 , MolmoAct 2
the same two, per VLA
yes, committed before each result
ahead of frozen on all three
Real: SmolVLA
the same two
written before its result, committed two hours after it
ahead of frozen; same-feature rerun is exploratory †
Calibration
success-only FAR near α ; other rules not
predicted in the plan
as predicted, except compound drift
Exploratory
early alarms, controls, ρ=1 , real-data extras, readout checks
rule before each output
reported in full (Secs. 6.3 – 6.4 , App. I )
Appendix
Table 3: Evidence map: what was fixed before which result. Commits and dates are in Fig. 5 . Every outcome is reported, including the null and the narrow pass.
Condition
LIBERO-10
LIBERO-Goal
Spatial, new seeds
Spatial, our policy
Spatial (discovery)
nominal
26%
12%
17%
22%
17%
dim light
warm light × 0.5 (40%)
light × 0.25 (27%)
light × 0.25 (42%)
light × 0.12 (37%)
light × 0.25 (38%)
objects
red objects (47%)
dark objects (30%)
red objects (39%)
red objects (31%)
red objects (39%)
camera
camera 3 cm, 3 ∘ (47%)
camera 5 cm, 5 ∘ (37%)
camera 5 cm, 5 ∘ (50%)
camera 2 cm, 2 ∘ (29%)
camera 5 cm, 5 ∘ (48%)
wear
finger friction × 0.1 (37%)
finger friction × 0.03 (38%)
action gain 0.7 (29%)
action gain 0.7 (58%)
action gain 0.7 (28%)
light + objects
warm light × 0.5, red objects (62%)
light × 0.25, dark objects (53%)
light × 0.25, red objects (77%)
light × 0.12, red objects (45%)
light × 0.25, red objects (78%)
Appendix
Table 4: Drift settings per suite and the failure rate of the recorded evaluation episodes under each condition.
LIBERO-Plus family
levels
failure rate
Background Textures
1, 2
17%
Camera Viewpoints
1, 2
55%
Language Instructions
1, 2
81%
Objects Layout
4, 5
34%
Light Conditions
1, 2
14%
Sensor Noise
1, 2
23%
Appendix
Table 5: LIBERO-Plus: the levels used from each perturbation family on the Spatial tasks, and the failure rate of the recorded evaluation episodes.
Suite
features
ρ
a
δ
start
LIBERO-10
state token (layer 4) + action-expert input (mean)
0.999
10
0.001
anchored
LIBERO-Goal
image-language mean (layer 12) + state token (layer 12) + action-expert input (mean)
0.999
10
0.001
anchored
LIBERO-Plus
image-language mean (layer 12) + state token (layer 12)
0.999
100
0.001
anchored
Spatial, new seeds
action-expert input (first action)
0.999
100
0.001
anchored
Spatial, our policy
state token (layer 4) + action-expert input (mean)
0.999
100
0.001
anchored
Spatial (discovery)
action-expert input (mean)
0.999
10
0.001
anchored
Appendix
Table 6: SFF’s locked settings per suite.
Suite
frozen
SFF
warm-up start
SFF-TD
detect-and- reset
novelty
SAFE- MLP
LIBERO-10 (primary)
0.707
0.736
0.743
0.689
0.656
0.663
0.739
LIBERO-Goal
0.698
0.748
0.750
0.697
0.672
0.805
0.729
LIBERO-Plus
0.781
0.802
0.815
0.793
0.759
0.796
0.800
Spatial, new seeds
0.719
0.752
0.760
0.718
0.671
0.756
0.728
Spatial, our policy
0.663
0.656
0.650
0.660
0.632
0.629
0.603
Spatial (discovery)
0.705
0.750
0.735
0.688
0.654
0.762
0.740
Appendix
Table 7: Within-task AUROC of every monitor (episodes 21–60 after each change; higher is better). The warm-up start fits the head on the warm-up episodes with no anchor.
Suite
Comparison
Difference [95% CI]
Holm p
LIBERO-10 (primary)
SFF − frozen
+0.030 [ −0.0004 , +0.063 ]
0.026
SFF − detect-and-reset
+0.080 [ +0.048 , +0.117 ]
0.001
SFF − SFF-TD
+0.048 [ +0.017 , +0.083 ]
0.004
SFF − warm-up start
−0.007 [ −0.021 , +0.010 ]
0.788
SFF − novelty
+0.073 [ +0.023 , +0.131 ]
0.007
LIBERO-Goal
SFF − frozen
+0.050 [ +0.017 , +0.081 ]
0.002
Appendix
Table 8: Every registered comparison of SFF and SFF+N: difference in within-task AUROC (episodes 21–60 after each change) with its 95% interval and one-sided Holm-adjusted p .
Figure 6: Registered differences in within-task AUROC per suite, with 95% intervals. (a) Minus the frozen monitor: SFF (blue), SFF+N (registered for LIBERO-Goal only) and SFF-TD (open). (b) SFF minus detect-and-reset. (c) SFF minus novelty.
Figure 7: SFF minus the frozen monitor per drift condition (within-task AUROC, episodes 21–60 after each change, averaged over that condition’s segments; point estimates). “Wear” is finger friction on LIBERO-10 and LIBERO-Goal and action gain on the Spatial suites (Table 4 ); “back to nominal” is the nominal condition after a drift; the benign drift is left out.
Figure 8: A selected LIBERO-Goal example under worn fingers the camera cannot see. Right: logged scores; SFF alarms at 3.5 s, the frozen monitor never does (peak 0.71, threshold 0.73). Left: re-simulated frames; “replay only” frames come after the replay leaves the log.
SFF against
SFF − other [95% CI]
Other monitors
detect-and-reset
+0.113 [ +0.074 , +0.149 ]
full reset at the true switch (perfect detector)
+0.044 [ +0.018 , +0.068 ]
retraining on all past runs, every 20 runs
+0.031 [ +0.007 , +0.054 ]
SAFE-MLP (SAFE’s probe, our run)
+0.102 [ +0.052 , +0.150 ]
SFF-TD (whole network, streaming TD)
+0.096 [ +0.062 , +0.130 ]
Appendix
Table 9: Recorded real SO-101 runs: SFF minus each alternative or variant, pooled over the four VLAs with equal weight (post hoc, descriptive), over all runs after the switch unless stated; intervals as in Table 2 . The warm-up start is that of Sec. 4 . Long streams visit every task once in a row, with no run twice; SmolVLA has only two tasks, so three VLAs.
Group
Monitor
Spatial (discovery)
Spatial, new seeds
SFF and its variants
SFF
0.750
0.752
SFF-TD (whole network, streaming TD)
0.688
0.718
SFF-TD + forecast head + surprise input ∘
0.701
0.763
SFF-TD + forecast head ∘
0.708
0.775
SFF-TD, λ=0 ∘
0.693
0.710
SFF-TD + L2-init ∘
0.685
0.718
Appendix
Table 10: Within-task AUROC (episodes 21–60 after each change) of every monitor run on the two suites where all of them ran. ∘ Never part of a comparison specified in advance. ∗ Uses more than a streaming monitor may: stored data or the true change points.
Figure 9: After a switch, SFF starts level with the frozen monitor and pulls ahead as outcomes arrive. (a) A failing and a successful run of hero/vla in the hero/n runs after an unannounced switch to hero/task (ArmnetBench front camera), picked by a rule fixed before looking: the segment whose gain is closest to the median over all hero/cands post-switch segments of hero/vlas (hero/gain against hero/median), then the failure and the success with the median SFF score. Over these runs SFF ranks the failing run higher in hero/sff% of failure–success pairs, the frozen monitor in hero/frozen%. (b) Within-task AUROC by blocks of 10 runs after a switch, mean of the four VLAs (exploratory).
Figure 10: False alarms stay near target, and SFF needs about 2 ms per call in constant memory. (a) False-alarm rate on successful episodes per suite (target 0.1): SFF as deployed, the frozen monitor, SFF under two drifts at once, and SFF with its threshold moved after every episode. (b) Time per policy call (median, bar to the 99th percentile) and monitor state on one CPU core of a Jetson Orin Nano, replaying a development stream.
Figure 11: No monitor is best on every suite. Every reported monitor per simulation suite, with the SFF − frozen difference specified in advance, its 95% interval and Holm p .
Figure 12: SFF is above the frozen monitor for every VLA. Real robot runs per VLA: SFF, the frozen monitor and detect-and-reset after an unannounced switch, with 95% intervals. Detect-and-reset’s test never fired for three VLAs, so it equals the frozen monitor there.
Figure 13: After a real switch, SFF starts level with the frozen monitor and pulls ahead as outcomes arrive (exploratory). (a) Within-task AUROC by blocks of 10 runs after a switch, mean of the four VLAs. (b) SFF − frozen per VLA in runs 1–10 and 21–30, with 95% intervals.
Vision Language Action models combine perception, language grounding, and control in a single policy, but their failures are hard to diagnose once visual conditions shift. We test whether OpenVLA feedforward activations contain linearly decodable information about near term task failure in LIBERO manipulation rollouts. The policy is fixed throughout. We log internal activations during execution and fit lightweight monitors after the rollouts are collected. Occlusion is the main controlled stress test. It reduces OpenVLA success from 57% to 17% over 100 episodes per condition. Under this shift, a logistic probe at layer 16 reaches AUROC 0.972 and AUPRC 0.352 for predicting failure within a 15 step horizon. It outperforms both a mean difference direction and an action disagreement baseline. A sparse layer sweep finds uneven decodability across depth: layer 16 is strongest among the tested layers, layer 8 remains informative, and layer 10 is weaker. To check whether the monitor is just an occlusion detector, we also evaluate color shift and camera jitter without refitting. Color shift produces no failures in this setting, so it is a benign control rather than a failure benchmark. Camera jitter does induce failures, and the occlusion trained monitor remains above random. The result is deliberately limited: OpenVLA internal states contain failure relevant structure under controlled perceptual shift, but these experiments do not establish a causal mechanism, task held out generalization, or a deployable recovery system.
Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. VLA-Tug enables real-time monitoring where the well-timed and informative probe supports lightweight steering that improves task success. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs at https://vla-tug.github.io.
Michal P. Podolinsky, Yunhao Yang, Daniel Little +6
University of Texas at Austin · Tufts University · University of Texas at Dallas
Vision-Language-Action (VLA) models can perform diverse manipulation tasks directly from visual observations and natural language instructions, but they remain prone to mistakes and unpredictable failures. Deploying these policies without continuous human supervision will require robust runtime monitors that detect when the policy has gone off-course. Existing monitors use external foundation models, such as vision-language models (VLMs) to supervise the policy, introducing significant complexity and computational cost for monitoring at runtime. In this work, we investigate an alternate approach: can a VLA's own internal activations provide the information needed to monitor its execution? Our method trains linear probes on policy activations from demonstrations to estimate progress, then flags executions that deviate from expected progress. In experiments with the open-source VLA π0.5 and across simulated and real-world manipulation tasks, this approach is competitive with or outperforms VLM-based monitors in task failure detection, as well as methods which require failure data for training. These findings suggest that simple probes of VLA internals can be used to develop reliable runtime monitors, opening a path toward building policies that can act reliably and monitor their execution without external foundation models.