Streaming Failure Forecasting: VLA Failure Monitors That Keep Learning After Deployment
Organizations: Purdue University Fort Wayne
Abstract
Failure monitors for vision-language-action (VLA) policies are trained offline and then frozen. After deployment, lighting, objects and cameras drift, and the policy itself is updated or moved to new tasks. On recorded real-robot runs, after a switch of task or policy version, a frozen monitor fit on the old task ranks failures little better than chance. We present Streaming Failure Forecasting (SFF), a monitor that keeps learning from the one label every episode provides: whether it succeeded. SFF keeps the offline monitor's network fixed and refits only its last linear layer after each episode, at 2.18 ms per policy call on a Jetson CPU core. Its alarm threshold moves only after successes, which in simulation kept false alarms near target except when two drifts came at once. With every setting locked before testing, SFF ranks failures better than the same monitor kept frozen on 4 of 5 simulated test suites under unannounced drift, and above resetting on detected drift on every suite. Carried with its locked learner settings to 434 recorded runs of four VLAs on real SO-101 arms, replayed across unannounced switches of task or policy version, it ranks failures better than the frozen monitor for every VLA and, in an exploratory comparison, above a full reset at the true switch. A deployed failure monitor need not stay frozen: refitting its readout from outcomes is cheap and helps after the changes we tested.
Figures & tables
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Evidence | Comparison and criterion | Fixed in advance? | Outcome |
|---|---|---|---|
| LIBERO-10 (primary) | SFF vs. frozen, one-sided Holm | yes, before any evaluation run | passed, ; interval reaches 0 |
| LIBERO-Goal, LIBERO-Plus, Spatial new seeds, Spatial our policy | SFF vs. frozen and vs. detect-and-reset | yes; Goal before its data existed | ahead of frozen on all but our fine-tuned policy; ahead of detect-and-reset on all |
| Real: , , MolmoAct 2 | the same two, per VLA | yes, committed before each result | ahead of frozen on all three |
| Real: SmolVLA | the same two | written before its result, committed two hours after it | ahead of frozen; same-feature rerun is exploratory † |
| Calibration | success-only FAR near ; other rules not | predicted in the plan | as predicted, except compound drift |
| Exploratory | early alarms, controls, , real-data extras, readout checks | rule before each output | reported in full (Secs. 6.3 – 6.4 , App. I ) |
| Condition | LIBERO-10 | LIBERO-Goal | Spatial, new seeds | Spatial, our policy | Spatial (discovery) |
|---|---|---|---|---|---|
| nominal | 26% | 12% | 17% | 22% | 17% |
| dim light | warm light 0.5 (40%) | light 0.25 (27%) | light 0.25 (42%) | light 0.12 (37%) | light 0.25 (38%) |
| objects | red objects (47%) | dark objects (30%) | red objects (39%) | red objects (31%) | red objects (39%) |
| camera | camera 3 cm, 3 ∘ (47%) | camera 5 cm, 5 ∘ (37%) | camera 5 cm, 5 ∘ (50%) | camera 2 cm, 2 ∘ (29%) | camera 5 cm, 5 ∘ (48%) |
| wear | finger friction 0.1 (37%) | finger friction 0.03 (38%) | action gain 0.7 (29%) | action gain 0.7 (58%) | action gain 0.7 (28%) |
| light + objects | warm light 0.5, red objects (62%) | light 0.25, dark objects (53%) | light 0.25, red objects (77%) | light 0.12, red objects (45%) | light 0.25, red objects (78%) |
| LIBERO-Plus family | levels | failure rate |
|---|---|---|
| Background Textures | 1, 2 | 17% |
| Camera Viewpoints | 1, 2 | 55% |
| Language Instructions | 1, 2 | 81% |
| Objects Layout | 4, 5 | 34% |
| Light Conditions | 1, 2 | 14% |
| Sensor Noise | 1, 2 | 23% |
| Suite | features | start | |||
|---|---|---|---|---|---|
| LIBERO-10 | state token (layer 4) + action-expert input (mean) | 0.999 | 10 | 0.001 | anchored |
| LIBERO-Goal | image-language mean (layer 12) + state token (layer 12) + action-expert input (mean) | 0.999 | 10 | 0.001 | anchored |
| LIBERO-Plus | image-language mean (layer 12) + state token (layer 12) | 0.999 | 100 | 0.001 | anchored |
| Spatial, new seeds | action-expert input (first action) | 0.999 | 100 | 0.001 | anchored |
| Spatial, our policy | state token (layer 4) + action-expert input (mean) | 0.999 | 100 | 0.001 | anchored |
| Spatial (discovery) | action-expert input (mean) | 0.999 | 10 | 0.001 | anchored |
| Suite | frozen | SFF | warm-up start | SFF-TD | detect-and- reset | novelty | SAFE- MLP |
|---|---|---|---|---|---|---|---|
| LIBERO-10 (primary) | 0.707 | 0.736 | 0.743 | 0.689 | 0.656 | 0.663 | 0.739 |
| LIBERO-Goal | 0.698 | 0.748 | 0.750 | 0.697 | 0.672 | 0.805 | 0.729 |
| LIBERO-Plus | 0.781 | 0.802 | 0.815 | 0.793 | 0.759 | 0.796 | 0.800 |
| Spatial, new seeds | 0.719 | 0.752 | 0.760 | 0.718 | 0.671 | 0.756 | 0.728 |
| Spatial, our policy | 0.663 | 0.656 | 0.650 | 0.660 | 0.632 | 0.629 | 0.603 |
| Spatial (discovery) | 0.705 | 0.750 | 0.735 | 0.688 | 0.654 | 0.762 | 0.740 |
| Suite | Comparison | Difference [95% CI] | Holm |
| LIBERO-10 (primary) | SFF frozen | [ , ] | 0.026 |
| SFF detect-and-reset | [ , ] | 0.001 | |
| SFF SFF-TD | [ , ] | 0.004 | |
| SFF warm-up start | [ , ] | 0.788 | |
| SFF novelty | [ , ] | 0.007 | |
| LIBERO-Goal | SFF frozen | [ , ] | 0.002 |
| SFF against | SFF other [95% CI] |
|---|---|
| Other monitors | |
| detect-and-reset | [ , ] |
| full reset at the true switch (perfect detector) | [ , ] |
| retraining on all past runs, every 20 runs | [ , ] |
| SAFE-MLP (SAFE’s probe, our run) | [ , ] |
| SFF-TD (whole network, streaming TD) | [ , ] |
| Group | Monitor | Spatial (discovery) | Spatial, new seeds |
|---|---|---|---|
| SFF and its variants | SFF | 0.750 | 0.752 |
| SFF-TD (whole network, streaming TD) | 0.688 | 0.718 | |
| SFF-TD + forecast head + surprise input ∘ | 0.701 | 0.763 | |
| SFF-TD + forecast head ∘ | 0.708 | 0.775 | |
| SFF-TD, ∘ | 0.693 | 0.710 | |
| SFF-TD + L2-init ∘ | 0.685 | 0.718 |