Learned robot policies produce actions, but their outputs alone do not establish whether execution is progressing as intended. Robot execution monitoring requires identifying the current execution phase, detecting failures, and recognizing task completion from observations available during execution. Training such monitors requires annotations that are scarce in datasets collected for robot-policy learning. We present RoboMonitor, a label-efficient vision--language execution monitor that learns from these datasets before introducing monitoring supervision. We pre-train on 25 hours of multi-camera trajectories spanning 12 manipulation tasks and two robot embodiments, using action-conditioned future-feature prediction, inverse dynamics, and masked-present prediction. We then transfer the learned visual and context encoders to a causal monitor and apply temporal supervised fine-tuning (Temporal SFT), which combines supervision throughout each observation window with consistency objectives within and across overlapping windows. At deployment, RoboMonitor requires only the task instruction and camera observations. On a four-task monitoring benchmark, RoboMonitor trained with 52 labeled episodes achieves 93.1% mean phase accuracy and 85.9% macro recall over two fine-tuning seeds, exceeding Qwen3-VL and Robometer trained with the same monitoring supervision. Its phase accuracy also exceeds that of both Qwen3-VL and Robometer trained with 100 episodes. A Qwen3-VL ablation shows that Temporal SFT reduces mean spurious phase switching from 15.23% to 4.95%. In closed-loop deployment, the integrated system completes 39 of 40 simulated Toolbox Sorting trials and 35 of 40 real-world Reel Packing trials, with no false recovery triggers observed.
Figures & tables
Fig. 1: Overview of RoboMonitor. Training has two stages: (1) predictive self-supervised pre-training on unlabeled robot trajectories, which uses multi-view observations together with recorded robot state and executed actions to learn execution-relevant visual and temporal representations; and (2) causal supervised fine-tuning (Temporal SFT) on a smaller monitoring-labeled dataset to predict execution phase, failure, and task completion. At deployment, RoboMonitor receives only the task instruction and camera observations, so it can monitor both end-to-end visuomotor policies and task-orchestration systems without access to the underlying controller.
Fig. 2: Predictive robot pre-training (stage 1). (a) Visual and context encoders process recent multi-camera observations; a predictor uses their features, recorded robot state, and commands to estimate future visual features. A fixed copy of the original visual encoder supplies the targets. (b) Masked-present prediction estimates features at masked spatial locations. (c) Inverse dynamics predicts commands from the difference between context summaries. The adapted visual and context encoders are retained for execution monitoring.
Fig. 3: Causal execution monitor (stage 2). The VLM processes the task instruction and a multi-camera observation window, producing a causal readout at each timestep. A separate frozen visual/context branch supplies features from the corresponding recent observation pair. Residual adapters combine these features using Equation 7 . Phase and completion share fusion adapters; failure uses separate adapters. Deployment uses the predictions at the final timestep.
Phase
Completion
Failure
Spurious
SFT episodes
Method
Accuracy
Macro Recall
AP
AP
Switch Rate ↓
52
RoboMonitor-4B (Ours)
93.09 ± 0.29
85.85 ± 0.41
87.60 ± 0.01
45.87 ± 0.68
5.58 ± 0.22
Robometer-4B + Temporal SFT
88.77 ± 1.06
81.78 ± 0.41
85.42 ± 0.31
59.39 ± 3.11
5.27 ± 0.40
Qwen3-VL-4B + Temporal SFT
86.76 ± 2.29
74.23 ± 5.69
74.01 ± 10.85
46.37 ± 6.99
4.95 ± 0.12
Qwen3-VL-4B + Pointwise SFT
85.42 ± 0.58
73.93 ± 0.99
70.14 ± 2.29
30.34 ± 1.16
15.23 ± 7.30
100
RoboMonitor-4B (Ours)
93.63 ± 1.04
88.55 ± 0.73
92.29 ± 1.17
72.01 ± 0.14
3.17 ± 0.04
TABLE I: Offline execution-monitoring performance. All values are percentages; higher is better except spurious-switch rate ( ↓ ). Fine-tuned results are mean ± standard deviation over two SFT seeds. Bold denotes the best mean within each annotation budget. Pointwise SFT is evaluated only at 52 episodes. Gemini uses few-shot examples without parameter updates.
Metric
Reel Packing
Toolbox Sorting
Real
Sim
Trials
40
40
Failure events
18
12
Failures detected
15/18 (83.3%)
11/12 (91.7%)
False recovery triggers
0
0
Recovery success
13/15 (86.7%)
11/11 (100%)
TABLE II: Closed-loop deployment of RoboMonitor. Recovery success is measured over attempted recoveries.
Despite rapid progress, generalist robot policies remain brittle during execution. Reliable deployment therefore requires runtime monitoring that provides timely warnings of failure. Existing monitors commonly estimate risk from current observations or policy signals without explicitly predicting the future consequences of proposed actions. To anticipate such risks before action execution, we introduce ReAP, a future-aware runtime monitoring framework that combines visual failure recognition with action-conditioned failure prediction. Specifically, ReAP first trains a visual detector on successful and failed rollouts to learn a failure-relevant representation space and estimate current risk. An action-conditioned predictor then forecasts the resulting future representation in this space to estimate future failure risk. At runtime, the detector and predictor provide complementary signals for current-state failure recognition and future-aware risk assessment. Experiments across diverse robot policies in simulation and the real world demonstrate that ReAP effectively balances failure-detection accuracy and warning timeliness.
Jingyuan Liu, Yunpeng Qing, Changxu Wei +1
ACE Robotics · Fudan University · Zhejiang University +1
Vision-language models (VLMs) are increasingly explored as visual critics, reward generators, and failure detectors in robotic manipulation. These roles implicitly require models to judge not only final task success, but also how a manipulation execution is physically and temporally progressing. However, existing evaluations fail to test whether VLMs possess fine-grained process understanding. To address this gap, we present RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation. RoboProcessBench decomposes such capability into two complementary dimensions, static monitoring and dynamic reasoning, instantiated as 12 diagnostic question families covering phase, contact, motion, coordination, primitive-local progress, temporal order, outcome, and primitive-level transitions. Built from physically grounded execution traces, the curated benchmark corpus ProcessData contains \textasciitilde 58k question-answer pairs across 260 manipulation tasks, which is further split into ProcessData-SFT and ProcessData-Eval for post-training and evaluation purposes. Extensive evaluation of various VLMs on ProcessData-Eval reveals broad limitations across 12 diagnostic task families, suggesting current models still lack robust process-aware understanding of manipulation executions. But with ProcessData-SFT, the post-trained Qwen2.5-VL-7B and InternVL-3-8B exhibit consistent gains on local state, motion, progress, and primitive-aware cues. These results demonstrate that RoboProcessBench serves as both an evaluation benchmark and a learnable supervision source for developing VLMs capable of monitoring and evaluating robotic manipulation processes. Project webpage: https://processbench-2026.github.io.
Dayu Xia, Yue Shi, Yao Mu +7
Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University +2
Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose Hide-and-Seek, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, π0, and π0.5.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.
Seongheon Park, Wendi Li, Changdae Oh +4
University of Wisconsin–Madison · Georgia Institute of Technology