Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models

Atiksh Bhardwaj; Edward W. Duan; Junwon Seo; Prithwish Dan; Wei-Chiu Ma; Andrea Bajscy; Preston Culbertson

cs.ROOct 22, 2026

Progress Probes $P^2$ : Lightweight Runtime Monitoring for Vision-Language-Action Models

Authors: Atiksh Bhardwaj, Edward W. Duan, Junwon Seo, Prithwish Dan, Wei-Chiu Ma, Andrea Bajscy, Preston Culbertson

Organizations: Carnegie Mellon University. · Cornell University.

Abstract

Vision-Language-Action (VLA) models can perform diverse manipulation tasks directly from visual observations and natural language instructions, but they remain prone to mistakes and unpredictable failures. Deploying these policies without continuous human supervision will require robust runtime monitors that detect when the policy has gone off-course. Existing monitors use external foundation models, such as vision-language models (VLMs) to supervise the policy, introducing significant complexity and computational cost for monitoring at runtime. In this work, we investigate an alternate approach: can a VLA's own internal activations provide the information needed to monitor its execution? Our method trains linear probes on policy activations from demonstrations to estimate progress, then flags executions that deviate from expected progress. In experiments with the open-source VLA $π_{0.5}$ and across simulated and real-world manipulation tasks, this approach is competitive with or outperforms VLM-based monitors in task failure detection, as well as methods which require failure data for training. These findings suggest that simple probes of VLA internals can be used to develop reliable runtime monitors, opening a path toward building policies that can act reliably and monitor their execution without external foundation models.

Figures & tables

Fig. 1: Progress Probes ( $P^{2}$ ) decode task progress directly from the VLA’s internal activations. Using only successful demonstrations, we assign each timestep a normalized progress $\zeta_{t}$ and fit a linear probe to predict it from $z_{t}^{i}$ , the activation at layer $i$ . At runtime, successful executions follow the expected progression through the task, whereas failed executions cause the decoded progress to lag or deviate from this nominal trajectory. Thus, the policy’s own representations enable us to construct a lightweight runtime monitor without failure data or an external foundation model.

	TPR@5%FPR by episode fraction $\uparrow$				Det. time $\downarrow$
Detector	25%	50%	75%	100%	(frac. of ep.)
Progress Probe $\bm{(P^{2})}$	0.00	0.19	0.54	0.64	0.60
SAFE-MLP	0.00	0.08	0.41	0.77	0.75
SAFE-LSTM	0.11	0.37	0.59	0.72	0.53
Robometer (progress)	0.01	0.08	0.21	0.33	0.67
Robometer FT (progress)	0.00	0.34	0.62	0.72	0.56

TABLE III: Simulation: Detection Time Results.

Fig. 2: Simulation: Progress Probe vs. VLMs. ( Top ): Progress prediction results using progress probes trained on different representations. ( Bottom ): Progress prediction mean- $R^{2}$ and failure detection results using progress probes trained on different representations.

Method	Block Stack	Knife Put	Stacked Utensil	Candy Bag	Coffee Bag	Coffee Bean	Average AUROC
FAIL-Detect (lopO)	0.40	0.46	0.59	0.67	0.49	0.42	0.54
FIPER (RND-OE)	0.61	0.82	0.74	0.50	0.60	0.52	0.61
FIPER (ACE)	0.61	0.68	0.62	0.50	0.60	0.50	0.57
STAC	0.59	0.57	0.37	0.31	0.57	0.34	0.45
SAFE-MLP	0.60	0.80	0.60	0.72	0.39	0.38	0.73
SAFE-LSTM	0.70	0.70	0.57	0.76	0.50	0.33	0.58

TABLE IV: Hardware: All-task. State-level failure-detection AUROC is reported for each task. The rightmost column reports the average performance across all six tasks.

Method	Block Stack	Knife Put	Stacked Utensil	Candy Bag	Coffee Bag	Coffee Bean	Average AUROC
FAIL-Detect (lopO)	0.51	0.66	0.54	0.55	0.60	0.49	0.56
FIPER (RND-OE)	0.40	0.83	0.76	0.62	0.51	0.56	0.61
FIPER (ACE)	0.61	0.68	0.62	0.50	0.60	0.50	0.59
STAC	0.59	0.57	0.37	0.31	0.57	0.34	0.46
SAFE-MLP	0.60	0.78	0.64	0.75	0.53	0.31	0.60
SAFE-LSTM	0.54	0.64	0.56	0.70	0.51	0.14	0.51

TABLE V: Hardware: Unseen Task. State-level failure detection AUROC is reported for each task. The rightmost column reports the average performance across all six tasks.

Fig. 3: Hardware: Runtime Monitoring. Episode-level and state-level AUROC for failure detection. $P^{2}$ achieves strong discriminability at both levels for runtime monitoring.

Fig. 4: Hardware: Qualitative. Red curves show failed trajectories and $P^{2}$ predictions, while green curves show predictions on successful trajectories for the Knife Put task. $P^{2}$ correctly identifies failures while remaining within the nominal range on successful executions, demonstrating reliable separation between successful and failed behavior.

Failure Detection for VLA Models

See all 16 papers

Decoding Task Progress from VLA Representations
Aug 13, 2026Atiksh Bhardwaj, Edward Weiyi Duan, Prithwish Dan +2Vision-Language-Action Models Reasoning for VLA Models
Cornell University
ProbeAct: Probe-Guided Training-Free Failure Recovery in Vision-Language-Action Models
Jun 8, 2026Fan Zhang, Seongbin Park, Baharan Mirzasoleiman +2VLM Robustness Robot Failure Recovery
University of California Los Angeles United States
Early Warning Signals for OpenVLA Failure under Visual Distribution Shift
Jun 29, 2026Dipesh Tharu Mahato, Rachel RenRobot Failure Detection Distribution Shift
New York University

Aug 13, 2026J/K move · Enter open · S save

Decoding Task Progress from VLA Representations

Atiksh Bhardwaj, Edward Weiyi Duan, Prithwish Dan, Wei-Chiu Ma+1

Cornell University

Vision-Language-Action Models Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-language-action models (VLAs) are moving rapidly towards deployment as general-purpose manipulation policies, but we currently lack basic tools for understanding what these models represent internally or for monitoring them at runtime. Leveraging ideas from mechanistic interpretability, we probe the residual stream of

π_{0.5}

and find that task progress, the normalized time remaining in a trajectory, is linearly readable from the activations. We find that this signal is present in the pretrained PaliGemma backbone prior to training on any robot-specific data. A single linear probe generalizes to unseen tasks and varies under language counterfactuals when trained on multi-prompt data, but does not enable meaningful steering of the policy. These properties make the signal directly useful for instrumenting deployed VLAs. We use the probe as a simple label-free OOD detector, which detects stalled task progress, and find it competitive with state-of-the-art methods. Our results suggest that VLAs have rich, linearly readable internal representations of semantic quantities like task progress, and that learning to read these signals offers a lightweight, interpretable path toward monitoring deployed visuomotor policies.

Abstract

Failure Detection for VLA Models

Explore similar work

Decoding Task Progress from VLA Representations

ProbeAct: Probe-Guided Training-Free Failure Recovery in Vision-Language-Action Models

Early Warning Signals for OpenVLA Failure under Visual Distribution Shift