Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing

Michal P. Podolinsky; Yunhao Yang; Daniel Little; Hong Lu; Rohan Siva; Olin Goudey; Neel P. Bhatt; Matthias Scheutz

cs.RONov 5, 2026

Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing

Authors: Michal P. Podolinsky, Yunhao Yang, Daniel Little, Hong Lu, Rohan Siva, Olin Goudey, Neel P. Bhatt, Matthias Scheutz,

Organizations: University of Texas at Austin · Tufts University · University of Texas at Dallas

Abstract

Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. VLA-Tug enables real-time monitoring where the well-timed and informative probe supports lightweight steering that improves task success. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs at https://vla-tug.github.io.

Figures & tables

Figure 1: We introduce VLA-Tug , a probe on the VLA backbone embeddings that predicts failures and their mode. (a) A recurrent encoder maps the embedding into a latent failure space; a score head and a calibrated trigger raise the alarm, and a mode head then predicts the failure mode. (b) VLA-Tug alarms before the failure, allowing early stopping or intervention by an expert model.

Method	Trigger	FPG	TPG	EE	FM
Failure detectors (no steering)
SAFE ( Gu et al., 2025 )	one-sided conformal	✓	$\times$	$\circ^{\dagger}$	$\times$
ActProbe ( Huang et al., 2026 )	split conformal	✓	$\times$	$\times^{\ddagger}$	$\times$
STAC ( Agia et al., 2024 )	one-sided conformal	✓	$\times$	$\times$	$\times$
FailureSpot ( Ma et al., 2026 )	one-sided conformal	✓	$\times$	$\circ^{\dagger}$	$\times$
Steering methods

Table 1: Comparison with related methods. FPG / TPG: formal calibrated guarantee on the false-positive / true-positive rate. EE: early evaluation, i.e. a per-episode criterion against annotated failure time;

\circ

denotes partial use. FM: predicts the failure mode. Trigger: what decides when steering happens. A dash means n/a.

Panel of Figure 2 — Figure 2: How the embeddings $e_{t}$ of a failed rollout are labelled for the four supervision strategies.

Figure 3: A successful (top) and a failed (bottom) rollout with their timeline , failure time , and failure mode labels.

Dataset	Split	Seen tasks	Unseen tasks
LIBERO	leave-three-out	7	3
LIBERO-X	new-task	175	43
	new-signature	162	56
	new-skill ⋆	159–209	9–59
Real-world	leave-one-out	7	1

Table 2: Task count by dataset and split.

		LIBERO-10 $\pi_{0.5}$		LIBERO-10 GR00T		LIBERO-X task		LIBERO-X signature		LIBERO-X skill		Real robot		Cost (LIBERO-X)
Method	Cal.	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Fit (s)	State (MB)	GPU (ms)	CPU (ms)
SAFE-LSTM	FC	75.5	34.9	74.1	51.2	64.6	40.4	64.7	41.2	62.5	30.1	44.5	25.1	43.5	26.2	0.066	0.86
SAFE-MLP	FC	74.1	39.9	62.9	39.2	59.3	42.3	53.7	46.2	56.8	23.5	40.1	20.2	114.5	6.3	0.034	0.032
STAC	MC	33.8	36.4	24.2	17.5	21.0	28.0	25.6	28.7	24.6	21.5	–	–	0.0	0.0	21.30	–
ActProbe	MC	39.3	26.7	32.4	30.1	12.6	22.7	11.9	13.9	20.3	13.6	–	–	7.1	0.09	0.15	0.11
FailureSpot	FC	61.9	36.2	36.9	44.7	26.3	20.4	28.0	26.2	30.4	19.6	–	–	9.2	6.4	0.033	0.029

Table 3: Early failure detection on seen and unseen tasks: area under the TPR early vs. FPR curve, averaged over splits. Seen: held-out episodes of training tasks; unseen: held-out tasks. Cal.: calibrator (MC max-conformal, FC functional conformal). Cost on LIBERO-X (lower is better): fit time, memory to score a new episode, and per-step latency on a GPU and a CPU. Dashes: not available. Best in bold, second best in italics.

Panel of Figure 4 — Figure 4: Task- and episode-level data splits. Episodes from seen tasks are divided into training (65%), validation (15%), calibration (10%), and held-out seen (10%) sets. Unseen tasks form a separate evaluation set. Task-level proportions are schematic.

Figure 5: Early detection on seen (top) and unseen (bottom) tasks: the share of failures detected at or before the failure time ( $\mathrm{TPR}_{\mathrm{early}}$ ) against the false-positive rate, for VLA-Tug and the learned and action-based baselines. Each curve is the mean over the splits of a setting, and the area under it up to $40\%$ FPR is the metric of Table 3 . Panel titles give the benchmark, the policy, and the held-out split.

Panel of Figure 6 — Figure 6: Steering demonstrations showing failure scores and corrections for simulated and real-world tasks. When the probe’s predicted failure score rises above the threshold, the VLA policy is automatically paused. For the simulated bowl task, we find that nudging the end-effector toward the drop-off location can help lead to success. For the real-world mushroom task, we find that nudging the wrist camera towards the target grasp object can help. Future work could systematically explore steering strategies. Refer to Figure 17(a) for more steering strategies and Figure 17(b) for another real world steering example.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 4 — Table 4: The eight real-world tasks and the number of demonstrations used to fine-tune the policy ( 510 in total).

	$\pi_{0.5}$ LIBERO-10-Occ		GR00T LIBERO-10-Occ		GR00T LIBERO-X		GR00T real robot
Mode	% of failures	Time (s)	% of failures	Time (s)	% of failures	Time (s)	% of failures	Time (s)
stuck_or_no_progress	24.5	$10.3\pm 3.5$	30.2	$8.6\pm 3.7$	14.1	$15.5\pm 10.7$	12.2	$3.8\pm 3.0$
object_displacement	18.2	$11.9\pm 5.3$	17.8	$11.8\pm 6.7$	10.8	$15.1\pm 10.9$	9.4	$4.9\pm 2.3$
placement_or_insertion	13.6	$15.2\pm 4.2$	16.0	$12.9\pm 7.0$	9.9	$21.9\pm 10.2$	11.4	$8.3\pm 4.5$
grasp_failure	17.3	$9.6\pm 5.4$	13.0	$8.6\pm 4.3$	9.0	$12.0\pm 9.7$	56.7	$4.7\pm 2.3$
open_close_failure	7.3	$12.6\pm 4.8$	9.5	$13.9\pm 3.1$	4.7	$18.7\pm 11.5$	7.5	$3.8\pm 1.7$

Appendix

Table 5: Failure-mode distribution and mean

\pm

std failure time (s) per mode, by policy and dataset.

LIBERO-10

\pi_{0.5}

LIBERO-10 GR00T

LIBERO-X task

LIBERO-X signature

LIBERO-X skill

Real robot

Cost (LIBERO-X)

Method

Cal.

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Fit (s)

State (MB)

GPU (ms)

CPU (ms)

SAFE-LSTM

75.5

\pm

13.4

34.9

\pm

3.5

74.1

\pm

13.0

51.2

\pm

8.4

64.6

\pm

3.5

40.4

\pm

2.3

64.7

\pm

7.4

41.2

\pm

7.1

62.5

\pm

2.4

30.1

\pm

8.5

44.5

\pm

13.5

25.1

\pm

18.8

43.5

26.2

0.066

0.86

SAFE-MLP

74.1

\pm

2.0

39.9

\pm

15.1

62.9

\pm

18.2

39.2

\pm

9.4

59.3

\pm

2.0

42.3

\pm

2.9

53.7

\pm

2.4

46.2

\pm

2.3

56.8

\pm

6.0

23.5

\pm

12.4

40.1

\pm

15.9

20.2

\pm

16.6

114.5

6.3

0.034

0.032

STAC

33.8

\pm

5.1

36.4

\pm

5.3

24.2

\pm

7.5

17.5

\pm

9.8

21.0

\pm

2.3

28.0

\pm

0.1

25.6

\pm

3.7

28.7

\pm

0.3

24.6

\pm

5.0

21.5

\pm

8.0

–

0.0

21.30

–

ActProbe

39.3

\pm

11.3

26.7

\pm

2.6

32.4

\pm

16.8

30.1

\pm

9.3

12.6

\pm

4.2

22.7

\pm

4.6

11.9

\pm

3.1

13.9

\pm

1.5

20.3

\pm

9.4

13.6

\pm

7.1

–

7.1

0.09

0.15

0.11

FailureSpot

61.9

\pm

3.0

36.2

\pm

21.0

36.9

\pm

19.8

44.7

\pm

18.6

26.3

\pm

2.1

20.4

\pm

1.2

28.0

\pm

2.5

26.2

\pm

5.6

30.4

\pm

3.4

19.6

\pm

7.9

–

9.2

6.4

0.033

0.029

Appendix

Table 6: Table 3 with the standard deviation over the splits: area under TPR early vs. FPR up to 40% (in %). Dashes: the action chunks the method needs were not saved. Best in bold, second best in italics.

Figure 7: $\mathrm{TPR}_{\mathrm{early}}$ against FPR for every setting and method, on seen (top) and unseen (bottom) tasks; mean over the splits of a setting.

Seen tasks

Unseen tasks

FPR target

TPR-early target

PPV bound

Method

Runs

FPR holds

TPR holds

FPR

TPR early

PPV

PPV holds

PPV holds (own

\pi

)

FPR holds

TPR holds

FPR

TPR early

PPV

PPV holds

PPV holds (own

\pi

)

10%

–

LTT

0% (0/14)

–

Max conformal (MC)

100% (14/14)

57%

–

9.3

\pm

4.4

53.4

\pm

12.7

90.1

–

50%

–

15.6

\pm

12.3

35.3

\pm

20.5

80.7

–

Functional conformal (FC)

100% (14/14)

71%

–

9.8

\pm

5.8

33.9

\pm

17.2

90.7

–

36%

–

14.9

\pm

9.4

26.3

\pm

16.2

84.3

–

15%

58.1

LTT

50% (7/14)

100%

5.3

\pm

3.1

39.2

\pm

8.0

92.5

100%

86%

71%

9.8

\pm

11.5

27.1

\pm

14.7

85.8

100%

–

Max conformal (MC)

100% (14/14)

57%

–

14.2

\pm

7.3

64.8

\pm

9.0

88.1

–

14%

–

25.8

\pm

12.3

50.7

\pm

15.7

77.5

–

Appendix

Table 7: Learn-then-Test (LTT), max-conformal (MC), and functional conformal (FC) at the same FPR target, over the 14 LIBERO-X runs (

\delta=0.1

). LTT certifies the TPR-early target of the second column together with the FPR target, and abstains otherwise; its percentages are over the certified runs. PPV bound:

\beta\pi/(\beta\pi+\alpha(1-\pi))

	Top-1 accuracy (%)								Top-3 accuracy (%)
	task		signature		skill		Real robot		task		signature		skill		Real robot
Method	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
Most common	14.3	14.3	14.3	14.3	14.3	14.3	20.0	20.0	42.9	42.9	42.9	42.9	43.9	39.0	60.4	32.6
Random	14.3	14.3	14.3	14.3	14.3	14.3	20.0	20.0	42.9	42.9	42.9	42.9	42.9	42.9	60.0	60.0
VLA-Tug (Ours), own detection (first alarm)
onset_pre	34.2	25.5	35.6	22.6	33.6	20.3	54.3	21.7	65.1	59.9	69.8	56.3	69.4	56.3	88.2	70.4

Appendix

Table 8: Balanced top-1 and top-3 failure-mode accuracy (%), pooled over the splits of each kind. Shaded rows: our probe, by label variant; the oracle rows use the true failure time and are an upper bound. Best in bold, second best in italics, among the other rows.

	LIBERO-10 $\pi_{0.5}$		LIBERO-10 GR00T		LIBERO-X task		LIBERO-X signature		LIBERO-X skill
	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
Heads (LSTM, pre-onset label)
Modes + CLIP (ours)	75.1 $\pm$ 12.5	47.9 $\pm$ 20.9	70.5 $\pm$ 10.4	47.4 $\pm$ 13.5	67.9 $\pm$ 5.0	40.2 $\pm$ 1.2	67.0 $\pm$ 6.6	47.0 $\pm$ 0.6	67.8 $\pm$ 4.8	39.0 $\pm$ 7.0
No head	82.3 $\pm$ 6.4 (+10%)	51.1 $\pm$ 4.0 (+7%)	70.1 $\pm$ 11.0 ( $-$ 1%)	55.1 $\pm$ 5.5 (+16%)	67.1 $\pm$ 1.8 ( $-$ 1%)	42.7 $\pm$ 1.5 (+6%)	68.3 $\pm$ 5.7 (+2%)	45.0 $\pm$ 1.3 ( $-$ 4%)	69.2 $\pm$ 4.4 (+2%)	43.6 $\pm$ 11.5 (+12%)
Modes only	80.4 $\pm$ 8.4 (+7%)	48.3 $\pm$ 8.8 (+1%)	70.9 $\pm$ 8.2 (+0%)	52.9 $\pm$ 8.4 (+12%)	69.4 $\pm$ 1.0 (+2%)	39.9 $\pm$ 3.8 ( $-$ 1%)	68.0 $\pm$ 8.8 (+2%)	46.6 $\pm$ 3.5 ( $-$ 1%)	68.3 $\pm$ 3.5 (+1%)	43.7 $\pm$ 10.2 (+12%)
CLIP only	78.2 $\pm$ 11.4 (+4%)	58.3 $\pm$ 15.3 (+22%)	72.6 $\pm$ 4.8 (+3%)	44.4 $\pm$ 12.7 ( $-$ 6%)	66.1 $\pm$ 3.1 ( $-$ 3%)	41.4 $\pm$ 2.4 (+3%)	67.3 $\pm$ 8.3 (+1%)	46.2 $\pm$ 2.8 ( $-$ 2%)	67.8 $\pm$ 5.5 (+0%)	38.5 $\pm$ 9.2 ( $-$ 1%)

Appendix

Table 9: Ablations of our probe in the simulated settings: area under TPR early vs. FPR up to 40% (in %), mean

\pm

standard deviation over the splits, and in brackets the change relative to our probe (shaded). Best in bold, second best in italics, within each block.

Top-1 accuracy (%)

Top-3 accuracy (%)

task

signature

skill

Real robot

task

signature

skill

Real robot

Method

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

Seen

Unseen

At the first alarm of our probe

Score + modes + CLIP (ours)

41.0

\pm

1.4

27.2

\pm

2.5

37.3

\pm

7.2

23.7

\pm

0.7

42.3

\pm

5.3

32.2

\pm

4.3

57.9

\pm

10.6

61.5

\pm

44.0

76.0

\pm

2.6

65.9

\pm

5.7

74.2

\pm

1.0

62.8

\pm

1.8

74.6

\pm

3.4

63.2

\pm

5.6

93.3

\pm

7.3

100.0

\pm

0.0

Score + modes

40.6

\pm

3.7

25.8

\pm

2.5

44.0

\pm

4.6

25.6

\pm

0.8

42.5

\pm

5.3

28.0

\pm

7.0

57.4

\pm

11.8

64.1

\pm

45.4

76.4

\pm

1.5

64.0

\pm

2.8

75.6

\pm

2.8

63.3

\pm

1.0

75.7

\pm

2.9

61.9

\pm

7.9

91.9

\pm

7.0

100.0

\pm

0.0

Modes + CLIP

37.3

\pm

5.8

23.7

\pm

1.4

29.3

\pm

11.1

17.7

\pm

9.3

44.0

\pm

6.4

30.5

\pm

5.3

55.0

\pm

14.2

59.0

\pm

42.8

72.3

\pm

7.7

64.9

\pm

3.3

62.6

\pm

16.4

54.3

\pm

13.4

75.3

\pm

5.2

61.0

\pm

5.6

90.4

\pm

7.9

100.0

\pm

0.0

Appendix

Table 10: What the CLIP head and the score term add to failure-mode prediction: top-1 and top-3 accuracy (in %), mean

\pm

standard deviation over the splits. Our probe is shaded. Best in bold, second best in italics, per column and anchor.

	LIBERO-10 $\pi_{0.5}$			LIBERO-10 GR00T			LIBERO-X task			LIBERO-X signature			LIBERO-X skill			Real robot
Method	0 s	+1 s	+2 s	0 s	+1 s	+2 s	0 s	+1 s	+2 s	0 s	+1 s	+2 s	0 s	+1 s	+2 s	0 s	+1 s	+2 s
SAFE-LSTM	34.9	37.2	39.2	51.2	53.1	55.0	40.4	42.1	44.4	41.2	42.8	44.9	30.1	31.7	33.6	25.1	26.8	28.9
SAFE-MLP	39.9	42.1	43.9	39.2	40.6	43.9	42.3	43.3	44.7	46.2	47.2	47.9	23.5	24.3	25.7	20.2	21.4	22.8
STAC	36.4	43.0	53.2	17.5	22.0	27.6	28.0	29.5	32.2	28.7	30.6	34.0	21.5	22.6	25.1	–	–	–
ActProbe	26.6	31.0	37.2	30.1	33.3	38.6	22.7	24.7	27.5	13.9	16.1	18.8	13.6	14.3	15.7	–	–	–
FailureSpot	36.2	39.4	42.7	44.7	46.5	46.6	20.4	20.9	21.7	26.2	27.2	29.1	19.6	20.8	22.8	–	–	–

Appendix

Table 11: Early-detection area (in %) on the unseen episodes when the failure time is moved later by 0, 1, and 2 s; nothing is retrained. Ours is shaded. Best in bold, second best in italics.

Figure 8: VLA backbone embedding, by outcome.

Figure 9: VLA backbone embedding, by failure mode.

Figure 10: Probe LSTM state, by outcome.

Figure 11: Probe LSTM state, by failure mode.

Figure 12: Failure score head layer, by outcome.

Figure 13: Failure score head layer, by failure mode.

Figure 14: Failure mode head layer, by failure mode (failures only).

Figure 15: Seen tasks (training episodes). A success (green, “pick up the bowl on the plate and place it to the left of the bowl drainer”) and an object displacement failure (red, “place the striped book in the top region of the shelf”).

Figure 16: Unseen tasks. A success (green, “place the red bowl in the basket”) and a press failure (red, “turn on the flat stove”).

Panel of Figure 17 — Figure 17: Steering strategies, in simulation and on the real robot.

Failure Detection for VLA Models

See all 19 papers

VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models
Jun 19, 2026Florian Seligmann, Emiliyan Gospodinov, Enes Ulas Dincer +1Failure Detection for VLA Models Mahalanobis Distance
Autonomous Learning Robots Lab, Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik, Karlsruhe
VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Models
Sep 18, 2026Kaiwen Zhu, Dongfang Liu, Liangkai LiuFailure Detection for VLA Models Robustness of VLA Models
Department of Computer Science, Texas Tech University, Lubbock, TX, USA · Purdue University, West Lafayette, IN, USA
Progress Probes $P^2$ : Lightweight Runtime Monitoring for Vision-Language-Action Models
Oct 22, 2026Atiksh Bhardwaj, Edward W. Duan, Junwon Seo +4Failure Detection for VLA Models Vision-Language-Action Models
Carnegie Mellon University. · Cornell University.

Jun 19, 2026J/K move · Enter open · S save

VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models

Florian Seligmann, Emiliyan Gospodinov, Enes Ulas Dincer, Gerhard Neumann

Autonomous Learning Robots Lab, Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik, Karlsruhe

Failure Detection for VLA Models Mahalanobis Distance Vision-Language-Action Models OOD Detection Robustness of VLA Models

Open paper AskPDF arXiv

Vision-language-action models (VLAs) achieve state-of-the-art performance on many robotic manipulation tasks, yet they can still behave unpredictably in out-of-distribution scenarios. Runtime failure detection is therefore essential for the safe real-world deployment of VLAs.However, existing task failure detectors require computationally expensive action sampling, are based on architectural assumptions that limit their applicability to VLAs, or need access to failure rollouts. We propose VLA-FAIL, a lightweight and broadly applicable failure detection framework for VLAs that combines two novel failure detectors with minimal overhead, without requiring failure data. The first, last-layer Mahalanobis distance (LLMD), detects out-of-distribution states by measuring token-wise deviations in last-layer features relative to the training data. The second, action chunk consistency (ACC), exploits the temporal overlap induced by receding-horizon control and detects failures when consecutive action chunks become inconsistent. To capture the trade-off between detection accuracy and detection latency, we introduce AUCPDT, a threshold-independent metric that jointly evaluates precision, recall, and detection time. Through extensive real-world and simulation experiments, we demonstrate that LLMD and ACC capture complementary failure modes whose combination enables reliable and early failure detection across diverse tasks, frequently outperforming significantly more expensive baseline methods. Project page: https://vla-fail.github.io/vla-fail-2026

Abstract

Failure Detection for VLA Models

Explore similar work

VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models

VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Models

Progress Probes P2P^2P2: Lightweight Runtime Monitoring for Vision-Language-Action Models

Progress Probes $P^2$ : Lightweight Runtime Monitoring for Vision-Language-Action Models