Auditing Latent-Space Monitors for Autonomous Driving

Nikhil Kamalkumar Advani; Vishwajeet Shivaji Hogale; Saurav Kumar

cs.ROSep 24, 2026

Auditing Latent-Space Monitors for Autonomous Driving

Authors: Nikhil Kamalkumar Advani, Vishwajeet Shivaji Hogale, Saurav Kumar

Organizations: Independent Researcher · Noertheastern University · Bosch North America

Abstract

Runtime failure monitors can use a model's internal representations to anticipate failures. We audit this monitoring strategy across two autonomous-driving tasks: online vectorized map generation with LaneSegNet and end-to-end planning with VAD. We find that frame-level errors are predictable at inference in both tasks. For LaneSegNet, a supervised latent probe reaches Area Under the Receiver Operating Characteristic curve (AUROC) 0.780 for high Chamfer error; to our knowledge, this is the first post-hoc frame-level failure monitor for online vectorized map generation. For VAD, a supervised planning-latent probe reaches AUROC 0.868 for mean-ADE failure. Our audit shows that internal access is not necessary for strong failure prediction. A monitor using only LaneSegNet's prediction outputs reaches AUROC 0.825, while for VAD, ego state, driving command, and the planner's predicted trajectory reach 0.924 on the same mean-ADE endpoint. Adding latent features to either baseline yields no statistically resolved improvement. This observation persists across a broad suite of planning failure endpoints, including endpoints whose labels depend on geometry unavailable to the non-latent baseline. Thus, predicting failure from an internal representation does not establish that the representation provides useful information beyond observable inputs and outputs. We propose an evaluation protocol for testing the incremental value of latent access and release our per-frame failure endpoint labels.

Figures & tables

Information available	Approach	AP $\uparrow$	AUROC $\uparrow$	Coverage	Mean Chamfer (m)		$AP_{ls}^{*}$
					flagged	retained	flagged	retained
All-camera FPN channel means (latent)	PCA-128 + GMM NLL	.545	.636	49.6%	.552	.454	0.305	0.405
Road-focused per-camera image (latent)	PCA-128 + GMM NLL	.611	.679	29.1%	.600	.462	.281	.387
BEV spatial $2{\times}2$ (latent)	PCA-128 + GMM NLL	.666	.729	15.9%	.662	.472	.198	.386
Road-focused image (latent)	Supervised MLP	.736	.753	28.0%	.629	.453	.257	.395
BEV spatial $2{\times}2$ (latent)	Supervised MLP	.746	.773	33.9%	.639	.432	.228	.422

Table 1: Prediction of mean matched-lane Chamfer

>0.5

m for LaneSegNet

AP_{ls}^{*}

denotes a frame-level analogue used only as a secondary descriptive metric and is distinct from the official dataset-level OpenLane-V2

AP_{ls}

. Results are on the validation set.

Figure 1: LaneSegNet monitor hierarchy and incremental-value audit. All monitors predict mean matched-lane Chamfer

>0.5

m. Model selection and operating-point selection use training data only. (a) Failure rate among retained frames as increasingly high-risk frames are rejected. (b) Mean matched-lane Chamfer across risk-score deciles for representative density, supervised-latent, and prediction-output monitors; bands show 95% source-bootstrap intervals. (c) Precision, recall, and alert coverage at frozen training-selected operating points.

Endpoint	Information available	Approach	AUROC	AP
ADE $>2$ m or agent collision	Four-level front-camera FPN (latent)	PCA-128 + GMM NLL	.500	.312
	BEV (latent)	PCA-128 + GMM NLL	.526	.320
	Ego state + command (non-latent)	Supervised MLP	.753	.546
	Predicted trajectory + command (non-latent)	Supervised MLP	.815	.666
	Planning representation (latent )	Supervised MLP	.844	.712
	Ego state + trajectory + command (non-latent)	Supervised MLP	.900	.839

Table 2: VAD monitoring across alternative planning endpoints. Each row specifies the information available to the monitor and the approach used to convert it into a risk score. “Latent + non-latent” concatenates the planning representation with ego state, predicted trajectory, and/or driving command. Results are AUROC and AP on official validation.

Figure 2: VAD monitoring hierarchy and conditional latent-value audit. (a) Average precision for front-image and BEV representation-novelty scores remain near endpoint prevalence (0.314), while a supervised planning representation monitor is strongly predictive. A non-latent monitor using ego state, predicted trajectory, and driving command is stronger still, and appending the planning representation does not improve it. (b) Selective-risk curves. At 80% retained coverage, failure rate falls from 31.37% overall to 20.00% with the planning representation monitor, 17.05% with the non-latent baseline, and 17.61% after appending the planning representation. (c) Paired change in average precision from appending the planning representation to the complete non-latent baseline across endpoints. Error bars denote 95% scene-bootstrap confidence.

Autonomous Driving Safety Evaluation

Autonomous driving safety evaluation studies how to measure, test, and improve the safety of self-driving systems in routine and hazardous conditions.

See all 158 papers

DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving
Aug 11, 2026Zebin Xing, Yupeng Zheng, Qiang Chen +10Memory-Augmented VLMs End-to-End Autonomous Driving
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Chongqing Chang’an Technology Co., Ltd., Chongqing, China
Structured Representation Probing for Risk Anticipation in End-to-End Autonomous Driving
Oct 27, 2026Jiawei Liang, Tianrui Lou, Le Yang +3Autonomous Driving Autonomous Driving Safety Evaluation
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China · School of Mathematics, Sun Yat-sen University, Guangzhou 510275, China
PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning
Aug 2, 2026Volodymyr Havrylov, Faris Janjoš, Andreas Look +2End-to-End Autonomous Driving Autonomous Driving Planning
University of Tübingen · Bosch Center for Artificial Intelligence · Coburg University +1

Aug 11, 2026J/K move · Enter open · S save

DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving

Zebin Xing, Yupeng Zheng, Qiang Chen, Linbo Wang+9

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Chongqing Chang’an Technology Co., Ltd., Chongqing, China

Memory-Augmented VLMs End-to-End Autonomous Driving Vision-Language-Action Models Test-Time Adaptation of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for end-to-end autonomous driving by enabling unified reasoning across perception, language, and planning. However, existing approaches lack mechanisms to exploit past failures or adapt to distribution shifts, causing the model to persistently underperform on similar scenarios where it has previously failed. In this paper, we propose DriveVLA-M0, a retrieval-augmented VLA with failure-aware latent memory. We construct a latent memory pool that stores failure cases along with their structure scene representations and expert trajectory labels, and design a dedicated Retrieve Model that decouples static road structure and dynamic agent interactions to enable structurally grounded retrieval. At inference time, retrieved cases are injected into the model via a lightweight decoupled LoRA-based test-time training (TTT) mechanism, allowing targeted and scenario-specific correction without modifying the backbone. Extensive experiments on NAVSIMv1 and NAVSIMv2 benchmark demonstrate that our approach consistently outperforms prior methods, achieving 94.1 PDMS on Navtest and 47.0 EPDMS on Navhard with only 26.44 ms TTT backward latency overhead. Furthermore, we show that DriveVLA-M0 scales effectively with additional memory, enabling training-free performance gains through memory expansion. The code is available at https://github.com/ZebinX/DriveVLA-M0.

Abstract

Autonomous Driving Safety Evaluation

Explore similar work

DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving

Structured Representation Probing for Risk Anticipation in End-to-End Autonomous Driving

PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning