A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

Kunal Runwal; Swaraj Gajare; Daniel Adejumo; Omkar Ankalkope; Siddhant Baroth; Aliasghar Arab

cs.ROMar 30, 2026

A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

Authors: Kunal Runwal, Swaraj Gajare, Daniel Adejumo, Omkar Ankalkope, Siddhant Baroth, Aliasghar Arab

Organizations: Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA (ASAS-Labs). · Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA (ASAS-Labs). · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.

Abstract

Semantic anomalies-context-dependent hazards that pixel-level detectors cannot reason about-pose a critical safety risk in autonomous driving. We propose a semantic observer layer: a quantized vision-language model (VLM) running at 1--2,Hz alongside the primary AV control loop, monitoring for semantic edge cases, and triggering fail-safe handoffs when detected. Using Nvidia Cosmos-Reason1-7B with NVFP4 quantization and FlashAttention2, we achieve ~500 ms inference a ~50x speedup over the unoptimized FP16 baseline (no quantization, standard PyTorch attention) on the same hardware--satisfying the observer timing budget. We benchmark accuracy, latency, and quantization behavior in static and video conditions, identify NF4 recall collapse (10.6%) as a hard deployment constraint, and a hazard analysis mapping performance metrics to safety goals. The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms.

Figures & tables

Fig. 1 : Qualitative results on the Hazard Perception Test Dataset [ 8 ] using Cosmos-Reason1-7B [ 9 ] . Top row (Samples 11–12): correctly classified normal frames. Bottom row (Samples 13–14): a normal frame and a detected anomaly, demonstrating context-aware semantic reasoning.

Fig. 2 : Semantic observer layer architecture. The VLM observer runs at 1–2 Hz alongside the primary AV control loop, processing temporal windows of RGB frames with a structured prompt. Upon detecting a high-confidence semantic constraint violation, it triggers a fail-safe handoff. Visual tokens from Cosmos-Reason1-7B are projected into the language embedding space and evaluated against context-conditioned semantic constraints to produce a binary {Normal, Anomaly} decision.

Fig. 3 : High-level architecture of Cosmos-Reason1-7B for anomaly detection. Visual features from the vision encoder are projected into the language embedding space and jointly processed with prompt tokens by a decoder-only transformer backbone (see Fig. 4 for block details).

Fig. 4 : Architecture used in Cosmos-Reason1-7B. Visual tokens extracted by the Qwen2.5-VL vision encoder are projected into the language embedding space via a two-layer MLP merger and concatenated with prompt tokens. NVFP4 quantization is applied to the backbone weight matrices, and FlashAttention2 accelerates attention computation. Cosmos-Reason1-7B retains the Qwen2.5-VL architecture and is further fine-tuned on robotics and embodied reasoning data for physical AI tasks.

Fig. 5 : Dataset orchestration for FCDD training. RDD2022 [ 17 ] images (all damage types merged) serve as the anomalous class, while Cityscapes [ 18 ] images filtered for $\geq$ 25% road coverage provide the normal class. An 80/20 train-test split yields 31,386 training samples (2,598 normal, 28,788 anomalous) and 7,643 test samples (447 normal, 7,196 anomalous).

Panel of Figure 6 — Fig. 6 : FCDD training metrics across 5 iterations. (a) Both class errors converge near 0.0 after 100 epochs with maintained separation. (b) Combined error decreases under step learning rate decay ( $10^{-3}\to 2\times 10^{-4}$ at epoch 100).

Panel of Figure 7 — Fig. 7 : FCDD global heatmaps. Red regions indicate high anomaly scores. (a) Clean roads receive uniformly low scores. (b) Damaged sections trigger pronounced activations.

Model	ROC-AUC ∗ $\uparrow$	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
FCDD	1.00	—	—	—
Cosmos (7B, NF4)	—	0.60	0.83	0.47

TABLE I : Paradigm comparison of FCDD and Cosmos-Reason1-7B. The two models solve fundamentally different problems and are evaluated on different metrics by design: FCDD optimizes distributional separation (ROC-AUC) while Cosmos produces actionable semantic classifications (F1/Precision/Recall). Direct metric comparison is intentionally omitted. Note that the models are also evaluated on different test sets: FCDD on a held-out split of RDD2022+Cityscapes (7,643 images); Cosmos on the same RDD2022+Cityscapes split used in Experiment II (2,162 images). *ROC-AUC measures ranking quality; F1/Precision/Recall measure classification at a fixed threshold.

Quantization	Prompt	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Lat.(s) $\downarrow$
INT8	Verbose	84.1%	45.1%	58.7%	1.33
	Pruned	53.9%	12.5%	20.3%	1.37
	Minimal	—	—	0.0%	1.36
NF4	Verbose	82.8%	47.0%	60.0%	0.80
	Pruned	98.0%	5.7%	10.8%	0.85
	Minimal	—	—	0.0%	0.84

TABLE II : Cosmos-Reason1-7B performance across quantization and prompt configurations. Dataset imbalance: 79% anomalous, 21% normal. F1 score is the primary metric due to class imbalance.

	NF4+Verbose	INT8+Verbose	INT8+Pruned
TP	806	773	215
FN	909	942	1500
FP	168	146	184
TN	279	301	263
F1 Score	60.0%	58.7%	20.3%
Precision	82.8%	84.1%	53.9%

TABLE III : Confusion matrices for top-performing Cosmos configurations. Positive class = Anomaly (road damage). NF4+Verbose is the optimal static image configuration: highest F1 (60.0%), best recall (47.0%), and fastest inference, demonstrating that aggressive quantization succeeds only when paired with verbose structured prompts.

Quantization.	TP	TN	FP	FN	Precision $\uparrow$	Recall. $\uparrow$
BF16	51	96	84	15	37.8%	77.3%
INT8	50	99	81	16	38.2%	75.8%
NF4	7	162	18	59	28.0%	10.6%
Quantization	F1 $\uparrow$		Acc. $\uparrow$		Latency (s) $\downarrow$
BF16	50.8%		59.8%		0.485
INT8	50.8%		60.6%		0.787

TABLE IV : Cosmos-Reason1-7B video stream performance across quantizations. Hazard Perception Test Dataset (224 clips, 5s windows, 2s stride). BF16 is the optimal configuration: balances recall (77.3%), F1 (50.8%), and latency (0.485s), while NF4’s speed advantage is negated by catastrophic recall collapse (10.6%).

Hazardous Event	ASIL	Safety Goal
False positive: spurious fail-safe trigger	B	Precision $\geq$ 80%; debounce triggers
False negative: undetected hazard	D	Recall $\geq$ 90%; redundant detection
Excessive latency	B	Latency $\leq$ 1 s; watchdog monitor
NF4 silent recall collapse (video)	D	Prohibit NF4 in video path

TABLE V : HARA for the VLM observer module (ISO 26262).

Anomaly Detection

Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.

See all 654 papers

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
Mar 18, 2026Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal +7VLM Post-Training VLMs for Autonomous Driving
NVIDIA, California, USA
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous Driving LLM-Guided RL
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
Vision-Language Models as copilots for Autonomous UAV Navigation: Analysis of Latency and Reliability in Degraded Environments
Aug 10, 2026Hiago Sodre, Sebastian Barcelona, Vincent Sandin +6Aerial VLN UAV Navigation
Technological University of Uruguay, Uruguay

Mar 18, 2026J/K move · Enter open · S save

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal, Hao Wang+6

NVIDIA, California, USA

VLM Post-Training VLMs for Autonomous Driving Vision-Language Models Autonomous Driving Perception Video Anomaly Detection Traffic Accident Anticipation

Open paper AskPDF arXiv

The rapid growth of ego-centric dashcam footage presents a major challenge for detecting safety-critical events such as collisions and near-collisions, scenarios that are brief, rare, and difficult for generic vision models to capture. While multimodal large language models (MLLMs) demonstrate strong general reasoning ability, they underperform in driving contexts due to domain and temporal misalignment. We introduce VLM-AutoDrive, a modular post-training framework for adapting pretrained Vision-Language Models (VLMs) to high-fidelity anomaly detection. The framework integrates metadata-derived captions, LLM-generated descriptions, visual question answering (VQA) pairs, and chain-of-thought (CoT) reasoning supervision to enable domain-aligned and interpretable learning. Off-the-shelf VLMs such as NVIDIA's Cosmos-Reason1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings; fine-tuning with VLM-AutoDrive improves Collision F1 from 0.00 to 0.69 and overall accuracy from 35.35% to 77.27%. VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks. Evaluated on real-world Nexar dashcam videos, it achieves substantial gains in Collision and Near-Collision detection while producing interpretable reasoning traces, bridging the gap between perception, causality, and decision reasoning in autonomous driving.

Abstract

Anomaly Detection

Explore similar work

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Vision-Language Models as copilots for Autonomous UAV Navigation: Analysis of Latency and Reliability in Degraded Environments