Action-Conditioned VLM Monitoring for Autonomous Vehicle Semantic Anomalies
Authors: Daniel Adejumo, Omkar Ankalkope, Kunal Runwal, Aliasghar Arab
Organizations: Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA. · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.
We study action-conditioned monitoring of autonomous-vehicle behavior in semantically unusual driving scenes. The target is whether the executed ego action is appropriate for the scene: the same scene can admit both normal and anomalous behavior, so visual unusualness alone does not determine the monitoring label. A Vision-Language Model (VLM) first predicts the scene-required action from an outcome-hidden video window; the monitor then compares that expectation with the realized ego-motion trajectory under an explicit compatibility rule. To study this monitoring task, we contribute a six-field taxonomy of 112 paired scenarios spanning both failure polarities (42 derived from documented incidents), a generated video corpus with estimated trajectories, and an exploratory two-stage monitor on that corpus. On 235 clips from 15 scenario families, the two-stage protocol with a fixed T-2.5 s window obtains 83.0% accuracy and anomaly recall 0.773, versus 74.5% and 0.536 for a single-stage video-only baseline; specificity is 0.880 versus 0.928.
Figures & tables
Fig. 1: Action-conditioned monitoring. Context displacement defines paired scene-action labels; a VLM infers the scene-required action from an outcome-hidden window, then compares it with canonicalized ego motion.
#
Field
Description
1
Object in Driving Scene
Entity with a mandated response (stop sign, brake lights, pedestrian, fire hose)
2
Nominal Context
Where the object normally lives; reacting to it there is correct
3
OOD Context
Object or look-alike displaced outside its nominal context, appearance preserved
4
Normal Action in OOD
Context-aware response: react to the scene, not the pattern
5
Anomalous Action in OOD
Failure mode: react to the pattern as if nominal, or miss a real hazard presenting atypically
6
Example Scenario
One-sentence description; the generation / evaluation prompt seed
TABLE I: The six-field semantic-anomaly framework
Field
Fire hose
Brake lights
Nominal context
Coiled on a fire engine, off the roadway
Rear of a lead vehicle in the ego lane
OOD context
Charged hose across an active lane at a fire scene
Life-size car ad with glowing brake lamps on a night digital billboard
Normal action
Stop before the hose; never cross; wait to be waved through
Recognize the screen as media; hold speed and lane
Anomalous action
Classifies it as debris; drives over the hose
Reads a braking lead vehicle; brake-checks at speed
Example
A car drives over a charged fire hose at an active fire scene
A car brake-checks for brake lights on a nighttime video billboard
Tag
REAL ⋅ FN
SYN ⋅ FP
TABLE II: Example scenario pairs, one per failure polarity
Fig. 2: Illustrative synthetic driving scenarios: (A) traffic signals carried as cargo, (B) a stop-symbol depiction on clothing, (C) a road mural, and (D) a stop-symbol advertisement. The columns juxtapose the scene, expected action, depicted ego response, and scene-level interpretation.
Fig. 3: Action-conditioned monitoring through expectation-action comparison. Stage 1 predicts the scene-required action from an outcome-hidden window; stage 2 compares the canonicalized ego motion with an explicit compatibility relation.
Method
Balanced accuracy ↑
Anomaly recall ↑
Normal specificity ↑
Accuracy (correct/total)
Accuracy 95% Wilson CI
Single-stage video-only (baseline)
0.732
0.536 (59/110)
0.928 (116/125)
0.745 (175/235)
[0.685, 0.796]
Two-stage, fixed T−2.5 s (ours, primary)
0.826
0.773 (85/110)
0.880 (110/125)
0.830 (195/235)
[0.777, 0.872]
Primary − baseline (pp)
+9.4
+23.6
−4.8
+8.5
–
Two-stage, annotated t∗ (offline oracle)
0.818
0.764 (84/110)
0.872 (109/125)
0.821 (193/235)
[0.767, 0.865]
Annotated t∗− baseline (pp)
+8.6
+22.7
−5.6
+7.7
–
Majority class (reference)
0.500
0.000 (0/110)
1.000 (125/125)
0.532 (125/235)
–
TABLE III: Exploratory monitoring results on 235 clips (110 Anomalous, 125 Normal): primary comparison and offline oracle ablation.
Model
Early S/L
Full S
Δ
Verdict acc.
Cosmos3 (Nano, driving-tuned) [ 18 , 27 ]
18 / 38
22
+4
0.736
Cosmos3 (Super, driving-tuned)
8 / 26
12
+4
–
Qwen3.8-27B (selected)
31 / 46
30
−1
0.597
Qwen3.6-27B
30 / 41
28
−2
0.694
Qwen3.5-27B
29 / 42
32
+3
0.639
Qwen3-VL-32B-Thinking
19 / 37
20
+1
0.681 a
TABLE IV: Model comparison on the 72-clip selection subset: stage-1 expectation (early window), outcome-anchoring diagnostic, and single-stage verdict
The rapid growth of ego-centric dashcam footage presents a major challenge for detecting safety-critical events such as collisions and near-collisions, scenarios that are brief, rare, and difficult for generic vision models to capture. While multimodal large language models (MLLMs) demonstrate strong general reasoning ability, they underperform in driving contexts due to domain and temporal misalignment. We introduce VLM-AutoDrive, a modular post-training framework for adapting pretrained Vision-Language Models (VLMs) to high-fidelity anomaly detection. The framework integrates metadata-derived captions, LLM-generated descriptions, visual question answering (VQA) pairs, and chain-of-thought (CoT) reasoning supervision to enable domain-aligned and interpretable learning. Off-the-shelf VLMs such as NVIDIA's Cosmos-Reason1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings; fine-tuning with VLM-AutoDrive improves Collision F1 from 0.00 to 0.69 and overall accuracy from 35.35% to 77.27%. VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks. Evaluated on real-world Nexar dashcam videos, it achieves substantial gains in Collision and Near-Collision detection while producing interpretable reasoning traces, bridging the gap between perception, causality, and decision reasoning in autonomous driving.
Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal +7
Vision-Language Models (VLMs) are increasingly proposed for autonomous driving tasks, yet their performance on sequential driving scenes remains poorly characterized, particularly regarding how input configurations affect their capabilities. We introduce VENUSS (VLM Evaluation oN Understanding Sequential Scenes), a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes, establishing baselines for future research. Building upon existing datasets, VENUSS extracts temporal sequences from driving videos, and generates structured evaluations across custom categories. By comparing 25+ existing VLMs across 2,600+ scenarios, we reveal how even top models achieve only 57% accuracy, not matching human performance under similar constraints (65%) and exposing significant capability gaps. Our analysis shows that VLMs excel with static object detection but struggle with understanding vehicle dynamics and temporal relations. VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations - resolution, frame count, temporal intervals, spatial layouts, and presentation modes - affect performance on sequential driving scenes. Supplementary material available at https://TUM-AVS.github.io/VENUSS/.
Roberto Brusnicki, Mattia Piccinini, Johannes Betz
Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, Munich, Germany
Monitoring planned driving trajectories requires accurately estimating the collision likelihood with actors whose motion is itself impacted by the ego motion. Existing classical approaches are often limited by the quality of their forecasting model. Vision-language models (VLMs) have shown promise in reasoning about the consequences of high-level actions, yet their approximate predictions are unsuitable for safety-critical applications such as autonomous driving. Conformal prediction (CP) has emerged as a data-driven framework for quantifying the uncertainty of black-box model predictions. We propose Split Label-Localized Conformal Prediction (SLLCP), a post-hoc calibration layer over frozen VLMs that transforms their unreliable predictions into probabilistically calibrated safety prediction sets. We consider how the ability to estimate safety can depend on the observed driving scene and introduce a localized procedure that upweights relevant past experience when calculating uncertainty thresholds. We provide label-conditional finite-sample distribution-free coverage under exchangeability. Evaluated over 15k CARLA trajectories from unseen scenarios, SLLCP correctly flags 89.6% of collision-causing trajectories with a Qwen backbone and 88.4% with a Cosmos backbone, while the base VLMs only flagged 4.6% and 39.1% of the collision-causing trajectories, respectively. These results indicate that local, label-conditional calibration can reduce missed unsafe trajectories.