Action-Conditioned VLM Monitoring for Autonomous Vehicle Semantic Anomalies
Authors: Daniel Adejumo, Omkar Ankalkope, Kunal Runwal, Aliasghar Arab
Organizations: Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA. · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.
We study action-conditioned monitoring of autonomous-vehicle behavior in semantically unusual driving scenes. The target is whether the executed ego action is appropriate for the scene: the same scene can admit both normal and anomalous behavior, so visual unusualness alone does not determine the monitoring label. A Vision-Language Model (VLM) first predicts the scene-required action from an outcome-hidden video window; the monitor then compares that expectation with the realized ego-motion trajectory under an explicit compatibility rule. To study this monitoring task, we contribute a six-field taxonomy of 112 paired scenarios spanning both failure polarities (42 derived from documented incidents), a generated video corpus with estimated trajectories, and an exploratory two-stage monitor on that corpus. On 235 clips from 15 scenario families, the two-stage protocol with a fixed T-2.5 s window obtains 83.0% accuracy and anomaly recall 0.773, versus 74.5% and 0.536 for a single-stage video-only baseline; specificity is 0.880 versus 0.928.
Figures & tables
Fig. 1: Action-conditioned monitoring. Context displacement defines paired scene-action labels; a VLM infers the scene-required action from an outcome-hidden window, then compares it with canonicalized ego motion.
#
Field
Description
1
Object in Driving Scene
Entity with a mandated response (stop sign, brake lights, pedestrian, fire hose)
2
Nominal Context
Where the object normally lives; reacting to it there is correct
3
OOD Context
Object or look-alike displaced outside its nominal context, appearance preserved
4
Normal Action in OOD
Context-aware response: react to the scene, not the pattern
5
Anomalous Action in OOD
Failure mode: react to the pattern as if nominal, or miss a real hazard presenting atypically
6
Example Scenario
One-sentence description; the generation / evaluation prompt seed
TABLE I: The six-field semantic-anomaly framework
Field
Fire hose
Brake lights
Nominal context
Coiled on a fire engine, off the roadway
Rear of a lead vehicle in the ego lane
OOD context
Charged hose across an active lane at a fire scene
Life-size car ad with glowing brake lamps on a night digital billboard
Normal action
Stop before the hose; never cross; wait to be waved through
Recognize the screen as media; hold speed and lane
Anomalous action
Classifies it as debris; drives over the hose
Reads a braking lead vehicle; brake-checks at speed
Example
A car drives over a charged fire hose at an active fire scene
A car brake-checks for brake lights on a nighttime video billboard
Tag
REAL ⋅ FN
SYN ⋅ FP
TABLE II: Example scenario pairs, one per failure polarity
Fig. 2: Illustrative synthetic driving scenarios: (A) traffic signals carried as cargo, (B) a stop-symbol depiction on clothing, (C) a road mural, and (D) a stop-symbol advertisement. The columns juxtapose the scene, expected action, depicted ego response, and scene-level interpretation.
Fig. 3: Action-conditioned monitoring through expectation-action comparison. Stage 1 predicts the scene-required action from an outcome-hidden window; stage 2 compares the canonicalized ego motion with an explicit compatibility relation.
Method
Balanced accuracy ↑
Anomaly recall ↑
Normal specificity ↑
Accuracy (correct/total)
Accuracy 95% Wilson CI
Single-stage video-only (baseline)
0.732
0.536 (59/110)
0.928 (116/125)
0.745 (175/235)
[0.685, 0.796]
Two-stage, fixed T−2.5 s (ours, primary)
0.826
0.773 (85/110)
0.880 (110/125)
0.830 (195/235)
[0.777, 0.872]
Primary − baseline (pp)
+9.4
+23.6
−4.8
+8.5
–
Two-stage, annotated t∗ (offline oracle)
0.818
0.764 (84/110)
0.872 (109/125)
0.821 (193/235)
[0.767, 0.865]
Annotated t∗− baseline (pp)
+8.6
+22.7
−5.6
+7.7
–
Majority class (reference)
0.500
0.000 (0/110)
1.000 (125/125)
0.532 (125/235)
–
TABLE III: Exploratory monitoring results on 235 clips (110 Anomalous, 125 Normal): primary comparison and offline oracle ablation.
Model
Early S/L
Full S
Δ
Verdict acc.
Cosmos3 (Nano, driving-tuned) [ 18 , 27 ]
18 / 38
22
+4
0.736
Cosmos3 (Super, driving-tuned)
8 / 26
12
+4
–
Qwen3.8-27B (selected)
31 / 46
30
−1
0.597
Qwen3.6-27B
30 / 41
28
−2
0.694
Qwen3.5-27B
29 / 42
32
+3
0.639
Qwen3-VL-32B-Thinking
19 / 37
20
+1
0.681 a
TABLE IV: Model comparison on the 72-clip selection subset: stage-1 expectation (early window), outcome-anchoring diagnostic, and single-stage verdict