cs.CVNov 5, 2026

Action-Conditioned VLM Monitoring for Autonomous Vehicle Semantic Anomalies

Authors: Daniel Adejumo, Omkar Ankalkope, Kunal Runwal, Aliasghar Arab

Organizations: Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA. · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.

Abstract

We study action-conditioned monitoring of autonomous-vehicle behavior in semantically unusual driving scenes. The target is whether the executed ego action is appropriate for the scene: the same scene can admit both normal and anomalous behavior, so visual unusualness alone does not determine the monitoring label. A Vision-Language Model (VLM) first predicts the scene-required action from an outcome-hidden video window; the monitor then compares that expectation with the realized ego-motion trajectory under an explicit compatibility rule. To study this monitoring task, we contribute a six-field taxonomy of 112 paired scenarios spanning both failure polarities (42 derived from documented incidents), a generated video corpus with estimated trajectories, and an exploratory two-stage monitor on that corpus. On 235 clips from 15 scenario families, the two-stage protocol with a fixed T-2.5 s window obtains 83.0% accuracy and anomaly recall 0.773, versus 74.5% and 0.536 for a single-stage video-only baseline; specificity is 0.880 versus 0.928.

Figures & tables

Explore similar work

CardsList
  1. How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study

    Apr 8, 2026Roberto Brusnicki, Mattia Piccinini, Johannes BetzVisual Evidence Sensitivity in VLMsTemporal Reasoning in VLMs

  2. Localized Conformal Safety Monitoring with Vision-Language Models for Autonomous Driving

    Oct 2, 2026Luís Marques, Rong Fang, Disha Kamale +1Autonomous Driving Safety EvaluationAutonomous Driving