VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

Mohammad Qazim Bhat; Yufan Huang; Niket Agarwal; Hao Wang; Michael Woods; John Kenyon; Tsung-Yi Lin; Xiaodong Yang

cs.CVMar 18, 2026

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

Authors: Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal, Hao Wang, Michael Woods, John Kenyon, Tsung-Yi Lin, Xiaodong Yang, +2 more

Organizations: NVIDIA, California, USA

Abstract

The rapid growth of ego-centric dashcam footage presents a major challenge for detecting safety-critical events such as collisions and near-collisions, scenarios that are brief, rare, and difficult for generic vision models to capture. While multimodal large language models (MLLMs) demonstrate strong general reasoning ability, they underperform in driving contexts due to domain and temporal misalignment. We introduce VLM-AutoDrive, a modular post-training framework for adapting pretrained Vision-Language Models (VLMs) to high-fidelity anomaly detection. The framework integrates metadata-derived captions, LLM-generated descriptions, visual question answering (VQA) pairs, and chain-of-thought (CoT) reasoning supervision to enable domain-aligned and interpretable learning. Off-the-shelf VLMs such as NVIDIA's Cosmos-Reason1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings; fine-tuning with VLM-AutoDrive improves Collision F1 from 0.00 to 0.69 and overall accuracy from 35.35% to 77.27%. VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks. Evaluated on real-world Nexar dashcam videos, it achieves substantial gains in Collision and Near-Collision detection while producing interpretable reasoning traces, bridging the gap between perception, causality, and decision reasoning in autonomous driving.

Figures & tables

Figure 3: Data Pipeline Examples.

Category	Annotation Type	Approx. Count	Source/Description
Understanding	MCQ Data (Unbalanced)	$\sim$ 53,000	From Nexar videos - 1k Collision, 9k Near-Collision, 43k Normal
	MCQ Data (Upsampled)	$\sim$ 68,000	Balanced (11k Collision, 14k Near-Collision, 43k Normal)
	Detailed Caption Data	$\sim$ 20,000	Rule-based, using Nexar video metadata
	Gemini Caption Data	$\sim$ 50,000	From chunked Nexar videos using Gemini
	VQA Data	$\sim$ 124,000	From detailed captions via LLaMA-3.1-70B
Reasoning	AV Reasoning Data	$\sim$ 12,000	From CR1 SFT stage

Table 1: Summary of Data Mixture for SFT grouped into Understanding and Reasoning splits

Model	Collision			Near-Collision			Normal Driving			Accuracy $\uparrow$
	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
Cosmos-Reason1 Zero-shot	0.000	0.000	0.000	0.454	0.0075	0.129	0.347	0.984	0.513	35.35
Cosmos-Reason1 Fine-tuned	0.947	0.545	0.692	0.696	0.833	0.758	0.765	0.939	0.843	77.27
NVILA Zero-shot	0.000	0.000	0.000	0.250	0.500	0.333	0.500	0.667	0.571	38.89
NVILA Fine-tuned	0.901	0.696	0.786	0.782	0.924	0.847	0.927	0.969	0.948	86.36

Table 2: Per-class precision, recall, and F1-score for different models and settings. Model-A includes both zero-shot and fine-tuned performance.

FPS	F1-scores			Overall Accuracy $\uparrow$
	C $\uparrow$	NC $\uparrow$	N $\uparrow$
1	0.341	0.029	0.536	40.91
15	0.537	0.634	0.733	65.66
30	0.640	0.757	0.849	76.26

Table 3: Effect of Frame Rate (FPS)

Model Size	Overall Accuracy $\uparrow$
NVILA-8B	83.83
NVILA-33B	80.30

Table 4: Effect of Model Size.

Learning Rate	F1-scores			Overall Accuracy $\uparrow$
	C $\uparrow$	NC $\uparrow$	N $\uparrow$
1e-06	0.567	0.271	0.674	55.55
2e-05	0.646	0.725	0.802	73.73
1e-05	0.640	0.757	0.849	76.26

Table 5: Effect of learning rate.

Video Duration	Spatial Resolution	F1-score $\uparrow$			Overall Accuracy $\uparrow$
		C	NC	N
6 sec	32x4	0.500	0.702	0.820	70.20
6 sec	128x32	0.640	0.757	0.849	76.26
4 sec	128x32	0.672	0.744	0.845	76.26
4 sec	192x48	0.711	0.748	0.857	77.77

Table 6: Impact of Video Duration and Resolution.

Data Distribution	F1-scores			Overall Accuracy $\uparrow$
	C $\uparrow$	NC $\uparrow$	N $\uparrow$
Unbalanced	0.00	0.603	0.748	56.57
Balanced	0.640	0.757	0.849	76.26

Table 7: Effect of Data Balancing.

Model	MCQ	Caption	VQA	Accuracy $\uparrow$
NVILA	✓			80.30
	✓	✓		83.83
	✓	✓	✓	86.36

Table 8: Accuracy for NVILA model with incremental data inclusion.

Model	MCQ	Caption	VQA	Reasoning MCQ	Reasoning MCQ x2	Reasoning VQA	Accuracy (Reasoning Off ✗)	Accuracy (Reasoning On ✓)
CR1	✓	✓	✓	-	-	-	74.75	n/a
	✓	✓	✓	✓	-	-	75.25	52.75
	✓	✓	✓	✓	✓	-	73.73	61.11
	✓	✓	✓	✓	✓	✓	77.27	63.13

Table 9: Effect of Chain-of-Thought Reasoning Supervision. Reasoning MCQs target classification alignment; Reason×2 doubles that data; Reason-VQA enables open-ended scene reasoning. Combining both yields the highest classification and reasoning-mode performance.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Prompt used to generate understanding VQA from LLaMA3.1-70B for enhancing generic event understanding capability, given programmatically generated structured-captions as input.

Figure A.2 : Prompt used to generate reasoning VQA from DeepSeek-R1 for enhancing generic reasoning capability, given programmatically generated structured-captions and VLM generated scene description as inputs.

Figure A.3 : Prompt used to generate thinking traces from DeepSeek-R1, given programmatically generated structured-captions, VLM generated scene description and MCQ as inputs.

Explore similar work

Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis
May 21, 2026Tomaso Trinci, Henrique Piñeiro Monteagudo, Leonardo TaccariMultimodal Large Language Models Autonomous Driving
Verizon Connect, Florence, Italy.
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous Driving LLM-Guided RL
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets
Mar 26, 2026Alex Koran, Dimitrios Sinodinos, Hadi Hojjati +3End-to-End Autonomous Driving Traffic Accident Anticipation
McGill University, Montréal, Canada · Mila - Québec AI Institute, Montréal, Canada · Nissan Motor Corporation, Yokohama, Japan

May 21, 2026J/K move · Enter open · S save

Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis

Tomaso Trinci, Henrique Piñeiro Monteagudo, Leonardo Taccari

Verizon Connect, Florence, Italy.

Multimodal Large Language Models Autonomous Driving Temporal Video Understanding

Open paper AskPDF arXiv

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding. However, their application to safety-critical driving scenarios remains limited by an inability to accurately perceive and reason about rare high-stakes dynamic events, such as collisions or near-collisions. To address this, we introduce a pipeline that enhances MLLM perception by fusing downsampled video frames with synchronized high-frequency telematics data (IMU and GPS) and semantic insights from specialized computer vision models. Our pipeline generates high-quality pseudo-labels, including descriptive captions and question-answer pairs, specifically designed to train MLLMs to identify and describe Safety-Critical Events (SCEs) in real-world driving footage. We show the effectiveness of our approach fine-tuning the open-source QwenVL-2.5 model via DoRA adapters: our experiments demonstrate significant improvements in identifying and explaining safety-critical events, with fewer than 50M trainable parameters and limited computational budget.

Abstract

Explore similar work

Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets