Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets

Alex Koran; Dimitrios Sinodinos; Hadi Hojjati; Takuya Nanri; Fangge Chen; Narges Armanfard

cs.CVMar 26, 2026

Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets

Authors: Alex Koran, Dimitrios Sinodinos, Hadi Hojjati, Takuya Nanri, Fangge Chen, Narges Armanfard

Organizations: McGill University, Montréal, Canada · Mila - Québec AI Institute, Montréal, Canada · Nissan Motor Corporation, Yokohama, Japan

Abstract

High infraction rates remain the primary bottleneck for end-to-end (E2E) autonomous driving, as evidenced by the low driving scores on the CARLA Leaderboard. Despite collision-related infractions being the dominant failure mode in closed-loop evaluations, collision-aware representation learning has received limited attention. To address this gap, we first develop a Video-Language-Augmented Anomaly Detector (VLAAD), leveraging a Multiple Instance Learning (MIL) formulation to obtain stable, temporally localized collision signals for proactive prediction. To transition these capabilities into closed-loop simulations, we must overcome the limitations of existing simulator datasets, which lack multimodality and are frequently restricted to simple intersection scenarios. Therefore, we introduce CARLA-Collide, a large-scale multimodal dataset capturing realistic collision events across highly diverse road networks. Trained on this diverse simulator data, VLAAD serves as a collision-aware plug-in module that can be seamlessly integrated into existing E2E driving models. By integrating our module into a pretrained TransFuser++ agent, we demonstrate a 14.12% relative increase in driving score with minimal fine-tuning. Beyond closed-loop evaluation, we further assess the generalization capability of VLAAD in an open-loop setting using real-world driving data. To support this analysis, we introduce Real-Collide, a multimodal dataset of diverse dashcam videos paired with semantically rich annotations for collision detection and prediction. On this benchmark, despite containing only 0.6B parameters, VLAAD outperforms a multi-billion-parameter vision-language model, achieving a 23.3% improvement in AUC.

Figures & tables

Figure 1: Infraction rates by TF++ [ 9 ] on official CARLA Leaderboard training (Town 12) and validation (Town 13) routes. Collisions dominate total infractions, especially in validation.

Figure 2: Architecture overview. VLAAD Training : Using multimodal video-caption pairs as input, the XCLIP backbone remains frozen while the Adapter and Detector are optimized jointly using the cosine embedding loss, $\mathcal{L}_{cos}$ , and binary cross-entropy loss, $\mathcal{L}_{bce}$ . Driving Model Fine-Tuning & Inference : VLAAD operates as a frozen module, and only the video encoder is used. Its collision risk score is concatenated with high-level states (velocity, navigation commands) and passed to the global encoder to condition the transformer decoder’s waypoint and speed predictions. At timestep $t$ , the risk token is computed causally from the past buffer $[t-K,t]$ only (no future frames).

Figure 3: Overview of the CARLA-Collide data generation pipeline. Collision Clips: We record 40-frame (10s) segments from an online sensor agent (TF++), centering the collision event between 2.5s and 7.5s. Corresponding infraction logs are summarized by an LLM to produce event-specific captions. Normal Clips: Leveraging offline expert data from SimLingo, we perform a two-stage LLM summarization—first at the frame level (incorporating VQA and scenario data) and then at the clip level—to generate cohesive driving descriptions.

Split	Veh. Coll.	Ped. Coll.	Lay. Coll.	Total Clips	Unique Scenarios	Captions	Frames
Training	920	26	98	1044	37	✓	41.76k
Testing	411	29	37	477	34	✓	19.08k
Total	1331	55	135	1521	38	✓	60.84k

Table 1: Summary of collision clip statistics in CARLA-Collide.

Method		km ( $\uparrow$ )	Col/km ( $\downarrow$ )	RC ( $\uparrow$ )	IS ( $\uparrow$ )	DS ( $\uparrow$ )
Baseline	TF++	97.94	1.647	39.29	0.182	4.442
Fine-tuned	w/o VLAAD	97.11	1.730	38.61	0.183	4.380
	w/ VLAAD	89.86	1.871	36.18	0.187	4.440
	w/ VLAAD-MIL	104.29	1.609	41.89	0.217	5.069

Table 2: Closed-loop evaluation on Town 13 (average of 3 runs).

Model	AUC $(\uparrow)$	F1 $(\uparrow)$	Acc. $(\uparrow)$
XCLIP Baseline	0.499	0.640	0.552
LlavaNext (FT)	0.533	0.620	0.464
VLAAD (w/o Text)	0.721	0.692	0.715
VLAAD (w/o Adapter)	0.743	0.693	0.715
VLAAD (w/o MIL)	0.751	0.699	0.721
VLAAD (Full Model)	0.766	0.703	0.726

Table 3: Performance comparison with baselines on the Real-Collide dataset. Results are averaged over 5 runs. VLAAD models are trained on MMAU + BDDX.

Model	AUC $(\uparrow)$	F1 $(\uparrow)$	Acc. $(\uparrow)$
Baseline (w/o FT)	0.522	0.549	0.530
VLAAD (w/o MIL)	0.655	0.638	0.611
VLAAD (w/ MIL)	0.672	0.651	0.642

Table 4: Performance on simulated data. Results are averaged over five runs. VLAAD models are trained on CARLA-Collide (excluding Town 13).

Panel of Figure 4 — Figure 4: Temporal collision-risk predictions of VLAAD with and without MIL. (a) Real-Collide clip where a pickup truck cuts in and causes a collision (evaluated on the 20% held-out split). (b) CARLA-Collide validation clip in rain involving a minor collision and an open-door hazard (model trained on MMAU and BDDX).

Figure 5: Three normal driving clips from Real-Collide. VLAAD with MIL stays near zero across all videos, correctly identifying normal behavior, while the model w/o MIL shows persistent nonzero responses. Examples are from the held-out 20% test split.

Method	Driving Score $\uparrow$	Success Rate $\uparrow$
AD-MLP	18.05	0.00
TCP	40.70	15.00
VAD	42.35	15.00
UniAD	45.81	16.36
ThinkTwice	62.44	31.23
DriveAdapter	64.22	33.08

Table 5: Results on Bench2Drive . Scores for other methods are taken from [ 15 ] .

Method		km ( $\uparrow$ )	Col/km ( $\downarrow$ )	RC ( $\uparrow$ )	IS ( $\uparrow$ )	DS ( $\uparrow$ )
Baseline	TF++	97.94	1.647	39.29	0.182	4.442
Fine-tuned	w/o VLAAD	97.11	1.730	38.61	0.183	4.380
	w/ VLAAD	89.86	1.871	36.18	0.187	4.440
	w/ VLAAD-MIL	104.29	1.609	41.89	0.217	5.069
From-scratch	w/ VLAAD-MIL	111.50	1.583	46.41	0.199	4.881

Table 6: Closed-loop evaluation on Town 13 (average of 3 runs).

Figure 6: Temporal collision risk scores on a normal U-turn maneuver. The ego vehicle is initially stopped at a traffic light and driving nominally, then executes a left U-turn into the opposite driving direction, briefly swerving close to a utility pole in a way that visually resembles a near-crash, before completing the turn and proceeding normally. Despite the visually dramatic motion, the model maintains low snippet-level collision risk scores throughout, correctly treating the entire sequence as non-anomalous.

Figure 7: Temporal collision risk on a deer-crossing clip. The road is initially clear (left), a deer suddenly enters and crosses in front of the ego vehicle (middle), and the scene returns to normal once the animal exits the roadway (right). MIL keeps the collision-risk score near zero when the road is clear and produces a sharp, localized spike while the deer is on the road, whereas No MIL shows elevated scores throughout the clip, providing little temporal localization of the anomalous event.

Source	Ego-View	Role in Dataset	Scenarios
Nexar	Dashcam (Car)	Pos & Neg	Urban, Highway
RetroTrucks	Dashcam (Truck)	Pos Only	Anomalies
COOOL	Dashcam (Car)	Pos Only	Hazards, Nuisance
YouTube	Various	Pos & Neg	Diverse, Edge Cases

Table 7: Composition of the Curated Real-World Benchmark. The dataset combines standard passenger vehicle footage with truck-view anomalies and diverse web-scraped scenarios to ensure robust evaluation.

Figure 8: Infraction rates per kilometer on official CARLA leaderboard training (Town 12) and validation (Town 13) routes. Collisions dominate total infractions, especially in validation

Figure 9: Vehicle collision counts across leaderboard scenario types for training (Town 12) and validation (Town 13) routes. NoSpecificScenario means the collision occurred while in transit between scenarios.

Figure 10: Pedestrian collision counts across leaderboard scenario types for training (Town 12) and validation (Town 13) routes.

Figure 11: Layout (object) collision counts across leaderboard scenario types for training (Town 12) and validation (Town 13) routes. NoSpecificScenario means the collision occurred while in transit between scenarios.

Normal subset	AUC $\uparrow$	Train time (s)
sim5k	0.7048 $\pm$ 0.0157	173
sim10k	0.7373 $\pm$ 0.0138	229
sim15k	0.7361 $\pm$ 0.0296	$\sim$ 270
sim20k	0.7104 $\pm$ 0.0088	$\sim$ 310
sim25k	0.7347 $\pm$ 0.0178	$\sim$ 360

Table 8: Effect of scaling the amount of normal SimLingo training data while keeping the anomaly set fixed to carla_aug . All models use the same VLAAD-MIL architecture and optimization settings as Section B.2, including LSE pooling with

\gamma=10.0

, and are evaluated on the CARLA validation split (

n=425

). Results are reported as mean

\pm

std over 5 random seeds. Performance improves substantially from sim5k to sim10k , then saturates; larger normal sets provide no consistent AUC gain.

Explore similar work

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
Mar 18, 2026Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal +7VLMs for Autonomous Driving Vision-Language Models
NVIDIA, California, USA
A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving
Aug 21, 2026Jingtao Sun, Xiaohai He, Yike Zhang +4End-to-End Autonomous Driving Autonomous Driving Planning
SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
Oct 14, 2026Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6Autonomous Driving End-to-End Autonomous Driving
UC Irvine · UC San Diego · University of Cincinnati +1

Mar 18, 2026J/K move · Enter open · S save

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal, Hao Wang+6

NVIDIA, California, USA

VLMs for Autonomous Driving Vision-Language Models Autonomous Driving Perception Video Anomaly Detection Traffic Accident Anticipation

Open paper AskPDF arXiv

The rapid growth of ego-centric dashcam footage presents a major challenge for detecting safety-critical events such as collisions and near-collisions, scenarios that are brief, rare, and difficult for generic vision models to capture. While multimodal large language models (MLLMs) demonstrate strong general reasoning ability, they underperform in driving contexts due to domain and temporal misalignment. We introduce VLM-AutoDrive, a modular post-training framework for adapting pretrained Vision-Language Models (VLMs) to high-fidelity anomaly detection. The framework integrates metadata-derived captions, LLM-generated descriptions, visual question answering (VQA) pairs, and chain-of-thought (CoT) reasoning supervision to enable domain-aligned and interpretable learning. Off-the-shelf VLMs such as NVIDIA's Cosmos-Reason1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings; fine-tuning with VLM-AutoDrive improves Collision F1 from 0.00 to 0.69 and overall accuracy from 35.35% to 77.27%. VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks. Evaluated on real-world Nexar dashcam videos, it achieves substantial gains in Collision and Near-Collision detection while producing interpretable reasoning traces, bridging the gap between perception, causality, and decision reasoning in autonomous driving.

Abstract

Explore similar work

VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving

SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving