High infraction rates remain the primary bottleneck for end-to-end (E2E) autonomous driving, as evidenced by the low driving scores on the CARLA Leaderboard. Despite collision-related infractions being the dominant failure mode in closed-loop evaluations, collision-aware representation learning has received limited attention. To address this gap, we first develop a Video-Language-Augmented Anomaly Detector (VLAAD), leveraging a Multiple Instance Learning (MIL) formulation to obtain stable, temporally localized collision signals for proactive prediction. To transition these capabilities into closed-loop simulations, we must overcome the limitations of existing simulator datasets, which lack multimodality and are frequently restricted to simple intersection scenarios. Therefore, we introduce CARLA-Collide, a large-scale multimodal dataset capturing realistic collision events across highly diverse road networks. Trained on this diverse simulator data, VLAAD serves as a collision-aware plug-in module that can be seamlessly integrated into existing E2E driving models. By integrating our module into a pretrained TransFuser++ agent, we demonstrate a 14.12% relative increase in driving score with minimal fine-tuning. Beyond closed-loop evaluation, we further assess the generalization capability of VLAAD in an open-loop setting using real-world driving data. To support this analysis, we introduce Real-Collide, a multimodal dataset of diverse dashcam videos paired with semantically rich annotations for collision detection and prediction. On this benchmark, despite containing only 0.6B parameters, VLAAD outperforms a multi-billion-parameter vision-language model, achieving a 23.3% improvement in AUC.
Figures & tables
Figure 1: Infraction rates by TF++ [ 9 ] on official CARLA Leaderboard training (Town 12) and validation (Town 13) routes. Collisions dominate total infractions, especially in validation.
Figure 2: Architecture overview. VLAAD Training : Using multimodal video-caption pairs as input, the XCLIP backbone remains frozen while the Adapter and Detector are optimized jointly using the cosine embedding loss, Lcos , and binary cross-entropy loss, Lbce . Driving Model Fine-Tuning & Inference : VLAAD operates as a frozen module, and only the video encoder is used. Its collision risk score is concatenated with high-level states (velocity, navigation commands) and passed to the global encoder to condition the transformer decoder’s waypoint and speed predictions. At timestep t , the risk token is computed causally from the past buffer [t−K,t] only (no future frames).
Figure 3: Overview of the CARLA-Collide data generation pipeline. Collision Clips: We record 40-frame (10s) segments from an online sensor agent (TF++), centering the collision event between 2.5s and 7.5s. Corresponding infraction logs are summarized by an LLM to produce event-specific captions. Normal Clips: Leveraging offline expert data from SimLingo, we perform a two-stage LLM summarization—first at the frame level (incorporating VQA and scenario data) and then at the clip level—to generate cohesive driving descriptions.
Split
Veh. Coll.
Ped. Coll.
Lay. Coll.
Total Clips
Unique Scenarios
Captions
Frames
Training
920
26
98
1044
37
✓
41.76k
Testing
411
29
37
477
34
✓
19.08k
Total
1331
55
135
1521
38
✓
60.84k
Table 1: Summary of collision clip statistics in CARLA-Collide.
Method
km ( ↑ )
Col/km ( ↓ )
RC ( ↑ )
IS ( ↑ )
DS ( ↑ )
Baseline
TF++
97.94
1.647
39.29
0.182
4.442
Fine-tuned
w/o VLAAD
97.11
1.730
38.61
0.183
4.380
w/ VLAAD
89.86
1.871
36.18
0.187
4.440
w/ VLAAD-MIL
104.29
1.609
41.89
0.217
5.069
Table 2: Closed-loop evaluation on Town 13 (average of 3 runs).
Model
AUC (↑)
F1 (↑)
Acc. (↑)
XCLIP Baseline
0.499
0.640
0.552
LlavaNext (FT)
0.533
0.620
0.464
VLAAD (w/o Text)
0.721
0.692
0.715
VLAAD (w/o Adapter)
0.743
0.693
0.715
VLAAD (w/o MIL)
0.751
0.699
0.721
VLAAD (Full Model)
0.766
0.703
0.726
Table 3: Performance comparison with baselines on the Real-Collide dataset. Results are averaged over 5 runs. VLAAD models are trained on MMAU + BDDX.
Model
AUC (↑)
F1 (↑)
Acc. (↑)
Baseline (w/o FT)
0.522
0.549
0.530
VLAAD (w/o MIL)
0.655
0.638
0.611
VLAAD (w/ MIL)
0.672
0.651
0.642
Table 4: Performance on simulated data. Results are averaged over five runs. VLAAD models are trained on CARLA-Collide (excluding Town 13).
Figure 4: Temporal collision-risk predictions of VLAAD with and without MIL. (a) Real-Collide clip where a pickup truck cuts in and causes a collision (evaluated on the 20% held-out split). (b) CARLA-Collide validation clip in rain involving a minor collision and an open-door hazard (model trained on MMAU and BDDX).
Figure 5: Three normal driving clips from Real-Collide. VLAAD with MIL stays near zero across all videos, correctly identifying normal behavior, while the model w/o MIL shows persistent nonzero responses. Examples are from the held-out 20% test split.
Method
Driving Score ↑
Success Rate ↑
AD-MLP
18.05
0.00
TCP
40.70
15.00
VAD
42.35
15.00
UniAD
45.81
16.36
ThinkTwice
62.44
31.23
DriveAdapter
64.22
33.08
Table 5: Results on Bench2Drive . Scores for other methods are taken from [ 15 ] .
Method
km ( ↑ )
Col/km ( ↓ )
RC ( ↑ )
IS ( ↑ )
DS ( ↑ )
Baseline
TF++
97.94
1.647
39.29
0.182
4.442
Fine-tuned
w/o VLAAD
97.11
1.730
38.61
0.183
4.380
w/ VLAAD
89.86
1.871
36.18
0.187
4.440
w/ VLAAD-MIL
104.29
1.609
41.89
0.217
5.069
From-scratch
w/ VLAAD-MIL
111.50
1.583
46.41
0.199
4.881
Table 6: Closed-loop evaluation on Town 13 (average of 3 runs).
Figure 6: Temporal collision risk scores on a normal U-turn maneuver. The ego vehicle is initially stopped at a traffic light and driving nominally, then executes a left U-turn into the opposite driving direction, briefly swerving close to a utility pole in a way that visually resembles a near-crash, before completing the turn and proceeding normally. Despite the visually dramatic motion, the model maintains low snippet-level collision risk scores throughout, correctly treating the entire sequence as non-anomalous.
Figure 7: Temporal collision risk on a deer-crossing clip. The road is initially clear (left), a deer suddenly enters and crosses in front of the ego vehicle (middle), and the scene returns to normal once the animal exits the roadway (right). MIL keeps the collision-risk score near zero when the road is clear and produces a sharp, localized spike while the deer is on the road, whereas No MIL shows elevated scores throughout the clip, providing little temporal localization of the anomalous event.
Source
Ego-View
Role in Dataset
Scenarios
Nexar
Dashcam (Car)
Pos & Neg
Urban, Highway
RetroTrucks
Dashcam (Truck)
Pos Only
Anomalies
COOOL
Dashcam (Car)
Pos Only
Hazards, Nuisance
YouTube
Various
Pos & Neg
Diverse, Edge Cases
Table 7: Composition of the Curated Real-World Benchmark. The dataset combines standard passenger vehicle footage with truck-view anomalies and diverse web-scraped scenarios to ensure robust evaluation.
Figure 8: Infraction rates per kilometer on official CARLA leaderboard training (Town 12) and validation (Town 13) routes. Collisions dominate total infractions, especially in validation
Figure 9: Vehicle collision counts across leaderboard scenario types for training (Town 12) and validation (Town 13) routes. NoSpecificScenario means the collision occurred while in transit between scenarios.
Figure 10: Pedestrian collision counts across leaderboard scenario types for training (Town 12) and validation (Town 13) routes.
Figure 11: Layout (object) collision counts across leaderboard scenario types for training (Town 12) and validation (Town 13) routes. NoSpecificScenario means the collision occurred while in transit between scenarios.
Normal subset
AUC ↑
Train time (s)
sim5k
0.7048 ± 0.0157
173
sim10k
0.7373 ± 0.0138
229
sim15k
0.7361 ± 0.0296
∼ 270
sim20k
0.7104 ± 0.0088
∼ 310
sim25k
0.7347 ± 0.0178
∼ 360
Table 8: Effect of scaling the amount of normal SimLingo training data while keeping the anomaly set fixed to carla_aug . All models use the same VLAAD-MIL architecture and optimization settings as Section B.2, including LSE pooling with γ=10.0 , and are evaluated on the CARLA validation split ( n=425 ). Results are reported as mean ± std over 5 random seeds. Performance improves substantially from sim5k to sim10k , then saturates; larger normal sets provide no consistent AUC gain.
The rapid growth of ego-centric dashcam footage presents a major challenge for detecting safety-critical events such as collisions and near-collisions, scenarios that are brief, rare, and difficult for generic vision models to capture. While multimodal large language models (MLLMs) demonstrate strong general reasoning ability, they underperform in driving contexts due to domain and temporal misalignment. We introduce VLM-AutoDrive, a modular post-training framework for adapting pretrained Vision-Language Models (VLMs) to high-fidelity anomaly detection. The framework integrates metadata-derived captions, LLM-generated descriptions, visual question answering (VQA) pairs, and chain-of-thought (CoT) reasoning supervision to enable domain-aligned and interpretable learning. Off-the-shelf VLMs such as NVIDIA's Cosmos-Reason1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings; fine-tuning with VLM-AutoDrive improves Collision F1 from 0.00 to 0.69 and overall accuracy from 35.35% to 77.27%. VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks. Evaluated on real-world Nexar dashcam videos, it achieves substantial gains in Collision and Near-Collision detection while producing interpretable reasoning traces, bridging the gap between perception, causality, and decision reasoning in autonomous driving.
Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal +7
Vision-Language-Action (VLA) models have emerged as a powerful paradigm for end-to-end autonomous driving by jointly integrating perception, reasoning, and decision making within a unified multimodal framework. However, most existing VLA models formulate end-to-end autonomous driving as a visual question answering task, leading to unreliable and less interpretable decision reasoning. In addition, they fail to establish effective multi-modal interaction across heterogeneous sensors, thereby limiting robust scene perception and reliable driving reasoning in long-tail driving scenarios. To this end, we propose a robust VLA-based end-to-end autonomous driving system that combines multi-modality interaction with multi-trajectory planning and optimization, enabling more reliable, interpretable, and safer driving decisions. Our method comprises three core components: (1) Affinity-Guided Optimal Transport for main-auxiliary modality two-way interaction; (2) Distribution-Consistent Modality Transfer for heterogeneous modality distribution transfer and cross-modal interaction; (3) Multi-modal Multi-Trajectory Planning along with Perception-Oriented Trajectory Refinement for better driving decisions to long-tail driving scenarios. Experimental results in open-loop and closed-loop datasets demonstrate improvements in safety long-horizon driving reasoning and road scene perception over existing driving systems, highlighting the ability of our mutli-modality interaction and multi-trajectory planning and optimization for scalable VLA-based systems.
Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.
Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6
UC Irvine · UC San Diego · University of Cincinnati +1