Organizations: Department of Computer Science, Iowa State University, Ames, IA, USA · Department of Civil, Construction and Environmental Engineering, Iowa State University, Ames, IA, USA
Crash detection from video feeds is a critical problem in intelligent transportation systems. Recent developments in large language models (LLMs) and vision-language models (VLMs) have transformed how we process, reason about, and summarize multimodal information. This paper surveys recent methods leveraging LLMs for crash detection from video data. We present a structured taxonomy of fusion strategies, summarize key datasets, analyze model architectures, compare performance benchmarks, and discuss ongoing challenges and opportunities. Our review provides a foundation for future research in this fast-growing intersection of video understanding and foundation models.
Figures & tables
Fig. 1: Taxonomy of LLM-based crash detection in videos categorized by fusion strategy, LLM role, and input granularity.
Fig. 2: Accuracy vs. Latency Trade-off Analysis for LLM-based Crash Detection Methods. The green zone represents deployment-ready systems ( <500 ms), yellow shows conditional deployment scenarios, and red indicates research-only systems.
Fig. 3: Fusion strategies: (a) Early fusion combines tokenized inputs. (b) Late fusion processes modalities separately. (c) Cross-attention integrates visual features into LLM attention layers.
Fig. 4: Radar chart comparing key attributes of crash detection datasets. ‘Size’ is shown on a normalized log scale, ‘Annotation Granularity’ is on a scale from 1 (weak) to 4 (fine-grained), and ‘Source Diversity’ is on a scale of 1 (single) to 2 (mixed).
Fig. 5: A generic architecture for an LLM-based crash detection system.
Feature
Visual Encoder + LLM Decoder
Frozen LLM + Learned Adapter
Joint Vision-Language Pretraining
Example Model(s)
Video-LLaMA, Flamingo
BLIP-2, MiniGPT-4, CrashLLM
VERA, Unified-IO
Key Innovation
End-to-end multimodal reasoning by treating video frames as a foreign language for the LLM.
Parameter-Efficient Fine-Tuning (PEFT) via lightweight bridges.
Deep, synergistic fusion of modalities learned from scratch.
Architectural Detail
A vision encoder (e.g., ViT-G) output is projected to the LLM’s word embedding space. The full model is fine-tuned.
A small network (e.g., Q-Former, LoRA) aligns frozen vision features with a frozen LLM.
A unified Transformer encoder-decoder trained on interleaved image-text data.
Trainable Parameters
Very High (e.g., > 8B for a ViT-G + 7B LLM).
Very Low (e.g., 1M–100M for adapters; > 99% frozen).
Extremely High (often > 10B).
Computational Cost (Training)
Prohibitive. Requires massive GPU memory ( ∼ TB) and high TFLOPs due to full backpropagation.
TABLE V: SSL/FSL Performance on TU-DAT Crash Detection (2025)
Method
AUC
BLEU
IoU
xAUC
PDS
FA–F1
CrashLLM
0.885
42.3
0.652
0.72
0.45
0.81
LAVAD
0.850
31.4
0.581
0.65
0.78
0.82
Holmes-VAD
0.865
52.8
0.684
0.74
0.52
0.80
VERA
0.866
56.1
0.713
0.76
0.62
0.84
CRASH
0.832
28.6
–
–
0.85
0.79
AnomalyCLIP
0.872
28.1
0.623
0.67
0.82
0.85
TABLE VI: Composite Metrics for Holistic Crash Detection Evaluation
Model
Precision
Memory (GB)
Latency (ms)
Accuracy Drop (%)
MiniGPT-4
FP32
4.8
650
Baseline
MiniGPT-4
FP16
2.4
450
-0.5
MiniGPT-4
INT8
0.6
180
-2.3
MiniGPT-4
INT4
0.3
95
-5.8
LAVAD
FP16
1.8
300
Baseline
LAVAD
INT8
0.45
85
-1.1
TABLE X: Quantization Impact on Crash Detection Performance
Fig. 6: Integration of an LLM-based crash detection module within a standard Autonomous Vehicle (AV) perception-planning-action pipeline. The model provides semantic risk assessment to the planning module, enabling more nuanced and proactive vehicle control.
Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While vision-language models (VLMs) show strong general reasoning capabilities, their performance in safety-critical traffic scenarios remains insufficiently evaluated due to the ego-vehicle focus of existing benchmarks. To bridge this gap, we present CrashSight, a large-scale vision-language benchmark for roadway crash understanding using real-world roadside camera data. The dataset comprises 250 crash videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy. Tier 1 evaluates the visual grounding of scene context and involved parties, while Tier 2 probes higher-level reasoning, including crash mechanics, causal attribution, temporal progression, and post-crash outcomes. We benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal reasoning in safety-critical scenarios. We provide a detailed analysis of failure scenarios and discuss directions for improving VLM crash understanding. The benchmark provides a standardized evaluation framework for infrastructure-assisted perception in cooperative autonomous driving. The CrashSight benchmark, including the full dataset and code, is accessible at https://mcgrche.github.io/crashsight.
Rui Gan, Junyi Ma, Pei Li +4
University of Wisconsin–Madison · University of Wyoming · Columbia University
Traffic surveillance cameras capture accidents continuously, yet converting raw CCTV footage into structured event records that pinpoint when, where, and what type of collision occurred remains unsolved at scale. The ACCIDENT @ CVPR benchmark evaluates exactly this joint prediction under a strict constraint: no labeled real-world training data is available. We introduce a training-free, two-pass coarse-to-fine pipeline that pairs a frozen Qwen3-VL-32B-Instruct vision-language model with YOLO11x object detection and BoT-SORT tracking. A first pass sparsely samples the full clip to anchor the collision moment in time; a second pass re-examines a tight window around that estimate using frames annotated with stable vehicle identities and normalized bounding-box coordinates, which gives the model both a visual overlay and an explicit numeric description of the same scene. On the official 2,027-clip real-CCTV test set, our system achieves a three-way harmonic mean score of 0.504, surpassing all organizer-published baselines including the best multi-model ensemble (0.412) by a 22% relative margin.
Dipit Saha, Shah Mohammad Abdul Mannan, Mohammad Raihan Rashid +2
Bangladesh University of Engineering and Technology
We introduce ACCIDENT, a benchmark dataset for traffic accident detection in CCTV footage, designed to evaluate models in supervised (IID and OOD) and zero-shot settings, reflecting both data-rich and data-scarce scenarios. The benchmark consists of a curated set of 2,027 real and 2,211 synthetic clips annotated with the accident time, spatial location, and high-level collision type. We define three core tasks: (i) temporal localization of the accident, (ii) its spatial localization, and (iii) collision type classification. Each task is evaluated using custom metrics that account for the uncertainty and ambiguity inherent in CCTV footage. In addition to the benchmark, we provide a diverse set of baselines, including heuristic, motion-aware, and vision-language approaches, and show that ACCIDENT is challenging. You can access the ACCIDENT at: https://accidentbench.github.io
Lukas Picek, Michal Čermák, Marek Hanzl +1
PiVa AI · MIT · University of West Bohemia in Pilsen +1