Organizations: Department of Computer Science, Iowa State University, Ames, IA, USA · Department of Civil, Construction and Environmental Engineering, Iowa State University, Ames, IA, USA
Crash detection from video feeds is a critical problem in intelligent transportation systems. Recent developments in large language models (LLMs) and vision-language models (VLMs) have transformed how we process, reason about, and summarize multimodal information. This paper surveys recent methods leveraging LLMs for crash detection from video data. We present a structured taxonomy of fusion strategies, summarize key datasets, analyze model architectures, compare performance benchmarks, and discuss ongoing challenges and opportunities. Our review provides a foundation for future research in this fast-growing intersection of video understanding and foundation models.
Figures & tables
Fig. 1: Taxonomy of LLM-based crash detection in videos categorized by fusion strategy, LLM role, and input granularity.
Fig. 2: Accuracy vs. Latency Trade-off Analysis for LLM-based Crash Detection Methods. The green zone represents deployment-ready systems ( <500 ms), yellow shows conditional deployment scenarios, and red indicates research-only systems.
Fig. 3: Fusion strategies: (a) Early fusion combines tokenized inputs. (b) Late fusion processes modalities separately. (c) Cross-attention integrates visual features into LLM attention layers.
Fig. 4: Radar chart comparing key attributes of crash detection datasets. ‘Size’ is shown on a normalized log scale, ‘Annotation Granularity’ is on a scale from 1 (weak) to 4 (fine-grained), and ‘Source Diversity’ is on a scale of 1 (single) to 2 (mixed).
Fig. 5: A generic architecture for an LLM-based crash detection system.
Feature
Visual Encoder + LLM Decoder
Frozen LLM + Learned Adapter
Joint Vision-Language Pretraining
Example Model(s)
Video-LLaMA, Flamingo
BLIP-2, MiniGPT-4, CrashLLM
VERA, Unified-IO
Key Innovation
End-to-end multimodal reasoning by treating video frames as a foreign language for the LLM.
Parameter-Efficient Fine-Tuning (PEFT) via lightweight bridges.
Deep, synergistic fusion of modalities learned from scratch.
Architectural Detail
A vision encoder (e.g., ViT-G) output is projected to the LLM’s word embedding space. The full model is fine-tuned.
A small network (e.g., Q-Former, LoRA) aligns frozen vision features with a frozen LLM.
A unified Transformer encoder-decoder trained on interleaved image-text data.
Trainable Parameters
Very High (e.g., > 8B for a ViT-G + 7B LLM).
Very Low (e.g., 1M–100M for adapters; > 99% frozen).
Extremely High (often > 10B).
Computational Cost (Training)
Prohibitive. Requires massive GPU memory ( ∼ TB) and high TFLOPs due to full backpropagation.
TABLE V: SSL/FSL Performance on TU-DAT Crash Detection (2025)
Method
AUC
BLEU
IoU
xAUC
PDS
FA–F1
CrashLLM
0.885
42.3
0.652
0.72
0.45
0.81
LAVAD
0.850
31.4
0.581
0.65
0.78
0.82
Holmes-VAD
0.865
52.8
0.684
0.74
0.52
0.80
VERA
0.866
56.1
0.713
0.76
0.62
0.84
CRASH
0.832
28.6
–
–
0.85
0.79
AnomalyCLIP
0.872
28.1
0.623
0.67
0.82
0.85
TABLE VI: Composite Metrics for Holistic Crash Detection Evaluation
Model
Precision
Memory (GB)
Latency (ms)
Accuracy Drop (%)
MiniGPT-4
FP32
4.8
650
Baseline
MiniGPT-4
FP16
2.4
450
-0.5
MiniGPT-4
INT8
0.6
180
-2.3
MiniGPT-4
INT4
0.3
95
-5.8
LAVAD
FP16
1.8
300
Baseline
LAVAD
INT8
0.45
85
-1.1
TABLE X: Quantization Impact on Crash Detection Performance
Fig. 6: Integration of an LLM-based crash detection module within a standard Autonomous Vehicle (AV) perception-planning-action pipeline. The model provides semantic risk assessment to the planning module, enabling more nuanced and proactive vehicle control.