Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving
Organizations: Korea Advanced Institute of Science and Technology, Daejeon, Korea
Abstract
Natural language explanations promised by modern Video Large Language Models (Video LLMs) offer autonomous driving systems a path toward detailed scene description and interpretable decision making on the road. While recent benchmarks have begun assessing these models on driving scenes, they mainly target routine traffic conditions and general comprehension tasks based on existing driving datasets and therefore fail to evaluate model performance on the rarest and most safety-critical driving events, also known as "corner cases". To provide a comprehensive evaluation of Video LLMs on precisely these scenarios, we introduce DashBench, a benchmark testing 25 Video LLMs on 1,709 carefully curated driving videos spanning not only collision-related corner cases but also rare non-collision ones. Through extensive prompts and open-ended question answering, we expose a critical capability gap: all evaluated models perform substantially below the threshold required for safety-critical driving applications, frequently misjudging or missing dangerous events on the road ahead and hallucinating non-existent agents or actions in their explanations. We believe DashBench, combining an extensive dataset with a rigorous evaluation toolkit and a detailed diagnosis of current model limitations, drives progress toward safer, more explainable autonomous driving systems. The code and data will be shared at: cornercase-dashbench.github.io
Figures & tables
| Model | Q1 | Q2 | Q3 | Q4 R | Q5 | Min Agree | Max Agree | Avg Agree | Q4 TF |
| Closed-source | |||||||||
| GPT-5 | 19.95 | 36.92 | 31.19 | 46.52 | 39.15 | 19.95 | 46.52 | 34.75 | 97.83 |
| Gemini-3-Flash | 32.36 | 35.28 | 29.84 | 39.58 | 34.99 | 29.84 | 39.58 | 34.41 | 94.85 |
| Open-source | |||||||||
| Qwen3-VL (235B) | 18.72 | 20.19 | 16.79 | 25.92 | 21.71 | 16.79 | 25.92 | 20.67 | 72.03 |
| Qwen3-VL (8B) | 18.90 | 15.51 | 14.45 | 22.00 | 21.53 | 14.45 | 22.00 | 18.48 | 74.96 |
| Judge | GPT-5 | Gemini-3 -Flash | Qwen3-VL (8B) | InternVL3.5 (8B) | Ovis2.5 (9B) | Avg |
|---|---|---|---|---|---|---|
| DeepSeekV3 | 0.98 | 0.93 | 0.93 | 0.95 | 1.00 | 0.96 |
| Llama3.1(70B) | 0.90 | 0.92 | 0.76 | 0.81 | 0.85 | 0.85 |
| Multi-LLM [ Verga et al.(2024)Verga, Hofstätter, Althammer, Su, Piktus, Arkhangorodsky, Xu, White, and Lewis ] | 0.63 | 0.57 | 0.44 | 0.52 | 0.54 | 0.54 |
| Model | Q1 | Q2 | Q3 | Q4 R | Q5 | Avg Agree | Q4 TF |
|---|---|---|---|---|---|---|---|
| Closed-source | |||||||
| GPT-5 | 20.90 | 36.09 | 29.28 | 47.39 | 38.40 | 34.41 | 96.96 |
| + CCW | 19.68_{{\scriptstyle{\color[rgb]{0.8,0,0}-1.22}}} | 42.41_{{\scriptstyle{\color[rgb]{0,0.55,0}+6.32}}} | 34.51_{{\scriptstyle{\color[rgb]{0,0.55,0}+5.23}}} | 51.40_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.01}}} | 43.01_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.61}}} | 38.20_{{\scriptstyle{\color[rgb]{0,0.55,0}+3.79}}} | 97.81_{{\scriptstyle{\color[rgb]{0,0.55,0}+0.85}}} |
| Gemini-3-Flash | 32.56 | 34.99 | 28.43 | 37.96 | 34.39 | 33.67 | 92.34 |
| + CCW | 34.63_{{\scriptstyle{\color[rgb]{0,0.55,0}+2.07}}} | 39.37_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.38}}} | 30.13_{{\scriptstyle{\color[rgb]{0,0.55,0}+1.70}}} | 45.44_{{\scriptstyle{\color[rgb]{0,0.55,0}+7.48}}} | 40.34_{{\scriptstyle{\color[rgb]{0,0.55,0}+5.95}}} | 37.98_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.31}}} | 94.29_{{\scriptstyle{\color[rgb]{0,0.55,0}+1.95}}} |
| Open-source | |||||||
| Model | Q1 | Q2 | Q3 | Q4 R | Q5 | Avg Agr. | Q4 TF |
|---|---|---|---|---|---|---|---|
| InternVL3.5 (8B) | 4.27 | 7.67 | 6.55 | 12.17 | 12.52 | 8.64 | 78.17 |
| + VDGD [ Ghosh et al.(2025)Ghosh, Evuru, Kumar, Tyagi, Nieto, Jin, and Manocha ] | 4.10{\scriptstyle{\color[rgb]{0.8,0,0}-0.17}} | 5.32{\scriptstyle{\color[rgb]{0.8,0,0}-2.35}} | 6.96{\scriptstyle{\color[rgb]{0,0.55,0}+0.41}} | 9.13{\scriptstyle{\color[rgb]{0.8,0,0}-3.04}} | 10.65{\scriptstyle{\color[rgb]{0.8,0,0}-1.87}} | 7.23{\scriptstyle{\color[rgb]{0.8,0,0}-1.41}} | 58.46{\scriptstyle{\color[rgb]{0.8,0,0}-19.71}} |
| + SECOND [ Park et al.(2025)Park, Kim, Kim, and Do ] | 4.04{\scriptstyle{\color[rgb]{0.8,0,0}-0.23}} | 7.55{\scriptstyle{\color[rgb]{0.8,0,0}-0.12}} | 7.14{\scriptstyle{\color[rgb]{0,0.55,0}+0.59}} | 13.40{\scriptstyle{\color[rgb]{0,0.55,0}+1.23}} | 12.87{\scriptstyle{\color[rgb]{0,0.55,0}+0.35}} | 9.00{\scriptstyle{\color[rgb]{0,0.55,0}+0.36}} | 81.33{\scriptstyle{\color[rgb]{0,0.55,0}+3.16}} |
| + TRiCD [ Xing et al.(2026)Xing, Zha, Zu, Li, Li, and Yan ] | 5.03{\scriptstyle{\color[rgb]{0,0.55,0}+0.76}} | 8.72{\scriptstyle{\color[rgb]{0,0.55,0}+1.05}} | 7.20{\scriptstyle{\color[rgb]{0,0.55,0}+0.65}} | 13.34{\scriptstyle{\color[rgb]{0,0.55,0}+1.17}} | 14.69{\scriptstyle{\color[rgb]{0,0.55,0}+2.17}} | 9.80{\scriptstyle{\color[rgb]{0,0.55,0}+1.16}} | 75.13{\scriptstyle{\color[rgb]{0.8,0,0}-3.04}} |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Collision Categories (46 types, 890 samples) | Non-Collision Categories (42 types, 819 samples) |
|---|---|
| car hits a curb (32) | car has failure of components (22) |
| car hits car (30) | car is out of control (105) |
| car hits cyclist (32) | cyclist falls down (6) |
| car hits motorbike (30) | ego-car has failure of components (1) |
| car hits pedestrian (31) | ego-car is out of control (81) |
| car hits telegraphpoles (22) | ego-car is overtaken by a car (105) |
| Collision Categories (46 types, 890 samples) | Non-Collision Categories (42 types, 819 samples) | ||
|---|---|---|---|
| car hits a curb | 32.50% | car has failure of components | 20.00% |
| car hits car | 9.33% | car is out of control | 31.05% |
| car hits cyclist | 31.25% | cyclist falls down | 23.33% |
| car hits motorbike | 28.00% | ego-car has failure of components | 0.00% |
| car hits pedestrian | 21.29% | ego-car is out of control | 22.47% |
| car hits telegraphpoles | 26.36% | ego-car is overtaken by a car | 11.24% |
| Model | Q1 | Q2 | Q3 | Q4 R | Q5 | Avg Agree |
|---|---|---|---|---|---|---|
| Closed-source | ||||||
| GPT-5 | ||||||
| Gemini-3-Flash | ||||||
| Open-source | ||||||
| Qwen3-VL (8B) | ||||||
| Ovis2.5 (9B) | ||||||
| Setting | Value |
|---|---|
| Backbone model | OpenGVLab/InternVL3_5-8B |
| Conversation template | internvl2_5 |
| Input frames per video | 16 |
| Frame resolution | (BICUBIC) |
| Image normalization | ImageNet mean / std |
| Data type | float16 |
| Model | Q1 | Q2 | Q3 | Q4 R | Q5 | Avg Agree | Q4 TF |
|---|---|---|---|---|---|---|---|
| Closed-source | |||||||
| GPT-5 | 20.90 | 36.09 | 29.28 | 47.39 | 38.40 | 34.41 | 96.96 |
| + CCW 16 frames | 19.68_{{\scriptstyle{\color[rgb]{0.8,0,0}-1.22}}} | 42.41_{{\scriptstyle{\color[rgb]{0,0.55,0}+6.32}}} | 34.51_{{\scriptstyle{\color[rgb]{0,0.55,0}+5.23}}} | 51.40_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.01}}} | 43.01_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.61}}} | 38.20_{{\scriptstyle{\color[rgb]{0,0.55,0}+3.79}}} | 97.81_{{\scriptstyle{\color[rgb]{0,0.55,0}+0.85}}} |
| + CCW 32 frames | 19.20_{{\scriptstyle{\color[rgb]{0.8,0,0}-1.70}}} | 40.22_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.13}}} | 32.44_{{\scriptstyle{\color[rgb]{0,0.55,0}+3.16}}} | 50.43_{{\scriptstyle{\color[rgb]{0,0.55,0}+3.04}}} | 43.01_{{\scriptstyle{\color[rgb]{0,0.55,0}+4.61}}} | 37.06_{{\scriptstyle{\color[rgb]{0,0.55,0}+2.65}}} | 98.18_{{\scriptstyle{\color[rgb]{0,0.55,0}+1.22}}} |
| Open-source | |||||||
| Qwen3-VL (8B) | 17.98 | 14.95 | 13.97 | 20.90 | 21.02 | 17.76 | 70.84 |
Video-Language Model Evaluation
Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.
See all 115 papers