Can MLLMs Reason over Surround-View Driving Videos? A Multi-View Autonomous Driving QA Benchmark
Organizations: College of Computer Science and Technology, Jilin University, Changchun, China · Yinwang Intelligent Technology Co., Ltd., Huawei, China · Stanford University, USA
Abstract
Surround-view video understanding in autonomous driving is a crucial capability for Multimodal Large Language Models (MLLMs) to achieve reliable driving reasoning. Real-world driving often requires information beyond the front-view camera. Existing MLLMs primarily focus on front-view information, while their spatio-temporal understanding and reasoning abilities remain unknown. To address this gap, we propose MVDriveBench, a high-information-density video QA benchmark for autonomous driving, characterized by synchronized surround-view inputs, spatio-temporal understanding, and reasoning-chain-oriented evaluation. We design 12 categories of QA tasks that cover the complete reasoning pipeline from low-level visual perception to high-level driving decision-making, enabling a systematic evaluation of MLLMs surround-view spatio-temporal understanding and reasoning capabilities in complex driving environments. We conduct a comprehensive evaluation of 23 open-source, closed-source, and expert MLLMs on MVDriveBench. The results show that current MLLMs still exhibit insufficient surround-view spatio-temporal reasoning ability. Meanwhile, closed-loop evaluations of specialist models in AlpaSim reveal a correlation between surround-view spatio-temporal reasoning capabilities and closed-loop driving performance, highlighting the importance of comprehensive multi-view understanding for reliable vehicle control.
Figures & tables
| Benchmark / Dataset | AD | #Videos/Images | #QA | #Tasks | Anno | Closed-ended | Open-ended | Multi-video | Multi-view |
| MVBench | ✗ | 4K | 4K | 20 | A | ✓ | ✗ | ✓ | ✗ |
| Video-MME | ✗ | 900 | 2.7K | 30 | M | ✓ | ✗ | ✓ | ✗ |
| LongVideoBench | ✗ | 3,763 | 6,678 | 17 | M | ✓ | ✗ | ✓ | ✗ |
| EgoSchema | ✗ | 5K | 5K | 1 | M | ✓ | ✗ | ✓ | ✗ |
| TempCompass | ✗ | 410 | 7,540 | 5 | M+A | ✓ | ✓ | ✓ | ✗ |
| BDD-X | ✓ | 6,984 | – | 2 | M | ✗ | ✓ | ✓ | ✗ |
| Model | Overall | Dynamic Motion Understanding | Multi-view Scene Understanding | Driving Reasoning | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rel. Mot. | Obj. Mot. | Tracking | Ego Mot. | Avg | Intent | Cross-view | Key Obj. | Key Event | Risk | Avg | Planning | Counterfactual | Rationality | Avg | ||
| Human | 88.5 | 87.5 | 80.3 | 90.1 | 88.8 | 86.7 | 77.7 | 94.1 | 90.1 | 81.6 | 92.3 | 87.2 | 95.9 | 94.1 | 89.5 | 93.2 |
| Closed Source Models | ||||||||||||||||
| GPT-5.5 [ 21 ] | 80.1 | 62.3 | 65.4 | 61.2 | 86.8 | 68.9 | 85.1 | 76.7 | 73.5 | 76.9 | 92.0 | 80.8 | 94.0 | 95.1 | 92.9 | 94.0 |
| GPT-5 [ 22 ] | 78.7 | 58.9 | 57.5 | 62.7 | 87.2 | 66.6 | 83.0 | 75.7 | 72.3 | 74.2 | 91.5 | 79.3 | 93.9 | 93.4 | 93.5 | 93.6 |
| GPT-4o [ 23 ] | 74.3 | 48.9 | 46.3 | 50.7 | 70.4 | 54.1 | 85.3 | 89.4 | 67.9 | 70.2 | 87.8 | 80.1 | 97.0 | 92.9 | 84.6 | 91.5 |
| Model | Overall | Dynamic Motion Understanding | Multi-view Scene Understanding | Driving Reasoning | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rel. Mot. | Obj. Mot. | Tracking | Ego Mot. | Avg | Intent | Cross-view | Key Obj. | Key Event | Risk | Avg | Planning | Counterfactual | Rationality | Avg | ||
| Closed Source Models | ||||||||||||||||
| GPT-5.5 | 78.8 1.3 | 61.0 | 64.2 | 59.8 | 85.5 | 67.6 | 82.5 | 76.5 | 73.2 | 76.5 | 89.0 | 79.5 | 92.6 | 92.9 | 91.6 | 92.7 |
| GPT-5 | 77.2 1.5 | 64.9 | 63.5 | 54.4 | 87.9 | 67.7 | 75.0 | 63.5 | 61.7 | 78.1 | 96.6 | 75.0 | 96.5 | 90.7 | 94.5 | 93.9 |
| GPT-4o | 65.1 9.2 | 41.5 | 35.7 | 25.0 | 77.7 | 45.0 | 77.2 | 42.7 | 54.6 | 68.6 | 85.0 | 65.6 | 98.1 | 90.7 | 85.3 | 91.4 |
| Gemini-2.5-Pro | 73.2 3.5 | 59.4 | 49.1 | 55.2 | 84.8 | 62.1 | 66.0 | 67.1 | 61.9 | 72.0 | 87.7 | 70.9 | 96.9 | 89.8 | 87.7 | 91.5 |
VLMs for Autonomous Driving
Vision-language models for autonomous driving connect camera and sensor data with language to interpret scenes, predict hazards, and plan actions.
See all 221 papers