Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

Xiaosong Jia; Yuqian Shao; Zhenjie Yang; Qifeng Li; Zhiyuan Zhang; Junchi Yan

cs.ROApr 1, 2026

Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

Authors: Xiaosong Jia, Yuqian Shao, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, Junchi Yan

Organizations: Institute of Trustworthy Embodied AI, Fudan University and Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, 200441, China. · School of Computer Science & School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, 200240, China.

Abstract

With the rise of vision-language models (VLM), their application for autonomous driving (VLM4AD) has gained significant attention. Meanwhile, in autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation, as it can evaluate the performance of the model under cumulative errors and out-of-distribution inputs. However, existing VLM4AD benchmarks evaluate the model`s scene understanding ability under open-loop, i.e., via static question-answer (QA) dataset. This kind of evaluation fails to assess the VLMs performance under out-of-distribution states rarely appeared in the human collected datasets.To this end, we present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving, which introduces: (1) DriveCommenter, a closed-loop generator that automatically generates diverse, behavior-grounded question-answer pairs for all driving situations in CARLA,including severe off-route and off-road deviations previously unassessable in simulation. (2) A unified protocol and interface that allows modern VLMs to be directly plugged into the Bench2Drive closed-loop environment to compare with traditional agents. (3) A flexible reasoning and control framework, supporting multi-format visual inputs and configurable graph-based chain-of-thought execution. (4) A complete development ecosystem. Together, these components form a comprehensive closed-loop benchmark for VLM4AD. All codes and annotated datasets are open sourced.

Figures & tables

Fig. 1: Overview of the Bench2Drive-VL. (i) The expert model DriveCommenter leverages privileged information to generate ground-truth answers for vision-language questions. (ii) The VLM under evaluation takes annotated sensor data as input, answers the same questions, and controls the ego vehicle. (iii) A VQA evaluator compares the VLM’s responses to the ground truth provided by DriveCommenter, while the action module converts the VLM’s natural language instructions into control signals for vehicle actuation.

Benchmark	Open-loop				Closed-loop
Benchmark	Percept.	Predict.	Plan.	Behavior	Percept.	Predict.	Plan.	Behavior
LangAuto [ 2 ]								CARLA
DriveBench [ 17 ]	VQA	VQA	VQA	VQA
DrivingVQA [ 18 ]	VQA	VQA	VQA	VQA
DriveLMM-o1 [ 26 ]	VQA	VQA	VQA	VQA
VLADBench [ 19 ]	VQA	VQA	VQA	traj. L2

TABLE I: Comparison of VLM4AD Benchmarks.

Fig. 2: Overview of the Bench2Drive-VL closed-loop evaluation framework. The expert model DriveCommenter leverages privileged information to generate ground-truth answers for vision-language questions. The VLM under evaluation takes sensor data as input, answers the same questions, and controls the ego vehicle. A VQA evaluator compares the VLM’s responses to the ground truth provided by DriveCommenter, while the action module converts the VLM’s natural language instructions into control signals.

Fig. 9: Overview of DriveCommenter. Our VQAs cover all tasks (perception, prediction, planning, behaviour) related to all categories (vehicles, pedestrians, traffic signs, weather and scene layouts).

Fig. 10: Example of A Frame Annotated by DriveCommenter. Only few questions are shown for space.

QID	Question	DriveLM -CARLA	Drive Commenter
Perception
1	How many pedestrians are there?	✓	✓
2	Is the ego vehicle affected by a stop sign?	✓	✓
3	Is the ego vehicle affected by a speed limit sign?		✓
4	List the traffic signs affecting the ego vehicle in the current scenario.		✓
5	Is the ego vehicle affected by a traffic light?	✓	✓

TABLE II: Comparison of questions supported by DriveCommenter and DriveLM-CARLA’s automatic annotation method.

Fig. 11: Comparison between Existing VLM4AD Annotation Methods and DriveCommenter under Out-of-Distribution (OOD) Conditions. Existing methods focus on states that lie close to the expert trajectory, leaving a large portion of the simulator state space underexplored. In contrast, DriveCommenter is capable of covering and annotating a much broader set of OOD states.

Fig. 12: Visual Inputs Provided by Bench2Drive-VL. Annotated RGB Images (left) and BEV Maps (right).

Fig. 13: The Corresponding Chain-of-Graph Reasoning Flow.

Model	Input	Chain	Imp. Obj.	T. Sign	S. Limit	Col. Obj.	C. Lane	Brake	A. Desc	A. Keys
DriveCommenter			100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Qwen2.5VL-3B-Instruct	cam	yes	53.62	75.47	27.58	5.21	81.81	62.94	69.69	46.51
Qwen2.5VL-3B-Instruct	BEV	yes	52.07	74.08	25.22	6.18	80.87	57.31	63.69	45.37
Qwen2.5VL-3B-Instruct	cam	no	50.37	78.76	24.64	4.82	75.04	57.67	58.07	51.03
Qwen2.5VL-3B-Instruct	BEV	no	53.45	72.06	30.33	8.31	66.27	50.39	52.63	44.42
Gemma3-4b-it	cam	yes	60.52	21.24	19.34	7.29	25.07	30.32	15.27	13.47

TABLE III: VQA performance of different VLMs with various integration strategies under Bench2Drive-VL . Abbreviations: Imp. Obj. = Import Objects (Q19), T. Sign = Traffic Sign (Q15), S. Limit = Speed Limit (Q7), Col. Obj. = Collide Object (Q47), C. Lane = Change Lane (Q13), Brake = (Q8), A. Desc. = Action Description (Q43), A. Key = Action Keys (Q50).

Model	Input	Chain	Driving Score $\uparrow$	Success Rate (%) $\uparrow$	Efficiency $\uparrow$	Comfortness $\uparrow$
DriveCommenter			100.00	100.00	154.91	12.48
Qwen2.5VL-3B-Instruct	cam	yes	42.24	0.00	65.61	36.20
Qwen2.5VL-3B-Instruct	BEV	yes	48.05	0.00	84.79	38.63
Qwen2.5VL-3B-Instruct	cam	no	58.93	20.00	74.72	37.67
Qwen2.5VL-3B-Instruct	BEV	no	41.44	20.00	82.41	47.01
Gemma3-4b-it	cam	yes	15.04	0.00	60.10	34.79

TABLE IV: Planning performance of different VLMs with various integration strategies under Bench2Drive-VL .

Fig. 14: Failure Case of CoT: Speed Limit Hallucation.

Fig. 15: Failure Case of BEV Map Input: Incorrect Lane Change.

Fig. 17: Data Overview Interface of DLVis.

Explore similar work

DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution
Sep 25, 2026Chengkai Xu, Jiaqi Liu, Yicheng Guo +2Autonomous Driving Benchmarks VLMs for Autonomous Driving
College of Transportation, Tongji University, Shanghai, China · Department of Computer Science, UNC Chapel Hill, NC, USA
LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
May 1, 2025Zhijie Qiao, Haowei Li, Zhong Cao +1VLM Evaluation Autonomous Driving Benchmarks
Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA
DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
May 22, 2026Hao Vo, Khoa Vo, Phu Loc Nguyen +10VLM Evaluation Autonomous Driving Datasets
University of Arkansas, USA · Google Research, Google · University of Liverpool, UK +1

Sep 25, 2026J/K move · Enter open · S save

DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution

Chengkai Xu, Jiaqi Liu, Yicheng Guo, Peng Hang+1

College of Transportation, Tongji University, Shanghai, China · Department of Computer Science, UNC Chapel Hill, NC, USA

Autonomous Driving Benchmarks VLMs for Autonomous Driving VLM Evaluation Autonomous Driving Simulation

Open paper AskPDF arXiv

Evaluating VLM-based autonomous driving remains difficult because driving competence is composite, where a capable system must ground traffic participants and hazards, integrate context across views and time, reason about future evolution, and act appropriately under closed-loop interaction. Existing benchmarks usually assess either open-loop understanding or closed-loop driving but provide limited structure for explaining how these abilities are organized, how they relate, and how they may inform model diagnosis and improvement. We present DriveHierarchy, a hierarchical benchmark that organizes VLM-based autonomous driving into four ranks, spanning perceptual grounding, contextual memory, mental reasoning, and closed-loop execution. To instantiate this hierarchy, we integrate multiple open-source autonomous-driving datasets into a unified open-loop benchmark with 76,798 question-answer pairs over 84,279 frames and develop a closed-loop simulation platform with interactive scenario construction on a real-world road network, from which 100 driving scenarios are curated for embodied evaluation. Experiments on 15 VLMs show that DriveHierarchy captures structured but non-redundant capability variation, relates open-loop understanding to closed-loop driving, and provides a practical basis for diagnosis and benchmark-guided optimization. DriveHierarchy therefore serves as a unified framework for evaluating and improving VLM-based autonomous driving systems. An anonymized project has been released on https://github.com/PerfectXu88/DriveHierarchy

Abstract

Explore similar work

DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving