DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution

Chengkai Xu; Jiaqi Liu; Yicheng Guo; Peng Hang; Jian Sun

cs.AISep 25, 2026

DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution

Authors: Chengkai Xu, Jiaqi Liu, Yicheng Guo, Peng Hang, Jian Sun

Organizations: College of Transportation, Tongji University, Shanghai, China · Department of Computer Science, UNC Chapel Hill, NC, USA

Abstract

Evaluating VLM-based autonomous driving remains difficult because driving competence is composite, where a capable system must ground traffic participants and hazards, integrate context across views and time, reason about future evolution, and act appropriately under closed-loop interaction. Existing benchmarks usually assess either open-loop understanding or closed-loop driving but provide limited structure for explaining how these abilities are organized, how they relate, and how they may inform model diagnosis and improvement. We present DriveHierarchy, a hierarchical benchmark that organizes VLM-based autonomous driving into four ranks, spanning perceptual grounding, contextual memory, mental reasoning, and closed-loop execution. To instantiate this hierarchy, we integrate multiple open-source autonomous-driving datasets into a unified open-loop benchmark with 76,798 question-answer pairs over 84,279 frames and develop a closed-loop simulation platform with interactive scenario construction on a real-world road network, from which 100 driving scenarios are curated for embodied evaluation. Experiments on 15 VLMs show that DriveHierarchy captures structured but non-redundant capability variation, relates open-loop understanding to closed-loop driving, and provides a practical basis for diagnosis and benchmark-guided optimization. DriveHierarchy therefore serves as a unified framework for evaluating and improving VLM-based autonomous driving systems. An anonymized project has been released on https://github.com/PerfectXu88/DriveHierarchy

Figures & tables

Figure 1: Overview of DriveHierarchy , which organizes VLM-based autonomous driving into four progressively structured ranks, spanning perceptual grounding, contextual memory, mental reasoning, and closed-loop execution.

Panel of Figure 2 — Figure 2: Distribution of the DriveHierarchy benchmark, which summarizes the data composition of the open-loop ranks and the scenario allocation of the closed-loop benchmark.

Figure 3: Closed-loop evaluation platform for $R4$ , which combines simulation with interactive scenario construction on a real-road layout, enabling controllable evaluation under diverse traffic interactions.

Models	Avg.Score	R1 Perceptual Grounding								R2 Contextual Memory				R3 Mental Reasoning
		R1_1_A	R1_1_B	R1_1_C	R1_2_A	R1_2_B	R1_2_C	R1_3	R1_4	R2_1	R2_2_A	R2_2_B	R2_3	R3_1	R3_2
		0.030	0.027	0.056	0.035	0.036	0.029	0.056	0.031	0.064	0.079	0.067	0.090	0.233	0.167
Generalist
Qwen3-VL(2B) Team (2025b)	43.91	68.80	29.54	39.10	25.79	38.97	35.46	0.99	50.53	33.78	41.36	49.90	39.70	59.20	46.58
InternVL3_5(2B) Wang et al. (2025b)	39.28	62.00	30.97	45.40	7.62	9.92	33.41	0.07	36.35	30.71	35.94	44.90	35.20	54.70	45.45
Qwen3-VL(8B) Team (2025b)	45.09	70.00	33.32	51.20	46.70	46.76	35.74	1.05	50.23	32.87	41.25	53.60	37.70	56.90	45.84

Table 1: Open-loop benchmark results on DriveHierarchy for 15 VLMs. The overall score is computed using the frozen calibration weights shown in the header. Gray and green cells mark the second-best and best results, respectively.

Panel of Figure 4 — Figure 4: Representative capability profiles revealed by DriveHierarchy. The radar plots highlight both within-family scaling trends and architecture-specific trade-offs among similarly sized models.

Models	Sce 1	Sce 2	Sce 3	Sce 4	Sce 5	Sce 6	Sce 7	Sce 8	Sce 9	Sce 10	Avg.Score	Rank
Generalist
Qwen3-VL(2B)	1.176	0.062	0.000	0.686	0.114	0.299	0.518	0.620	0.207	0.084	0.377	14
InternVL3_5(2B)	1.813	1.411	0.121	1.443	0.635	0.377	1.964	0.903	1.644	1.872	1.218	7
Qwen3-VL(8B)	1.278	1.726	0.384	1.128	0.574	0.271	1.379	0.394	0.661	1.227	0.902	9
InternVL3_5(8B)	0.001	0.246	0.960	0.939	0.848	0.642	1.285	0.662	0.412	1.025	0.702	13
ZwZ(8B)	1.147	1.034	0.415	1.312	0.471	0.164	0.773	0.508	0.948	1.322	0.809	10

Table 2: Closed-loop benchmark results on

R4

across 10 scenario families. Higher scores indicate better driving performance under the composed route, safety, and efficiency metric.

Panel of Figure 5 — Figure 5: Correlation structure of DriveHierarchy across subtasks and ranks, which shows that the benchmark contains both locally coupled abilities and globally progressive capability levels.

Models	Avg.Score	R1 Perceptual Grounding								R2 Contextual Memory				R3 Mental Reasoning
		R1_1_A	R1_1_B	R1_1_C	R1_2_A	R1_2_B	R1_2_C	R1_3	R1_4	R2_1	R2_2_A	R2_2_B	R2_3	R3_1	R3_2
		0.03	0.027	0.056	0.035	0.036	0.029	0.056	0.031	0.064	0.079	0.067	0.09	0.233	0.167
Qwen3-VL(8B)	-0.17	-0.05	-0.14	-0.16	-0.26	-0.34	-0.15	-0.98	-0.12	-0.33	-0.15	-0.06	-0.39	-0.28	-0.09
Baseline*	45.10	70.00	33.32	51.20	46.70	46.76	35.74	1.05	50.23	32.87	41.25	53.60	37.70	56.90	45.84
B+R1_2_A	+2.66	+7.70	+12.78	-1.20	+46.54	+14.71	+5.86	0.00	+1.06	+0.17	+5.49	-1.00	-1.20	-2.30	+0.37
B+R1_2_B	+3.57	+5.50	+10.91	+1.80	+35.78	+47.82	+4.91	+0.01	-2.09	+1.99	+3.03	+1.10	-1.30	-1.80	+0.32

Table 3: Open-loop transfer results of benchmark-guided SFT on Qwen3-VL(8B). The first row reports the baseline deficit relative to the task-wise SOTA result and is used only for target selection.

Models	Sce 1	Sce 2	Sce 3	Sce 4	Sce 5	Sce 6	Sce 7	Sce 8	Sce 9	Sce 10	Avg.Score
Baseline	1.278	1.726	0.384	1.128	0.574	0.271	1.379	0.394	0.661	1.227	0.902
B+R1_2_A+ R1_2_B+R1_3	9.436	4.788	0.816	3.199	2.418	8.468	5.048	1.924	9.958	3.379	4.943
B+R2_1+R2_3	15.181	3.279	0.207	4.095	1.778	8.579	4.299	2.176	9.924	2.192	5.171
B+R3_1	2.453	3.512	0.122	2.657	1.762	17.400	4.176	2.499	9.477	1.892	4.595
B+ALL	22.585	3.146	0.734	15.888	2.144	3.694	11.855	2.340	16.717	1.105	8.021

Table 4: Closed-loop transfer of benchmark-guided supervised fine-tuning on

R4

. Each intervention is defined by capability groups selected from DriveHierarchy.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Open QA	Closed loop	VLM iface.	Hier.	Diag.	Primary focus
DriveLM Sima et al. (2024)	✓	✗	✓	✗	$\triangle$	Language-grounded driving QA
LingoQA Marcu et al. (2024)	✓	✗	✓	✗	✗	Language-grounded driving QA
DriveBench Xie et al. (2025)	✓	✗	✓	✗	$\triangle$	The Visual Robustness of VLMs
NAVSIM Dauner et al. (2024b)	✗	$\triangle$	✗	✗	✗	Planning-oriented simulation
SafeBench Xu et al. (2022)	✗	✓	✗	✗	$\triangle$	Safety-critical scenarios
Bench2Drive Jia et al. (2024)	✗	✓	✗	✗	$\triangle$	Closed-loop driving agents

Appendix

Table 5: Comparison with representative autonomous-driving benchmarks. “Hier.” indicates whether the benchmark explicitly organizes evaluation into a capability hierarchy. “Diag.” indicates whether it exposes capability-level diagnostic outputs rather than only task or route scores.

Task group	Filtering and balance controls	Validation signal
Recognition/counting	visibility projection checks, category vocabulary normalization, scene-level quotas	nonnegative labels, valid category IDs, duplicate-item checks
Distance/grounding	calibrated ego-object geometry, projected-coordinate validity, invalid-box removal	finite distances, in-frame boxes, IoU-compatible box format
Situation description	LingoQA-context filtering for current-scene and ego-action semantics	learned-judge compatibility and prompt/answer schema checks
Multi-view/temporal memory	overlap trigger, fixed clip windows, instance-ID deduplication, transition balancing	valid synchronized views, valid temporal order, instance continuity checks
Relation/reasoning/planning	minimum visible-object constraints, randomized choices, nontrivial motion clips	answer-choice consistency, source-scene sanity checks, order-label validation

Appendix

Table 6: Construction-quality controls for the refined open-loop set. The released manifest stores per-item source identifiers and answer labels so that users can audit source proportions and answer distributions for each task.

Models	Provider	Full name	Inference Hardware
Qwen3-VL(2B)	Qwen	Qwen3-VL-2B-Instruct	1 $\times$ A800
InternVL3_5(2B)	OpenGVLab	InternVL3_5-2B-Instruct	1 $\times$ A800
Qwen3-VL(8B)	Qwen	Qwen3-VL-8B-Instruct	1 $\times$ A800
InternVL3_5(8B)	OpenGVLab	InternVL3_5-8B-Instruct	1 $\times$ A800
ZwZ(8B)	inclusionAI	ZwZ-8B	1 $\times$ A800
MiniCPM-V-4_5(9B)	openbmb	MiniCPM-V-4_5	1 $\times$ A800

Appendix

Table 7: Model configurations used in the open-loop evaluation.

Symbol	Meaning	Value
$T$	Decoding temperature	$0.0$
$p$	Nucleus sampling threshold	$1.0$
$N_{\mathrm{gen}}$	Maximum generated tokens	$256$
$\lambda_{\mathrm{rep}}$	Repetition control coefficient	$1.0$
$\lambda_{\mathrm{pre}}$	Presence penalty coefficient	$0.0$
$B$	Number of samples per inference batch	$2$

Appendix

Table 8: Open-loop inference settings.

Setting	Meaning	Value
Hardware	Evaluation node configuration for CARLA	1 $\times$ A800
Control rate	Frequency of VLM action execution	$4$ Hz
Simulation step	Fixed simulator step length	$0.05$ s
Conversation window	Number of retained dialogue turns	$1$
image size	Front-view image resolution sent to the model	$800\times 450$
Off-road grace time	Allowed off-road duration before termination	$1.5$ s

Appendix

Table 9: Closed-loop runtime settings used for the R4 benchmark.

Symbol	Meaning	Value
$r$	LoRA rank	$8$
$\alpha$	LoRA scaling factor	$32$
$d_{\mathrm{train}}$	Training dtype	bfloat16
$B_{\mathrm{train}}$	Per-device training batch size	$1$
$B_{\mathrm{eval}}$	Per-device evaluation batch size	$1$
$G$	Gradient accumulation steps	$8$

Appendix

Table 10: Training configuration for benchmark-guided SFT.

Panel of Figure 6 — Figure 6: Representative examples from Rank 1-1 Questions.

Panel of Figure 7 — Figure 7: Representative examples from Rank 1-2 Questions.

Panel of Figure 8 — Figure 8: Representative examples from Rank 1-3 Questions.

Panel of Figure 9 — Figure 9: Representative example from Rank 1-4 Questions.

Panel of Figure 10 — Figure 10: Representative example from Rank 2-1 Questions.

Panel of Figure 11 — Figure 11: Representative example from Rank 2-2-A Questions.

Panel of Figure 12 — Figure 12: Representative example from Rank 2-2-B Questions.

Figure 13: Representative example from Rank 2-3 Questions.

Figure 14: Representative example from Rank 3-1 Questions.

Panel of Figure 15 — Figure 15: Representative example from Rank 3-2 Questions.

Figure 16: Visualization of part of the Scene Editor.

Panel of Figure 17 — Figure 17: Representative Scenarios from Rank 4-1 to Rank 4-2.

Panel of Figure 18 — Figure 18: Representative Scenarios from Rank 4-3 to Rank 4-6.

Panel of Figure 19 — Figure 19: Representative Scenarios from Rank 4-7 to Rank 4-10.

Resource	Type	License / Terms	Official Link
NuScenes	Dataset	CC BY-NC-SA 4.0	Data
nuPlan	Dataset	Apache License 2.0	Data
NAVSIM	Dataset	Apache License 2.0	Code
DriveLM	Dataset	Apache License 2.0	Code
DriveBench	Dataset / Benchmark	Apache License 2.0	Project
LingoQA	Dataset	Non-commercial use	Project

Appendix

Table 11: Public resources used in this work. Exact checkpoint names for evaluated models are listed in Table 7 .

Explore similar work

Apr 1, 2026

Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

With the rise of vision-language models (VLM), their application for autonomous driving (VLM4AD) has gained significant attention. Meanwhile, in autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation, as it can evaluate the performance of the model under cumulative errors and out-of-distribution inputs. However, existing VLM4AD benchmarks evaluate the model`s scene understanding ability under open-loop, i.e., via static question-answer (QA) dataset. This kind of evaluation fails to assess the VLMs performance under out-of-distribution states rarely appeared in the human collected datasets.To this end, we present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving, which introduces: (1) DriveCommenter, a closed-loop generator that automatically generates diverse, behavior-grounded question-answer pairs for all driving situations in CARLA,including severe off-route and off-road deviations previously unassessable in simulation. (2) A unified protocol and interface that allows modern VLMs to be directly plugged into the Bench2Drive closed-loop environment to compare with traditional agents. (3) A flexible reasoning and control framework, supporting multi-format visual inputs and configurable graph-based chain-of-thought execution. (4) A complete development ecosystem. Together, these components form a comprehensive closed-loop benchmark for VLM4AD. All codes and annotated datasets are open sourced.

Xiaosong Jia, Yuqian Shao, Zhenjie Yang +3

Institute of Trustworthy Embodied AI, Fudan University and Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, 200441, China. · School of Computer Science & School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, 200240, China.

Autonomous Driving Benchmarks VLMs for Autonomous Driving Autonomous Driving Simulation

Ask PDF

Jun 15, 2026

DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models

Autonomous driving has shifted towards end-to-end policy learning, where reliable, interpretable policy evaluation is a fundamental challenge as driving quality is highly context-dependent. Commonly used rule-based driving metrics like EPDMS are interpretable but lack context-awareness, while recent VLMbased evaluations are context-aware but limited by ambiguous VLM outputs and weak physical grounding. To evaluate driving in a manner that is both interpretable and context-aware, we introduce DriveJudge. DriveJudge is a driving evaluation agent that combines rule-grounded evaluation with Vision-Language Model (VLM) reasoning and selectively invokes physically-grounded deterministic rule functions after interpreting the environmental context. To train and evaluate DriveJudge, we curate a large-scale dataset of 33,577 challenging driving samples with human annotations on whether the driving behavior is reasonable in the given scenario. With this dataset, we address the underexplored problem of driving metric evaluation, and introduce two human-aligned benchmark tasks: Driving Quality Classification and Trajectory Preference Selection. DriveJudge outperforms EPDMS for driving quality classification by 21.23 AUC, and the recent VLM-based DriveCritic for trajectory preference selection by 6.5%, setting a new standard for interpretable and precise driving evaluation.

Xinglong Sun, Kevin Xie, Jenny Schmalfuss +5

NVIDIA · KE:SAI

VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving

Ask PDF

Apr 8, 2026

How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study

Vision-Language Models (VLMs) are increasingly proposed for autonomous driving tasks, yet their performance on sequential driving scenes remains poorly characterized, particularly regarding how input configurations affect their capabilities. We introduce VENUSS (VLM Evaluation oN Understanding Sequential Scenes), a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes, establishing baselines for future research. Building upon existing datasets, VENUSS extracts temporal sequences from driving videos, and generates structured evaluations across custom categories. By comparing 25+ existing VLMs across 2,600+ scenarios, we reveal how even top models achieve only 57% accuracy, not matching human performance under similar constraints (65%) and exposing significant capability gaps. Our analysis shows that VLMs excel with static object detection but struggle with understanding vehicle dynamics and temporal relations. VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations - resolution, frame count, temporal intervals, spatial layouts, and presentation modes - affect performance on sequential driving scenes. Supplementary material available at https://TUM-AVS.github.io/VENUSS/.

Roberto Brusnicki, Mattia Piccinini, Johannes Betz

Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, Munich, Germany

VLM Evaluation VLMs for Autonomous Driving Autonomous Driving Benchmarks

Ask PDF