CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning

Kang Yang; Shuai Liu; Hang Li; Yance Fang; Deying Li; Yongcai Wang

cs.CVSep 22, 2026

CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning

Authors: Kang Yang, Shuai Liu, Hang Li, Yance Fang, Deying Li, Yongcai Wang

Organizations: Renmin University of China · Hebei Key Laboratory of Real-virtual Integrated Autonomous Systems (RIAS)

Abstract

Vision-language models (VLMs) have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes. Infrastructure-side observations provide views beyond the ego vehicle's field of view, yet conventional cooperative-driving systems typically transform them into geometric representations for downstream perception and planning. Directly incorporating these views into VLMs offers an opportunity to improve cooperative scene understanding and trajectory planning. However, question answering and trajectory planning have not been jointly evaluated on the same real-world vehicle-infrastructure scenes. We present CoVLM-Bench, a benchmark for cooperative driving question answering (CDQA) and cooperative planning (CP) on vehicle-infrastructure paired scenes. CoVLM-Bench provides scene-grounded CDQA annotations, three-part rationales as auxiliary supervision, and future trajectory targets derived from recorded ego motion. It contains 2,196 paired frames with 35,136 CDQA annotations, while CP predicts six waypoints over a three-second horizon. The annotations combine model-assisted drafting, record-based computation, and human verification. Built upon CoVLM-Bench, we introduce CoVLM-Drive, a unified VLM baseline that directly uses paired views for both CDQA and CP. Experiments show that CDQA adaptation improves answer accuracy and that CoVLM-Drive reaches a lower FDE than the compared V2X planners; QA initialization and rationale supervision each reduce planning error. Together, CoVLM-Bench and CoVLM-Drive support the training and comparison of VLMs for cooperative scene understanding and planning.

Figures & tables

Figure 1: CDQA and CP in cooperative driving . The schematic contrasts ego-centric driving VLMs (A), conventional V2X perception and planning (B), and the paired-view tasks supported by CoVLM-Bench (C). The benchmark associates CDQA annotations and CP targets with the same real-world scenes; CoVLM-Drive provides a unified VLM baseline for CDQA and CP. Illustrative task examples are shown.

Benchmark	V2X	Src	QA	CoT	Plan	Metrics
DriveLM ( 2024 )	✗	✗	✓			Text/GPT
Reason2Drive ( 2024 )	✗	✗	✓	✓		Reasoning
DriveBench ( 2025 )	✗	✗	✓			Accuracy/Text/GPT
D2-V2X ( 2026 )	✓	✓	✓	✓		Grounding/Reasoning
V2X-Seq ( 2023 )	✓		✗	✗		Detection/Forecast
UniV2X ( 2025 )	✓		✗	✗	✓	Planning

Table 1: Benchmark positioning . V2X, Src, QA, CoT, and Plan denote paired V2X input, source records, question answering, reasoning annotations, and planning targets; our row evaluates CDQA and CP. Text denotes text-similarity metrics. Green checks mark explicit benchmark support, amber half-circles mark a related signal without full joint evaluation, and red crosses mark absent support.

Annotation layer	Content	Construction
L1: Object evidence	Object attributes, source views, and motion states	Label matching and track analysis
L2: Scene descriptions	Ego view, roadside view, and complementary observations	Model drafting from paired images and L1
L3: Question–answer annotations	Driving questions, answers, and reasons	Template-based drafting and record-derived answers
L4: Three-part rationales	Scene overview, V2X-aware critical objects, and decision reasoning	Model drafting with scene evidence

Table 2: Annotation layers in CoVLM-Bench .

Figure 2: Source-traceable evidence chain . Paired observations link to object attributes and a three-part rationale summary. Matching colors identify the shared object E1 and roadside-only objects I2 and I3 across views, records, and text. The rationale separates the scene overview, the V2X-aware critical objects, and the decision reasoning.

Figure 3: Planning path of CoVLM-Drive . The ego image, roadside image, and prompt enter one VLM backbone. The language decoder generates the three-part rationale; two structured heads predict the driving command and six ego-relative waypoints. The same language decoder supports CDQA responses. The dashed ego-status input is used by the optional variant marked with an asterisk in Table 4 .

Model	Valid (%)	Best absent (%)	Shuffled (%)	$\Delta$ (pp)
Zero-shot VLMs
Qwen3-VL-4B	20.4	14.2 (Ego)	13.7	+6.1
Qwen3-VL-8B	24.7	19.5 (Ego)	19.6	+5.2
Qwen3-VL-32B	51.1	17.8 (Blk)	17.5	+33.2
Qwen2.5-VL-3B	24.9	19.3 (Ego)	22.4	+5.6
Qwen2.5-VL-7B	19.4	12.5 (Ego)	13.1	+6.9

Table 3: CDQA results on 2,861 eligible items per condition from five objective tasks. The two CoVLM-Drive checkpoints use CP supervision or CDQA SFT, respectively. Best absent is the better of ego-only and blank (named in parentheses); Shuffled uses a mismatched roadside scene.

\Delta

is Valid minus Best absent; green / red mark gaps of one point or more.

QA initialization	Rationale supervision	FDE (m) $\downarrow$	CDQA accuracy (%) $\uparrow$
No	No	5.197	24.8
No	Yes	4.911	25.8
Yes	No	5.083	37.4

Table 5: QA initialization and rationale supervision . All rows use Qwen3-VL-8B, paired images, the same planning heads and CP training budget, and no ego-status input. QA initialization adds a prior CDQA adaptation stage. FDE uses 654 CP examples; CDQA accuracy is measured after CP training on 2,861 items. Bold marks the best result in each column; blue marks the standard CP configuration.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Result
Annotation coverage
Scene descriptions	2,196 / 2,196
CDQA annotations	35,136; 16 per frame
Three-part rationales	2,129 / 2,129
CP train / validation examples	1,475 / 654
Consistency checks

Appendix

Table 7: Annotation validation . All checks are independent of model training.

Figure 4: Roadside evidence coverage in CoVLM-Bench . (a) Source assignments for 54,171 annotated objects. (b) Roadside-only objects across 2,196 paired frames. Roadside-only evidence occurs in 98.3% of frames (median 9 objects). The panels summarize complementary object records available for annotation; source assignments follow the matching convention in Section 3 .

Template	Instantiated question	Answer form	Use
Prediction
Object behavior I	What is the likely behavior of $\langle O_{a}\rangle$ over the next three seconds?	Continue / Slow or stop / Accelerate / Change lane or turn
Object behavior II	What is the likely behavior of $\langle O_{b}\rangle$ over the next three seconds?	Continue / Slow or stop / Accelerate / Change lane or turn
Relative distance I	Will $\langle O_{a}\rangle$ be closer, farther, or at roughly the same distance after three seconds?	Closer / Farther / Same / Unable to determine	✓
Relative distance II	Will $\langle O_{b}\rangle$ be closer, farther, or at roughly the same distance after three seconds?	Closer / Farther / Same / Unable to determine	✓
Maneuver QA

Appendix

Table 8: Complete CDQA annotation catalog . All 16 question templates are instantiated for every paired frame. A blue check marks a closed task used in the reported CDQA score; a gray check marks a task evaluated as a source-role diagnostic outside that score. For presentation,

O_{a}

O_{b}

, and

O_{c}

denote generic object placeholders;

O_{r}

denotes the queried roadside-grounded object. The maneuver choice question is scored against the recorded ego maneuver (Appendix I ).

Template	Annotated answer	Annotated reason
Prediction
Object behavior I	continue	[I1] is going_straight with a smooth speed of 14.15 m/s, indicating continuous forward motion.
Maneuver QA
Unsafe maneuvers	change lane/turn; accelerate	Lane-turning or accelerating could bring ego into conflict with [I3] (stopped_at_signal) or [I2] (crossing/turning) in rear-left, which are only visible via V2X.
Attention priority	I1; I8; I9	[I1] is closest (37.8m) and in same direction, posing immediate rear-end risk; [I8] and [I9] are farther (122.9m, 124.4m) and less immediate.
Counterfactual priority	I8; I9	Without [I1], [I8] and [I9] are the closest rear objects, both at $\sim$ 123m, so they become equal priority.

Appendix

Table 9: Unscored CDQA annotations for one validation frame ( val_003446 , ten roadside-only objects). Answers, less their multiple-choice option letter, and reasons are reproduced verbatim from the released annotations;

[I\ast]

and

[E\ast]

are the released identifiers for infrastructure- and ego-observed objects, which the evaluation prompts replace with camera-neutral tokens, and motion labels such as going_straight come from the recorded motion-state vocabulary. These templates are released for training and separate analysis and are not part of the reported CDQA accuracy.

Panel of Figure 5 — Figure 5: CDQA evaluation prompt overview . Scene content and object catalogs are instantiated per sample. Field names and instructions are summarized for readability; Table 10 gives the question templates. The five objective fields define Overall accuracy, while the two source-role fields describe responses to roadside evidence.

Task	Question template	Answer options	Items
Ego-view prediction	Will $[O_{e}]$ be closer/farther in 3s?	A. Closer / B. Farther / C. Roughly the same / D. Unable to determine	412
Roadside-view prediction	Will $[O_{r}]$ be closer/farther in 3s?	A. Closer / B. Farther / C. Roughly the same / D. Unable to determine	487
Maneuver QA	Safest ego action?	A. Maintain current speed / B. Slow down / C. Brake firmly / D. Change lane / E. Accelerate to clear intersection	654
Roadside-only objects	Comparing the two annotated views, which neutral object tokens are visible only in the matched infrastructure view?	JSON list of $O_{k}$ tokens, or an empty list if none are supported	654
Nearest roadside distance	First compare the annotated views to identify infrastructure-only $O_{k}$ tokens, then use their current catalog distances: what is the nearest such distance?	Short numeric string such as “7.3 m”, or N/A if none are valid	654
Roadside constraint	Does matched infrastructure evidence reveal an additional object that constrains the ego’s safe maneuver in this scene?	Yes / No	654

Appendix

Table 10: CDQA inference templates . Wording and options are reproduced from the inference prompt, in which “infrastructure view” denotes the roadside view; items are counted per input condition. Blue rows are the five objective tasks included in Overall accuracy; gray rows are response diagnostics excluded from that average.

[O_{e}]

and

[O_{r}]

denote the queried neutral token grounded in the ego and roadside views. The maneuver QA reference answer is the recorded ego maneuver, not an independent safety judgment (Appendix I ).

Model	Valid	Ego-only	Blank	Shuffled
Overall
Qwen3-VL-4B	20.4	14.2	13.8	13.7
Qwen3-VL-8B	24.7	19.5	19.0	19.6
Qwen3-VL-32B	51.1	17.0	17.8	17.5
Qwen2.5-VL-3B	24.9	19.3	19.0	22.4
Qwen2.5-VL-7B	19.4	12.5	11.5	13.1

Appendix

Table 11: CDQA accuracy by roadside-input condition (%). All models use the same 2,861 eligible items. Prediction pools 899 items, maneuver QA has 654, and V2X evidence pools 1,308. The last block excludes maneuver QA, leaving 2,207 items. Adapted rows use task-specific checkpoints of CoVLM-Drive with Qwen3-VL-8B.

Figure 7: Roadside-input controls . The sample, ego image, prompt, target, and model remain fixed. Valid uses the synchronized roadside image; ego-only removes the second image; blank supplies a black frame in the same input format; and shuffled supplies the roadside image of another scene. Ego-only and blank test roadside availability, whereas shuffled tests scene correspondence.

Explore similar work

Jun 10, 2026

VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving

Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.

Thach Nguyen, Danhua Guo, Tom Lampo +2

Uber AV Labs

Autonomous Driving Benchmarks CoT Faithfulness VLMs for Autonomous Driving Autonomous Driving Vision-Language-Action Models Reasoning for VLA Models

Ask PDF

Apr 3, 2026

V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views

Multimodal large language models (MLLMs) have shown strong potential for autonomous driving, yet existing benchmarks remain largely ego-centric and therefore cannot systematically assess model performance in infrastructure-centric and cooperative driving conditions. In this work, we introduce V2X-QA, a real-world dataset and benchmark for evaluating MLLMs across vehicle-side, infrastructure-side, and cooperative viewpoints. V2X-QA is built around a view-decoupled evaluation protocol that enables controlled comparison under vehicle-only, infrastructure-only, and cooperative driving conditions within a unified multiple-choice question answering (MCQA) framework. The benchmark is organized into a twelve-task taxonomy spanning perception, prediction, and reasoning and planning, and is constructed through expert-verified MCQA annotation to enable fine-grained diagnosis of viewpoint-dependent capabilities. Benchmark results across ten representative state-of-the-art proprietary and open-source models show that viewpoint accessibility substantially affects performance, and infrastructure-side reasoning supports meaningful macroscopic traffic understanding. Results also indicate that cooperative reasoning remains challenging since it requires cross-view alignment and evidence integration rather than simply additional visual input. To address these challenges, we introduce V2X-MoE, a benchmark-aligned baseline with explicit view routing and viewpoint-specific LoRA experts. The strong performance of V2X-MoE further suggests that explicit viewpoint specialization is a promising direction for multi-view reasoning in autonomous driving. Overall, V2X-QA provides a foundation for studying multi-perspective reasoning, reliability, and cooperative physical intelligence in connected autonomous driving. The dataset and V2X-MoE resources are publicly available at: https://github.com/junwei0001/V2X-QA.

Junwei You, Pei Li, Zhuoyu Jiang +7

Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China +1

Autonomous Driving Benchmarks VLMs for Autonomous Driving Autonomous Driving Datasets Autonomous Driving Multimodal Large Language Models

Ask PDF

Apr 9, 2026

CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning

Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While vision-language models (VLMs) show strong general reasoning capabilities, their performance in safety-critical traffic scenarios remains insufficiently evaluated due to the ego-vehicle focus of existing benchmarks. To bridge this gap, we present CrashSight, a large-scale vision-language benchmark for roadway crash understanding using real-world roadside camera data. The dataset comprises 250 crash videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy. Tier 1 evaluates the visual grounding of scene context and involved parties, while Tier 2 probes higher-level reasoning, including crash mechanics, causal attribution, temporal progression, and post-crash outcomes. We benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal reasoning in safety-critical scenarios. We provide a detailed analysis of failure scenarios and discuss directions for improving VLM crash understanding. The benchmark provides a standardized evaluation framework for infrastructure-assisted perception in cooperative autonomous driving. The CrashSight benchmark, including the full dataset and code, is accessible at https://mcgrche.github.io/crashsight.

Rui Gan, Junyi Ma, Pei Li +4

University of Wisconsin–Madison · University of Wyoming · Columbia University

Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation Autonomous Driving Temporal Video Understanding

Ask PDF