How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study

Roberto Brusnicki; Mattia Piccinini; Johannes Betz

cs.CVApr 8, 2026

How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study

Authors: Roberto Brusnicki, Mattia Piccinini, Johannes Betz

Organizations: Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, Munich, Germany

Abstract

Vision-Language Models (VLMs) are increasingly proposed for autonomous driving tasks, yet their performance on sequential driving scenes remains poorly characterized, particularly regarding how input configurations affect their capabilities. We introduce VENUSS (VLM Evaluation oN Understanding Sequential Scenes), a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes, establishing baselines for future research. Building upon existing datasets, VENUSS extracts temporal sequences from driving videos, and generates structured evaluations across custom categories. By comparing 25+ existing VLMs across 2,600+ scenarios, we reveal how even top models achieve only 57% accuracy, not matching human performance under similar constraints (65%) and exposing significant capability gaps. Our analysis shows that VLMs excel with static object detection but struggle with understanding vehicle dynamics and temporal relations. VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations - resolution, frame count, temporal intervals, spatial layouts, and presentation modes - affect performance on sequential driving scenes. Supplementary material available at https://TUM-AVS.github.io/VENUSS/.

Figures & tables

Fig. 1 : VENUSS framework overview. Starting from driving datasets, VENUSS generates structured evaluation data with controlled variations in image count, timing, resolution, layout, and presentation mode. It evaluates both VLMs and humans on identical tasks across custom categories, identifies optimal input configurations per model, and establishes performance baselines.

Fig. 2 : Visualization of all CoVLA textual descriptions automatically categorized by VENUSS. The framework identified seven categories from the natural language descriptions: motion types (blue), velocity descriptors (yellow), directional behaviors (orange), acceleration patterns (purple), following behavior (dark green), traffic light conditions (light red), and road curvature detection (gray).

Fig. 3 : Simplified example of the human evaluation questionnaire with the seven-category format (Sec. III-C ). The interface presents sequential driving images with temporal intervals, and seven multiple-choice questions for the corresponding seven categories. The final answer key (bottom right) concatenates the responses for comparison with ground truth annotations.

Phase	Target Variable	Varied Parameters	Fixed Parameters
1	Resolution	6 resolution levels	2×2 grid layout, 200ms intervals, collage mode
2	Frame Count	1-10 frames	Level 1 & Level 6 resolution, 200ms intervals, 1×N layout
3	Temporal	100-1000ms intervals	Level 1 & Level 6 resolution, 4 frames, 1×4 layout
4	Spatial	All possible grid layouts	Level 1 & Level 6 resolution, 200ms intervals
5	Presentation	Collage, separate, batch modes	Level 1 & Level 6 resolution, 2×2 grid, 200ms intervals

TABLE I : Five-phase experimental protocol for systematic VLM evaluation.

Model / Human	Overall Metrics				F1 score for each category							Avg Query
	Acc.	Prec.	Rec.	F1	Mot.	Dir.	Spd.	Fol.	Accel.	TL	Crv.	Time (s)
evaluator-1 (collage)	0.63	0.66	0.63	0.64	0.88	0.56	0.32	0.62	0.40	0.89	0.69	N/A
evaluator-2 (collage)	0.57	0.61	0.57	0.58	0.87	0.50	0.41	0.64	0.37	0.76	0.40	N/A
evaluator-3 (collage)	0.54	0.59	0.54	0.55	0.82	0.27	0.22	0.60	0.27	0.76	0.68	N/A
evaluator-4 (gif)	0.65	0.68	0.65	0.66	0.92	0.42	0.49	0.80	0.27	0.85	0.74	N/A
evaluator-5 (gif)	0.63	0.66	0.63	0.64	0.88	0.56	0.32	0.62	0.40	0.89	0.69	N/A

TABLE II : VLMs performance evaluation, with the unified metric score of eq. ( 3 ) (columns 2-5), F1 scores for our seven categories (motion state, direction, speed, following behavior, acceleration, traffic light detection, road curvature: columns 6-12), and query times.

Fig. 4 : Performance analysis across 4 dimensions as described in IV-A . (a) Performance by resolution level (1-6), (b) Performance by time interval (100ms increments, 1-10), (c) Performance by number of images (1-10), and (d) Performance by presentation mode (b=batch, c=collage, s=separate). Horizontal lines show human baselines: green for collage-based evaluation, blue for GIF-based evaluation. Box-and-whisker plots summarize the performance distribution of all 25+ evaluated VLMs for each configuration, with red dots indicating outliers.

Fig. 5 : Performance comparison across different grid formats for the image sequences. Specific grid layouts demonstrate better performance for temporal understanding tasks.

Explore similar work

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
May 1, 2025Zhijie Qiao, Haowei Li, Zhong Cao +1VLM Evaluation Autonomous Driving Benchmarks
Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA
DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
May 22, 2026Hao Vo, Khoa Vo, Phu Loc Nguyen +10VLM Evaluation Autonomous Driving Datasets
University of Arkansas, USA · Google Research, Google · University of Liverpool, UK +1
Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models
Apr 1, 2026Xiaosong Jia, Yuqian Shao, Zhenjie Yang +3Autonomous Driving Benchmarks VLMs for Autonomous Driving
Institute of Trustworthy Embodied AI, Fudan University and Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, 200441, China. · School of Computer Science & School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, 200240, China.

May 1, 2025J/K move · Enter open · S save

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

Zhijie Qiao, Haowei Li, Zhong Cao, Henry X. Liu

Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA

VLM Evaluation Autonomous Driving Benchmarks VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving

Open paper AskPDF arXiv

Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.

Abstract

Explore similar work

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving

Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models