Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving

Katharina Winter; Stefan Englmeier; Fabian B. Flohr

cs.CVSep 29, 2026

Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving

Authors: Katharina Winter, Stefan Englmeier, Fabian B. Flohr

Organizations: Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany

Abstract

Vision-Language Models are increasingly used in autonomous-driving systems, yet their ability to recover dynamic physical state from visual input remains insufficiently characterized. We study velocity understanding as a controlled diagnostic across three tasks: surrounding-agent speed, current ego speed, and short-horizon future ego-speed proposal. On nuScenes, we evaluate open-weight general-purpose and PhysicalAI VLMs, together with the driving-oriented Alpamayo-1.5 Vision-Language-Action model, using multiple input and output formulations. We combine verbal evaluation with temporal perturbations, counterfactual ego-speed hints and linear probes of hidden representations. The tasks exhibit distinct failure modes. Surrounding-agent speed is weakly encoded in an agent-specific form, whereas current ego speed is often internally accessible but poorly verbalized: continuous probes achieve 4.7-5.8 km/h MAE compared with 10.2-16.8 km/h MAE for verbal outputs. Multiple frames provide inconsistent verbal gains to single frame inputs, and frame order is rarely exploited. Under non-optimized QLoRA, task-specific adaptation improves both task-relevant latent speed representations and verbal readout, but continuous surrounding-agent speed estimation remains weak, while most future-speed gains survive frame shuffling, indicating limited temporal grounding. Driving specialized Alpamayo-1.5 shows stronger latent representations for surrounding-agent and future ego speed, while current ego-speed decodability is comparable and substantial probe-verbal gaps remain. Thus, driving specialization can strengthen motion representations but does not guarantee stronger encoding across both scene and ego states or reliable readout. The results show that plausible planning outputs do not necessarily imply reliable recovery or temporal grounding of the underlying dynamic state.

Figures & tables

Fig. 1: Overview of the evaluation framework and main findings for temporal-dynamic understanding in driving VLMs. Across three velocity tasks and controlled visual and ego-state input conditions, we assess output behavior and hidden representations. The results reveal weak agent-specific encoding, representation–verbalization gaps, limited temporal-order use, strong reliance on supplied ego state, and only partial correction through task-specific adaptation.

Model	Vision encoder	LLM backbone
Cosmos-Reason2-8B [ 36 ]	SigLIP-2 [ 37 ]	Qwen3-8B
Cosmos-Reason1-7B [ 38 ]	Qwen2.5-VL ViT	Qwen2.5-7B
Qwen2.5-VL-7B [ 20 ]	Qwen2.5-VL ViT	Qwen2.5-7B
Qwen3-VL-8B [ 21 ]	SigLIP-2 [ 37 ]	Qwen3-8B
InternVL2-8B [ 23 ]	InternViT-300M-448px	internlm2_5-7b-chat
InternVL2.5-8B [ 22 ]	InternViT-300M-448px-V2.5	internlm2_5-7b-chat

TABLE I: Vision encoders and language-model backbones of the evaluated open source VLMs.

	T1 surrounding vehicle				T2 ego current				T3 ego future
Model	1F	5F	1F+S	5F+S	1F	5F	1F+S	5F+S	1F	5F	1F+S	5F+S
Cosmos-R2	22.1	21.8	27.3	29.1	22.9	36.2	65.2	63.4	46.1	44.5	66.1	61.0
Cosmos-R1	43.3	42.2	44.7	44.7	31.4	33.1	57.2	62.0	45.6	55.1	64.8	72.7
Qwen2.5-VL	33.5	39.1	41.6	43.2	22.7	27.8	50.2	64.8	35.3	35.3	52.9	54.3
Qwen3-VL	42.0	30.5	37.8	39.1	40.4	40.9	78.8	82.0	51.8	51.4	61.1	69.4
LLaVA-Video	21.6	14.2	18.4	20.3	23.0	25.3	37.2	47.4	14.6	14.1	15.2	14.2

TABLE II: Three-class macro-F1 (%) for surrounding-agent speed (T1), current ego speed (T2), and 3s-ahead ego-speed proposal (T3) under single-frame (1F), five-frame (5F), and ego-speed-hint (+S) conditions. Visual-only performance remains strongly task- and model-dependent, with no consistent benefit from five-frame input. Ego-speed hints yield substantial gains for T2 and T3. Only MiniCPM-V † emitted unparsable responses, for 3.9% of samples on T1, 5F+S and

\leq

0.3% for other conditions.

Purpose	Split	Scenes	Samples
Benchmark evaluation	Val.	150	T1: 4,429; T2: 5,419; T3: 4,519
Probe training	Train	547	6,000 ( $\approx$ 2,000/class;joint-stratified)
QLoRA adaptation	Train	560	20,275

TABLE III: Use of the nuScenes training and validation splits across the study. Training and validation scenes are disjoint. Sample counts are a subset of the original split reflecting valid temporal history, target visibility, and future trajectories, as well as balanced sampling for probing and adaptation.

Fig. 2: Qualitative T1 failure case selected to illustrate errors in agent-speed estimation. The five input frames, ground-truth lead-vehicle speed, and corresponding model predictions are shown. Only Cosmos-R1 approximates the ground-truth speed in the regression output, with a 21% relative error. ∗ LLaVA-1.5 receives only the current frame.

Fig. 3: Macro-F1 of correct and counterfactual ego-speed hints on T2 and T3. Counterfactual hints reduce performance below the stochastic reference, showing strong reliance on supplied ego state without reliable visual verification. Hollow and filled markers denote single-frame and five-frame input, respectively.

Fig. 4: Predicted class distributions for T1–T3 under five-frame visual-only input. GT denotes the empirical distribution and violin width prediction frequency. Biases vary across models and tasks.

	T1 surrounding		T2 ego current		T3 ego future
Model	1F	5F	1F	5F	1F	5F
Cosmos-R2	60.1	64.7	25.9	76.6	46.5	74.2
Cosmos-R1	35.7	35.2	16.3	15.8	32.8	22.6
Qwen2.5-VL	37.4	37.4	16.6	27.6	33.3	40.6
Qwen3-VL	62.6	46.2	56.0	67.9	59.3	67.8
LLaVA-Video	48.6	36.6	15.2	15.0	14.0	14.0

TABLE IV: Binary stopped-versus-moving macro-F1 (%) for the target surrounding vehicle (T1), the current ego state (T2), and the proposed ego state 3s ahead (T3). Results are reported for the visual-only single-frame (1F) and five-frame (5F) conditions. Reducing the task to stopped-versus-moving does not eliminate output collapse or the inconsistent benefit of additional frames.

	T1 surrounding		T2 ego current		T3 ego future
Model	1F	5F	1F	5F	1F	5F
Cosmos-R2	10.8	10.5	19.0	14.2	15.2	12.0
Cosmos-R1	17.6	15.4	14.9	10.2	13.4	14.3
Qwen2.5-VL	11.7	17.0	18.4	12.2	12.8	14.2
Qwen3-VL	15.6	15.5	19.0	10.4	15.5	10.6
LLaVA-Video	11.6	11.9	19.1	18.5	15.1	14.5

TABLE V: Mean absolute error (MAE, km/h; lower is better) for numerical estimation of T1–T3 under 1F and 5F input. Continuous estimation remains challenging, with no consistent benefit from five-frame input. The optimal constant reference predicts the evaluation-set median for every sample. MAE is computed for parsable numerical outputs. Parsing rates are 99.8% for Phi-3.5-V on T1–5F and 98.5%, 77.5%, and 82.6% for MiniCPM-V † on T2–1F, T2–5F, and T3–1F, respectively.

Fig. 5: Macro-F1 for single-frame, ordered five-frame, and shuffled history input with fixed current frame on the main evaluation pools. Additional frames are neutral or detrimental in most settings; where gains occur, they are often preserved under shuffling, consistent with a bag-of-frames use of temporal context rather than strong reliance on chronological order.

	T1 – lead speed			T2 – ego speed			T3 – future speed
Model	1F	5F	1F+S	1F	5F	1F+S	1F	5F	1F+S
Cosmos-R1	35.6 ( -8 )	49.7 ( +8 )	43.4 ( -1 )	59.6 ( +28 )	82.3 ( +49 )	96.6 ( +39 )	57.4 ( +12 )	66.2 ( +11 )	69.2 ( +4 )
Cosmos-R2	36.7 ( +15 )	44.4 ( +23 )	38.8 ( +11 )	62.3 ( +39 )	75.3 ( +39 )	95.4 ( +30 )	59.5 ( +13 )	64.6 ( +20 )	68.6 ( +3 )
Qwen2.5-VL	38.9 ( +5 )	29.1 ( -10 )	41.9 ( +0 )	60.5 ( +38 )	77.4 ( +50 )	96.9 ( +47 )	56.5 ( +21 )	67.5 ( +32 )	73.3 ( +20 )
Qwen3-VL	25.6 ( -16 )	41.1 ( +11 )	21.8 ( -16 )	61.3 ( +21 )	77.3 ( +36 )	96.9 ( +18 )	59.3 ( +7 )	61.8 ( +10 )	68.5 ( +7 )
LLaVA-Video	38.1 ( +17 )	42.5 ( +28 )	41.9 ( +24 )	55.8 ( +33 )	74.1 ( +49 )	96.9 ( +60 )	54.1 ( +40 )	64.8 ( +51 )	73.1 ( +58 )

TABLE VI: Last-layer probe macro-F1 (%) for T1–T3. Parentheses show probe minus verbal performance in Table II (pp). T2 and T3 are more strongly decodable from hidden representations than reflected in verbal outputs, whereas T1 shows no consistent probe advantage.

Fig. 6: Layerwise probe macro-F1 for T2 and T3 under five-frame input. Dashed and solid lines denote vision-encoder and LLM representations, respectively; circles mark peak layers. Current ego speed can already be strongly represented in visual features for some models, while future-speed-related information tends to emerge later in the LLM; in both cases, peak decodability often precedes the final layer.

Fig. 7: Ground-truth versus predicted speed for numerical verbal outputs and last-layer regression probes under five-frame input. Panels report MAE and Spearman’s $\rho$ ; the dashed line denotes perfect agreement and gray lines the $\pm 30\%$ range. Probes recover substantially more precise speed information than verbal outputs, particularly for T2.

Fig. 8: Query-conditioned decoding of ego and surrounding-agent speed under five-frame input across six models. Probes decode both quantities from representations produced by either an ego-speed or agent-speed query. The ego-query representation is consistently more informative for both targets, while explicitly querying the marked vehicle does not improve agent-speed decodability.

Model	Task	Setting	Probe	Verbal
Qwen3-VL	T1	Zero-shot	41.1	30.5
		LoRA-full	69.9	71.0
	T2	Zero-shot	77.3	40.9
		LoRA-full	89.0	90.1
		LoRA LLM-only	87.5	89.0
		LoRA VE-only	85.9	87.6

TABLE VII: Probe and verbal three-class macro-F1 (%) before and after task-specific QLoRA, with adapter-placement ablations on T2. Adaptation improves both representation and readout in several settings, but not necessarily to the same extent. For T2, LLM-only adaptation is competitive with full adaptation. Training uses target-balanced weighted sampling, and probes use last-layer five-frame representations.

Fig. 9: Continuous speed estimation before and after task-specific full-LoRA adaptation of Qwen3-VL under five-frame input. Columns show verbal outputs and last-layer regression probes for T1–T3; rows show zero-shot and adapted models. The dashed line denotes perfect agreement and gray lines the $\pm 30\%$ relative-error range. Adaptation nearly closes the continuous representation–verbalization gap for T2 and strongly improves T3, while T1 remains substantially harder even at the representation level.

		Zero-shot		LoRA-full
Task	Readout	Ordered	Shuffled	Ordered	Shuffled
T2	Verbal	40.9	40.1	89.5	86.4
	Probe	77.3	76.5	89.0	83.6
T3	Verbal	51.4	51.3	72.4	72.2
	Probe	66.5	61.5	71.0	69.2

TABLE VIII: Verbal and probe three-class macro-F1 (%) for Qwen3-VL with ordered and randomly shuffled history inputs and fixed current frame before and after task-specific full-LoRA adaptation. T2 shows a larger ordered–shuffled gap after adaptation. For T3, adaptation improves overall performance while the ordered–shuffled gap remains small or decreases, indicating that most of the future-speed gain does not depend on chronological frame order.

Main finding	Evidence	Potential mitigation / related finding
Weak agent-specific encoding (T1). Surrounding-agent speed is not reliably encoded in an agent-specific representation by the VLM foundation models.	Section V - V-A (Table VI ); Section V - V-D (Fig. 8 ); Appendix A (Tables X and XI ); Section VI (Table VII )	Agent-centric temporal or relative-motion supervision. Alpamayo-1.5’s stronger zero-shot T1 probe and the substantial probe gain of adapted Qwen3-VL suggest that driving- and task-specific supervision can strengthen agent-speed representations (Section VI , Table VII ), while continuous T1 estimation on our non-optimized training remains weak (Fig. 9 ).
Representation–verbalization gap (T2). Current ego speed is strongly linearly accessible in hidden states but only incompletely reflected in verbal outputs.	Section IV - IV-A (Table II ); Section V - V-A (Table VI ); Section V - V-C (Fig. 7 ); Section VI (Table VII )	Improve readout while explicitly supervising temporal motion cues. Task-specific adaptation largely closes the readout gap and increases temporal-order sensitivity (Section VI , Table VII ; Fig. 9 ; Table VIII ).
Planning performance can mask weak perceptual grounding (T3). T3 predictions and representations can achieve nontrivial target agreement despite limited sensitivity to chronological motion evidence, yet improve strongly when accurate current ego speed is supplied. Thus, planning scores alone can understate the importance of reliable dynamic-state perception.	Section IV - IV-A (Table II ); Section IV - IV-B (Fig. 3 ) Section IV - IV-E (Fig. 5 );	Evaluate planning jointly with its perceptual prerequisites. Improving explicit recovery and verification of the current dynamic state may strengthen planning-related predictions, while target accuracy alone should not be interpreted as evidence of perceptual or temporal grounding.
Limited use of chronological history. Ordered and shuffled five-frame inputs perform similarly in most zero-shot settings, and rare gains from additional history often survive shuffling.	Section IV - IV-E (Fig. 5 )	Explicit temporal supervision. Task-specific adaptation produces limited evidence of increased order sensitivity (Section VI , Table VIII ), leaving temporal-order or motion supervision as an open direction.
Prompt-supplied ego state is integrated zero-shot and dominates visual cues. Without task-specific training on ego-state inputs, VLMs already use numerical ego-speed hints to improve T2/T3. Counterfactual hints strongly mislead both tasks, showing that prompt-supplied state is readily incorporated but weakly cross-checked against visual evidence.	Section IV - IV-A (Table II ); Section IV - IV-B (Fig. 3 )	Conflict-aware multimodal fusion and uncertainty calibration. Inconsistencies between prompt-supplied state and visually inferred state should be checked and biased prompts should be rejected when inconsistent. Counterfactual hints provide a direct robustness test for such failures.
Formulation-sensitive and biased verbal outputs. Paired categorical and metric predictions can contradict, and several models exhibit persistent class preferences or output collapse. Thus, different output interfaces do not reliably expose a consistent speed estimate.	Section IV - IV-C (paired cross-formulation analysis; Tables II , IV , and V ; Fig. 4 ); Section VI	Cross-format supervision with shared state representations. Task-specific adaptation improves the supervised format but transfers poorly across formulations. Cross-format training and shared state representations can be directions for further study.

TABLE IX: Summary of the main findings, supporting evidence, and potential mitigations. Related findings indicate results that partially improve or qualify the respective behavior.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	5F current	5F all	$\boldsymbol{\Delta}$
Cosmos-Reason2	21.8	22.9	+1.1
Cosmos-Reason1	42.2	43.2	+1.0
Qwen2.5-VL	39.1	38.5	-0.6
Qwen3-VL	30.5	26.1	-4.4
LLaVA-Video	14.2	13.6	+0.6
InternVL2.5	41.4	41.4	+0.0

Appendix

TABLE X: T1 speed macro-F1 (%) when the target is marked only in the current frame or in all five frames. Marking all frames provides no consistent improvement. † MiniCPM-V produces unparsable outputs for 0.8% for current marked frame and 1.4% for all marked frames.

\Delta

denotes all-frame minus current-frame macro-F1.

Model	Accuracy	Macro-F1
Cosmos-Reason2	76.2	34.0
Cosmos-Reason1	69.6	48.0
Qwen2.5-VL	73.6	49.3
Qwen3-VL	78.5	43.8
LLaVA-Video	75.8	36.7
InternVL2.5	70.9	44.8

Appendix

TABLE XI: Vehicle-type recognition for the marked T1 target under five-frame input. Type recognition is substantially stronger than T1 speed estimation, indicating that weak agent-speed performance cannot be explained by target recognition alone. Unparsable type responses are scored as errors. Cosmos-Reason1 (9.1%), MiniCPM-V (3.5%), and Qwen2.5-VL (3.1%) had non-negligible unparsable rates; all other models

\leq

0.1%.

	T1 surrounding		T2 ego current		T3 ego future
Model	1F	5F	1F	5F	1F	5F
Cosmos-R2	17.4	14.5	22.7	18.1	19.4	15.7
Cosmos-R1	23.9	19.2	18.7	15.5	18.1	26.5
Qwen2.5-VL	18.2	21.1	22.0	15.9	16.6	17.9
Qwen3-VL	22.3	19.8	22.7	13.7	19.5	14.0
LLaVA-Video	18.7	18.6	22.7	22.3	19.2	18.7

Appendix

TABLE XII: Root mean squared error (RMSE) in km/h for numerical speed estimation on T1–T3 under single-frame (1F) and five-frame (5F) input. Lower is better. LLaVA-1.5 receives only the current frame. Parsing rate is equivalent to MAE results: 99.8% for Phi-3.5-V on T1–5F and for MiniCPM-V † on T2–1F (98.5%), T2–5F (77.5%), and T3–1F (82.6%).

Condition	Target speed	Ego speed
1F	0.27	0.29
1F+S	0.32	0.77
5F+S	0.34	0.84

Appendix

TABLE XIII: Median Spearman’s

\rho

across the ten models (excluding LLaVA-1.5 from the 5F+S condition) between T1 numerical predictions and the target-vehicle or ego speed. Supplying ego speed shifts predictions strongly toward ego state rather than the queried vehicle.

Explore similar work

May 22, 2026cs.CV

DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving

Spatiotemporal intelligence in autonomous driving (AD) requires an agent to integrate multi-view observations into a coherent scene representation, maintain object continuity across viewpoints and time, and reason about spatial relations, interactions, and future dynamics. However, existing AD vision-language benchmarks largely focus on single-view, static, ego-centric, or single-source question answering, leaving it unclear whether current Vision-Language Models (VLMs) can truly construct and reason over dynamic driving scenes. We introduce DriveSpatial, a benchmark of 15.6K human-verified QA pairs across 20 tasks from five large-scale AD datasets. DriveSpatial evaluates four abilities: Cognitive Scene Construction, Multi-view Relational Understanding, Temporal Reasoning, and Generalization. Unlike prior benchmarks, DriveSpatial is generated from a dynamic multi-relational scene graph that encodes object states, spatial relations, interactions, camera visibility, and temporal correspondences, enabling QA pairs that enforce genuine cross-view and spatiotemporal reasoning. Evaluating 15 representative VLMs reveals a substantial human-model gap: the strongest model trails humans by 28.4 points, with Cognitive Scene Construction emerging as the key bottleneck. Further diagnostics show that language-only prompting is insufficient, while explicit BEV grounding consistently improves performance. These results suggest that current VLMs lack the scene-construction ability needed for reliable spatiotemporal driving intelligence. DriveSpatial and its construction pipeline will be released to support future research.

Hao Vo, Khoa Vo, Phu Loc Nguyen +10

University of Arkansas, USA · Google Research, Google · University of Liverpool, UK +1

Autonomous Driving Spatio-Temporal Reasoning Semantic Scene Understanding

Ask PDF

May 1, 2025cs.RO

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.

Zhijie Qiao, Haowei Li, Zhong Cao +1

Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA

Autonomous Driving Recent Vision-Language Models Longitudinal Data

Ask PDF

Dec 11, 2025cs.CV

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

End-to-end autonomous driving methods built on vision language models (VLMs) have undergone rapid development driven by their universal visual understanding and strong reasoning capabilities obtained from the large-scale pretraining. However, we find that current VLMs struggle to understand fine-grained 3D spatial relationships which is a fundamental requirement for systems interacting with the physical world. To address this issue, we propose SpaceDrive, a spatial-aware VLM-based driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens, enabling joint reasoning over semantic and spatial representations. SpaceDrive employs a universal positional encoder to all 3D coordinates derived from multi-view depth estimation, historical ego-states, and text prompts. These 3D PEs are first superimposed to augment the corresponding 2D visual tokens. Meanwhile, they serve as a task-agnostic coordinate representation, replacing the digit-wise numerical tokens as both inputs and outputs for the VLM. This mechanism enables the model to better index specific visual semantics in spatial reasoning and directly regress trajectory coordinates rather than generating digit-by-digit, thereby enhancing planning accuracy. Extensive experiments validate that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive closed-loop benchmark over existing VLM-based methods. Code is available at: https://github.com/zhenghao2519/SpaceDrive.

Peizheng Li, Zhenghao Zhang, David Holtz +6

Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4

Autonomous Driving 3D Spatial Reasoning Autonomous Hard Drive Disassembly Spatial Reasoning

Ask PDF