StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving

Yuan Gao; Dengyuan Hua; Mattia Piccinini; Finn Rasmus Schäfer; Korbinian Moller; Lin Li; Johannes Betz

cs.ROMar 10, 2026

StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving

Authors: Yuan Gao, Dengyuan Hua, Mattia Piccinini, Finn Rasmus Schäfer, Korbinian Moller, Lin Li, Johannes Betz

Organizations: Y. Gao, D. Hua, M. Piccinini, F. Schäfer, K. Moller, J. Betz are with the Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI) · L. Li is with the School of Mechanical and Aerospace Engineering, Nanyang Technological University

Abstract

Vision Language Models (VLMs) bridge visual perception and linguistic reasoning. In Autonomous Driving (AD), this synergy has enabled Vision Language Action (VLA) models, which translate high-level multimodal understanding into driving behaviors, typically represented as future trajectories. However, existing VLA models mainly generate generic collision-free trajectories. Beyond collision avoidance, adapting to diverse driving styles (e.g., sporty, comfortable) is essential for personalized driving. Moreover, many methods treat trajectory generation as naive token prediction, which can produce kinematically infeasible actions. To address these limitations, we present StyleVLA, a physics-informed VLA framework for generating diverse and physically plausible driving behaviors. We introduce a hybrid loss that combines a kinematic consistency constraint with a continuous regression head to improve trajectory feasibility. To train StyleVLA, built on Qwen3-VL-4B, we construct a large-scale instruction dataset with over 1.2k scenarios, 76k Bird's Eye View (BEV) samples, and 42k First Person View (FPV) samples, with ground-truth trajectories for five driving styles and natural-language instructions. Experiments show that our 4B-parameter StyleVLA significantly outperforms proprietary models (e.g., Gemini-3-Pro) and state-of-the-art VLA models. Using a composite driving score measuring success rate, physical feasibility, and style adherence, StyleVLA achieves 0.55 on BEV and 0.51 on FPV, versus 0.32 and 0.35 for Gemini-3-Pro. These results show that a specialized, physics-informed, lightweight model can surpass closed-source models on domain-specific tasks.

Figures & tables

Fig. 1: Concept of StyleVLA: Enabling driving style-aware trajectory generation via VLA model. Our framework yields diverse driving styles (Default, Balanced, Comfort, Sporty, Safety) in response to user instructions.

Fig. 2: Overview of the StyleVLA framework. Top (Dataset Construction): A motion planner generates style-specific ground-truth trajectories to create multimodal instruction samples. Instruction Dataset Generation: Details the instruction generation process and 3D scenario replay in CARLA. Bottom (Fine-tuning Architecture): The model predicts trajectory tokens using only an LLM head conditioned on visual context and language prompts. During training, an auxiliary MLP decoder maps the predicted tokens to continuous kinematic trajectories for physics-informed supervision. Training uses a physics-informed hybrid loss ( $\mathcal{L}_{\mathrm{total}}$ ) combining cross-entropy ( $\mathcal{L}_{\mathrm{ce}}$ ), regression ( $\mathcal{L}_{\mathrm{reg}}$ ), and kinematic consistency ( $\mathcal{L}_{\mathrm{pikc}}$ ). Trajectory Generation: Shows the model’s application in both 2D BEV and 3D FPV domains.

Cost Term	Comfort	Balanced	Sporty	Safety	Default
Kinematic Constraints ( $\mathbf{w}_{\mathrm{kin}}$ )
Longitudinal Jerk ( $w_{\mathrm{j,lon}}$ )	0.80	0.50	0.25	0.40	0.20
Lateral Jerk ( $w_{\mathrm{j,lat}}$ )	0.80	0.50	0.25	0.40	0.20
Velocity Offset ( $w_{\mathrm{v}}$ )	0.30	0.60	1.00	0.30	1.00
Distance to obstacles ( $w_{\mathrm{obs}}$ )	0.30	0.80	0.60	2.00	0.00
External Perception ( $\mathbf{w}_{\mathrm{ext}}$ )

TABLE I: Weights across Driving Styles, for the cost function ( 2 ) of the Frentix motion planner used for dataset generation.

Style Label	Samples	Avg Velocity	RMS Accel	RMS Jerk	Path Length
	(Count / %)	( $m/s$ )	( $m/s^{2}$ )	( $m/s^{3}$ )	( $m$ )
Balanced	14,102 (18.5%)	7.15	0.588	0.750	24.44
Comfort	13,766 (18.1%)	7.21	0.585	0.727	24.53
Default	17,684 (23.3%)	6.80	0.486	0.794	23.31
Sporty	14,790 (19.5%)	7.32	0.558	0.780	25.13
Safety	15,688 (20.6%)	6.39	0.583	0.746	21.44

TABLE II: Mean Kinematic Features and Distribution of Filtered Trajectories by Style

Panel of Figure 5 — Fig. 3: Example training dynamics of StyleVLA fine-tuning on the FPV instruction dataset. Top: loss terms ( $\mathcal{L}_{\mathrm{total}}$ , $\mathcal{L}_{\mathrm{ce}}$ , $\mathcal{L}_{\mathrm{reg}}$ , $\mathcal{L}_{\mathrm{pikc}}$ ). Bottom: learned log-variance parameters ( $w_{\mathrm{ce}}$ , $w_{\mathrm{reg}}$ ) that yield adaptive precision weights via $\exp(-w)$ .

Group	Value
Regression Loss Weights
$(\boldsymbol{w}_{\mathrm{reg}},w_{\mathrm{pikc}})$	$(2.0,\,2.0,\,0.5,\,0.5,\,0.5,\,1.5)$
Training Hyperparameters ( FPV )
Backbone	Qwen3-VL-4B-Instruct
Quantization	QLoRA, 4-bit
LoRA $(r,\alpha,\mathrm{dropout})$	$(256,\,512,\,0.05)$

TABLE III: Hyperparameters used in fine-tuning

Fig. 4: Qualitative comparison of style-conditioned trajectory generation under five driving styles (Default, Balanced, Comfort, Sporty, Safety). We visualize the goal, ground truth, and predicted trajectories from pretrained VLMs and SOTA baselines (see legend). ”*” failed to generate trajectories.

Configuration	ADE (m) $\downarrow$	FDE (m) $\downarrow$	PSR $\uparrow$	Heading MAE (rad) $\downarrow$
Impact of Training Data Scaling (with Physics-Informed Hybrid Loss)
Small (4.5k)	2.08	5.43	20.60%	0.073
Medium (20k)	1.51	3.92	27.14%	0.046
Large (40k)	1.47	3.81	29.37%	0.044
Standard (50k)	1.17	3.06	33.19%	0.035
Impact of Loss Components (50k Training dataset)

TABLE IV: Ablation on BEV StyleVLA (Qwen2.5-VL-7B, 3 s horizon). Scaling data and adding physics-informed hybrid loss consistently improves generalization

Model	Score $\uparrow$	PSR $\uparrow$	MR $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	KCE $\downarrow$	Time $\downarrow$
	(0-1)	(ADE $<1$ m)	(FDE $>2$ m)	(m)	(m)	(m)	(s)
Open Source Models
Qwen3-VL-4B	0.00	0.00%	100.0%	-	-	-	2.00
Qwen2.5-VL-7B	0.00	0.00%	100.0%	-	-	-	3.43
InternVL3-9B	0.00	0.00%	100.0%	12.63	25.89	3.72	5.77
Proprietary Models

TABLE V: Benchmarking across VLMs on BEV domain (Zero-Shot)

Model	Score $\uparrow$	PSR $\uparrow$	MR $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	KCE $\downarrow$	Time $\downarrow$
	(0-1)	(ADE $<1$ m)	(FDE $>2$ m)	(m)	(m)	(m)	(s)
Open Source Baselines
Qwen3-VL-4B (Base)	0.00	-	-	-	-	-	4.97
Proprietary Models
Gemini 2.5 Flash	0.12	9.04%	87.21%	9.39	18.21	3.42	1.80
Gemini 2.5 Pro	0.29	16.63%	70.74%	2.23	5.76	0.06	35.48

TABLE VI: Benchmarking across VLMs on FPV domain (Zero-Shot)

Explore similar work

Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous Driving VLM Distillation
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
Unifying Language-Action Understanding and Generation for Autonomous Driving
Mar 2, 2026Xinyang Wang, Qian Liu, Wenjie Ding +7Vision-Language-Action Models End-to-End Autonomous Driving
State Key Lab of CAD&CG, Zhejiang University · Li Auto
AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding
Oct 12, 2026Yuqi Ye, Caigao Jiang, Zhuonan Wang +12Vision-Language-Action Models Autonomous Driving Planning
Peking University · XPeng Motors · Fudan University

Jul 2, 2026J/K move · Enter open · S save

Teaching Vision-Language-Action Models What to See and Where to Look

Yuguang Yang, Canyu Chen, Zhewen Tan, Yizhi Wang+9

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University+5

End-to-End Autonomous Driving VLM Distillation Vision-Language-Action Models Autonomous Driving Planning Vehicle Trajectory Prediction

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Abstract

Explore similar work

Teaching Vision-Language-Action Models What to See and Where to Look

Unifying Language-Action Understanding and Generation for Autonomous Driving

AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding