StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving

Yuan Gao; Dengyuan Hua; Mattia Piccinini; Finn Rasmus Schäfer; Korbinian Moller; Lin Li; Johannes Betz

cs.ROMar 10, 2026

StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving

Authors: Yuan Gao, Dengyuan Hua, Mattia Piccinini, Finn Rasmus Schäfer, Korbinian Moller, Lin Li, Johannes Betz

Organizations: Y. Gao, D. Hua, M. Piccinini, F. Schäfer, K. Moller, J. Betz are with the Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI) · L. Li is with the School of Mechanical and Aerospace Engineering, Nanyang Technological University

Abstract

Vision Language Models (VLMs) bridge visual perception and linguistic reasoning. In Autonomous Driving (AD), this synergy has enabled Vision Language Action (VLA) models, which translate high-level multimodal understanding into driving behaviors, typically represented as future trajectories. However, existing VLA models mainly generate generic collision-free trajectories. Beyond collision avoidance, adapting to diverse driving styles (e.g., sporty, comfortable) is essential for personalized driving. Moreover, many methods treat trajectory generation as naive token prediction, which can produce kinematically infeasible actions. To address these limitations, we present StyleVLA, a physics-informed VLA framework for generating diverse and physically plausible driving behaviors. We introduce a hybrid loss that combines a kinematic consistency constraint with a continuous regression head to improve trajectory feasibility. To train StyleVLA, built on Qwen3-VL-4B, we construct a large-scale instruction dataset with over 1.2k scenarios, 76k Bird's Eye View (BEV) samples, and 42k First Person View (FPV) samples, with ground-truth trajectories for five driving styles and natural-language instructions. Experiments show that our 4B-parameter StyleVLA significantly outperforms proprietary models (e.g., Gemini-3-Pro) and state-of-the-art VLA models. Using a composite driving score measuring success rate, physical feasibility, and style adherence, StyleVLA achieves 0.55 on BEV and 0.51 on FPV, versus 0.32 and 0.35 for Gemini-3-Pro. These results show that a specialized, physics-informed, lightweight model can surpass closed-source models on domain-specific tasks.

Figures & tables

Fig. 1: Concept of StyleVLA: Enabling driving style-aware trajectory generation via VLA model. Our framework yields diverse driving styles (Default, Balanced, Comfort, Sporty, Safety) in response to user instructions.

Fig. 2: Overview of the StyleVLA framework. Top (Dataset Construction): A motion planner generates style-specific ground-truth trajectories to create multimodal instruction samples. Instruction Dataset Generation: Details the instruction generation process and 3D scenario replay in CARLA. Bottom (Fine-tuning Architecture): The model predicts trajectory tokens using only an LLM head conditioned on visual context and language prompts. During training, an auxiliary MLP decoder maps the predicted tokens to continuous kinematic trajectories for physics-informed supervision. Training uses a physics-informed hybrid loss ( $\mathcal{L}_{\mathrm{total}}$ ) combining cross-entropy ( $\mathcal{L}_{\mathrm{ce}}$ ), regression ( $\mathcal{L}_{\mathrm{reg}}$ ), and kinematic consistency ( $\mathcal{L}_{\mathrm{pikc}}$ ). Trajectory Generation: Shows the model’s application in both 2D BEV and 3D FPV domains.

Cost Term	Comfort	Balanced	Sporty	Safety	Default
Kinematic Constraints ( $\mathbf{w}_{\mathrm{kin}}$ )
Longitudinal Jerk ( $w_{\mathrm{j,lon}}$ )	0.80	0.50	0.25	0.40	0.20
Lateral Jerk ( $w_{\mathrm{j,lat}}$ )	0.80	0.50	0.25	0.40	0.20
Velocity Offset ( $w_{\mathrm{v}}$ )	0.30	0.60	1.00	0.30	1.00
Distance to obstacles ( $w_{\mathrm{obs}}$ )	0.30	0.80	0.60	2.00	0.00
External Perception ( $\mathbf{w}_{\mathrm{ext}}$ )

TABLE I: Weights across Driving Styles, for the cost function ( 2 ) of the Frentix motion planner used for dataset generation.

Style Label	Samples	Avg Velocity	RMS Accel	RMS Jerk	Path Length
	(Count / %)	( $m/s$ )	( $m/s^{2}$ )	( $m/s^{3}$ )	( $m$ )
Balanced	14,102 (18.5%)	7.15	0.588	0.750	24.44
Comfort	13,766 (18.1%)	7.21	0.585	0.727	24.53
Default	17,684 (23.3%)	6.80	0.486	0.794	23.31
Sporty	14,790 (19.5%)	7.32	0.558	0.780	25.13
Safety	15,688 (20.6%)	6.39	0.583	0.746	21.44

TABLE II: Mean Kinematic Features and Distribution of Filtered Trajectories by Style

Panel of Figure 5 — Fig. 3: Example training dynamics of StyleVLA fine-tuning on the FPV instruction dataset. Top: loss terms ( $\mathcal{L}_{\mathrm{total}}$ , $\mathcal{L}_{\mathrm{ce}}$ , $\mathcal{L}_{\mathrm{reg}}$ , $\mathcal{L}_{\mathrm{pikc}}$ ). Bottom: learned log-variance parameters ( $w_{\mathrm{ce}}$ , $w_{\mathrm{reg}}$ ) that yield adaptive precision weights via $\exp(-w)$ .

Group	Value
Regression Loss Weights
$(\boldsymbol{w}_{\mathrm{reg}},w_{\mathrm{pikc}})$	$(2.0,\,2.0,\,0.5,\,0.5,\,0.5,\,1.5)$
Training Hyperparameters ( FPV )
Backbone	Qwen3-VL-4B-Instruct
Quantization	QLoRA, 4-bit
LoRA $(r,\alpha,\mathrm{dropout})$	$(256,\,512,\,0.05)$

TABLE III: Hyperparameters used in fine-tuning

Fig. 4: Qualitative comparison of style-conditioned trajectory generation under five driving styles (Default, Balanced, Comfort, Sporty, Safety). We visualize the goal, ground truth, and predicted trajectories from pretrained VLMs and SOTA baselines (see legend). ”*” failed to generate trajectories.

Configuration	ADE (m) $\downarrow$	FDE (m) $\downarrow$	PSR $\uparrow$	Heading MAE (rad) $\downarrow$
Impact of Training Data Scaling (with Physics-Informed Hybrid Loss)
Small (4.5k)	2.08	5.43	20.60%	0.073
Medium (20k)	1.51	3.92	27.14%	0.046
Large (40k)	1.47	3.81	29.37%	0.044
Standard (50k)	1.17	3.06	33.19%	0.035
Impact of Loss Components (50k Training dataset)

TABLE IV: Ablation on BEV StyleVLA (Qwen2.5-VL-7B, 3 s horizon). Scaling data and adding physics-informed hybrid loss consistently improves generalization

Model	Score $\uparrow$	PSR $\uparrow$	MR $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	KCE $\downarrow$	Time $\downarrow$
	(0-1)	(ADE $<1$ m)	(FDE $>2$ m)	(m)	(m)	(m)	(s)
Open Source Models
Qwen3-VL-4B	0.00	0.00%	100.0%	-	-	-	2.00
Qwen2.5-VL-7B	0.00	0.00%	100.0%	-	-	-	3.43
InternVL3-9B	0.00	0.00%	100.0%	12.63	25.89	3.72	5.77
Proprietary Models

TABLE V: Benchmarking across VLMs on BEV domain (Zero-Shot)

Model	Score $\uparrow$	PSR $\uparrow$	MR $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	KCE $\downarrow$	Time $\downarrow$
	(0-1)	(ADE $<1$ m)	(FDE $>2$ m)	(m)	(m)	(m)	(s)
Open Source Baselines
Qwen3-VL-4B (Base)	0.00	-	-	-	-	-	4.97
Proprietary Models
Gemini 2.5 Flash	0.12	9.04%	87.21%	9.39	18.21	3.42	1.80
Gemini 2.5 Pro	0.29	16.63%	70.74%	2.23	5.76	0.06	35.48

TABLE VI: Benchmarking across VLMs on FPV domain (Zero-Shot)

Explore similar work

Jul 2, 2026

Teaching Vision-Language-Action Models What to See and Where to Look

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Yuguang Yang, Canyu Chen, Zhewen Tan +10

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5

End-to-End Autonomous Driving VLM Distillation Vision-Language-Action Models Autonomous Driving Planning Vehicle Trajectory Prediction

Ask PDF

Mar 2, 2026

Unifying Language-Action Understanding and Generation for Autonomous Driving

Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.

Xinyang Wang, Qian Liu, Wenjie Ding +7

State Key Lab of CAD&CG, Zhejiang University · Li Auto

Vision-Language-Action Models End-to-End Autonomous Driving Efficient VLA Model Inference

Ask PDF

Oct 12, 2026

AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding

Vision-language-action (VLA) models show strong potential for end-to-end autonomous driving by leveraging world knowledge and understanding to excel in long-tail scenarios. However, current VLAs have two limitations: 1) Degeneration into vision-action (VA) models. For low latency, regression/flow-based methods directly generate actions but underuse VLM understanding, causing unsafe, unexplainable actions. 2) Autoregressive high latency. Autoregressive methods sequentially generate chain-of-thought (CoT) and action tokens, fully leveraging VLM understanding, but high latency hinders deployment. To enable low-latency deployment while using CoT to assist action generation, we propose AUGUstus, an Autonomous-driving VLA model with Unified Generation and Understanding. We unify diffusion-based action generation and autoregressive CoT understanding in one model, where understanding only serves generation. Joint CoT supervision preserves VLM spatial understanding in shared representation for safer, more accurate trajectories. AUGUstus-mode orders action generation before CoT, so CoT supervises reliable actions during training while inference needs only action output, enabling low latency. AUGUstus-GRPO, a unified GRPO algorithm, jointly optimizes action and CoT, improving action performance. Experiments show AUGUstus achieves 94.6 PDMS on NAVSIMv1 and 90.7 EPDMS on NAVSIMv2, SOTA planning without scorer module. On nuScenes, it achieves average L2 error 0.27 m and average collision rate 0.05%, best among compared methods. To validate AUGUstus practicality, we train on in-house data with 512 NVIDIA H800 GPUs for ~2 days, evaluate on an internal 1,000 km dataset, and preliminarily deploy on a real vehicle, achieving 12 Hz operation. Project page: https://augustus-ad.github.io

Yuqi Ye, Caigao Jiang, Zhuonan Wang +12

Peking University · XPeng Motors · Fudan University

Vision-Language-Action Models Autonomous Driving Planning End-to-End Autonomous Driving Reasoning for VLA Models Diffusion Policy

Ask PDF