Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning

Yuehao Song; Shaoyu Chen; Hao Gao; Yifan Zhu; Weixiang Yue; Jialv Zou; Bo Jiang; Zihao Lu

cs.CVMar 11, 2026

Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning

Authors: Yuehao Song, Shaoyu Chen, Hao Gao, Yifan Zhu, Weixiang Yue, Jialv Zou, Bo Jiang, Zihao Lu,

Organizations: Huazhong University of Science & Technology · Horizon Robotics

Abstract

Vision-language models (VLMs) enhance the planning capability of end-to-end (E2E) driving policy by leveraging high-level semantic reasoning. However, existing approaches often overlook the dual-system consistency between VLM's high-level decision and E2E's low-level planning. As a result, the generated trajectories may misalign with the intended driving decisions, leading to weakened top-down guidance and decision-following ability of the system. To address this issue, we propose Senna-2, an advanced VLM-E2E driving policy that explicitly aligns the two systems for consistent decision-making and planning. Our method follows a consistency-oriented three-stage training paradigm. In the first stage, we conduct driving pre-training to achieve preliminary decision-making and planning, with a decision adapter transmitting VLM decisions to E2E policy in the form of implicit embeddings. In the second stage, we align the VLM and the E2E policy in an open-loop setting. In the third stage, we perform closed-loop alignment via bottom-up Hierarchical Reinforcement Learning in 3DGS environments to reinforce the safety and efficiency. Extensive experiments demonstrate that Senna-2 achieves superior dual-system consistency (19.3% F1 score improvement) and significantly enhances driving safety in both open-loop (5.7% FDE reduction) and closed-loop settings (30.6% AF-CR reduction).

Figures & tables

Figure 1: Consistency gap between VLM and the E2E planner. (a) The E2E planner may misalign with the VLM decision (①), e.g. , ②wrong direction or ③mismatched speed change. (b) Existing method [ 19 ] shows scattered speed distributions, inconsistent with speed decisions. (c) With consistency-oriented training, Senna-2 produces more distinct and decision-aligned speed distributions, reflecting improved dual-system consistency and decision-following ability. Relative speed ratio: the ratio between the planned speed at the 3rd second and the initial speed, reflecting the tendency of speed change of the planning trajectory.

Figure 2: Overall model architecture of Senna-2. Text and visual inputs are processed by the VLM to produce high-level driving decisions, which are converted by the Decision Adapter into VLM condition embeddings. The E2E planner then fuses the VLM condition with its own E2E features to generate a trajectory consistent with the high-level decisions.

Figure 3: Consistency-oriented training recipe. We perform three training stages, including driving pre-training, open-loop alignment, and closed-loop alignment with Hierarchical Reinforced Learning (HRL).

Method	Path (F1) $\uparrow$			Speed (F1) $\uparrow$				Avg.
Method	Straight	Left	Right	Keep	Acc.	Dec.	Stop	Avg.
Senna [ 19 ]	0.763	0.533	0.574	0.550	0.612	0.628	0.802	0.637
Ours	0.809	0.664	0.710	0.754	0.769	0.780	0.838	0.760

Table 1: Comparison of decision-planning consistency. Senna-2 exhibits higher consistency between the VLM decision and the E2E planning compared to Senna [ 19 ] .

Method	FDE (m) $\downarrow$	ADE (m) $\downarrow$	CR (%) $\downarrow$	DCR (%) $\downarrow$	SCR (%) $\downarrow$
TransFuser [ 5 ]	0.844	0.297	0.981	0.827	0.154
VAD [ 20 ]	0.722	0.262	0.621	0.554	0.067
GenAD [ 46 ]	0.806	0.290	0.520	0.491	0.030
ResAD [ 47 ]	0.634	0.234	0.378	0.367	0.011
Senna [ 19 ]	0.633	0.236	0.294	0.286	0.008
Ours	0.597	0.225	0.288	0.283	0.005

Table 2: Open-loop quantitative comparisons with existing method. Our method outperforms existing methods in both displacement error and collision rate metrics.

Method	CR $\downarrow$	AF-CR $\downarrow$	Safety@1 $\uparrow$	Safety@2 $\uparrow$
TransFuser [ 5 ]	0.435	0.269	0.531	0.454
VAD [ 20 ]	0.502	0.280	0.458	0.362
GenAD [ 46 ]	0.557	0.244	0.402	0.332
ResAD [ 47 ]	0.509	0.288	0.469	0.399
VADv2 [ 18 ]	0.422	0.199	0.514	0.458
RAD [ 11 ]	0.281	0.113	0.613	0.543

Table 3: Closed-loop quantitative comparisons with existing methods on the 3DGS evaluation benchmark. Our method significantly improves safety and reduces the collision rate.

Figure 4: Closed-loop speed control in an empty-road scenario. We visualize (a) driving trajectories, (b) speed curves, and (c) mileage curves under different VLM decisions. Our method exhibits strong decision-following ability. The low-level planning follows the high-level decision for speed control. Normal denotes using the VLM-predicted decision, while accelerate , keep and decelerate denotes using the fixed ones during the whole rollout.

Dual system	VLM token	Dec. token	Open-loop FDE (m) $\downarrow$	Open-loop CR (%) $\downarrow$	Closed-loop AF-CR $\downarrow$	Avg. F1 $\uparrow$
✗	✗	✗	0.695	0.387	0.288	-
✓	✗	✓	0.634	0.299	0.148	0.660
✓	✓	✗	0.624	0.312	0.151	0.688
✓	✓	✓	0.567	0.281	0.144	0.701

Table 4: Ablation study on the model architecture. Dual system: whether the VLM is integrated; VLM token / Dec. token: whether the VLM token/decision token is used in the decision adapter.

Stage 1	Stage 2	Stage 3	Open-loop FDE (m) $\downarrow$	Open-loop CR (%) $\downarrow$	Closed-loop AF-CR $\downarrow$	Avg. F1 $\uparrow$
✓			0.567	0.281	0.144	0.701
✓	✓		0.575	0.290	0.118	0.764
✓	✓	✓	0.597	0.288	0.077	0.760

Table 5: Ablation study on the training stages. Our three-stage training pipeline strikes a balance between open-loop and closed-loop performance, while also enhancing the consistency between high-level decisions and low-level planning.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TL $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Transfuser [ 5 ]	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
Hydra-MDP++ [ 24 ]	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4
DriveSuprim [ 45 ]	97.5	96.5	99.4	99.6	88.4	96.6	95.5	98.3	77.0	83.1
ARTEMIS [ 9 ]	98.3	95.1	98.6	99.8	81.5	97.4	96.5	98.3	-	83.1
DiffusionDrive [ 27 ]	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
ResAD [ 47 ]	97.8	97.2	99.5	99.8	88.2	96.9	97.0	98.4	88.2	85.5

Table 6: Performance on the NAVSIM v2 navtest Benchmark.

Figure 5: Closed-loop qualitative comparisons between Senna [ 19 ] and Senna-2. Senna suffers from (a) planning misalignment in stop scenario and (b) decision misalignment in cut-in scenario. In contrast, Senna-2 maintains both consistent decision-making and accurate trajectory planning across these challenging situations, demonstrating improved safety and reliability in closed-loop driving.

Config	Value
learning rate	$5\times 10^{-5}$
learning rate schedule	cosine decay
optimizer	AdamW [ 23 , 28 ]
optimizer hyper-parameters	$\beta_{1},\beta_{2},\epsilon=0.9,0.999,1\times 10^{-8}$
weight decay	$1\times 10^{-4}$
batch size	512

Table A1 : Training configurations for VLM pre-training.

Config	Value
learning rate	$1\times 10^{-4}$
learning rate schedule	cosine decay
optimizer	AdamW [ 23 , 28 ]
optimizer hyperparameters	$\beta_{1},\beta_{2},\epsilon=0.9,0.999,1\times 10^{-8}$
weight decay	$1\times 10^{-4}$
batch size	12288

Table A2 : Training configurations for E2E pre-training.

Config	Value
learning rate	$5\times 10^{-5}$
learning rate schedule	cosine decay
optimizer	AdamW [ 23 , 28 ]
optimizer hyper-parameters	$\beta_{1},\beta_{2},\epsilon=0.9,0.999,1\times 10^{-8}$
weight decay	$1\times 10^{-4}$
batch size	512

Table A3 : Training configurations for adapter pre-training.

Config	Value
learning rate	$1\times 10^{-5}$
learning rate schedule	cosine decay
optimizer	AdamW [ 23 , 28 ]
optimizer hyper-parameters	$\beta_{1},\beta_{2},\epsilon=0.9,0.999,1\times 10^{-8}$
weight decay	0
batch size	384

Table A4 : Training configurations for open-loop alignment.

Config	Value
learning rate	$1\times 10^{-5}$
learning rate schedule	cosine decay
optimizer	AdamW [ 23 , 28 ]
optimizer hyper-parameters	$\beta_{1},\beta_{2},\epsilon=0.9,0.999,1\times 10^{-8}$
weight decay	0
batch size	128

Table A5 : Training configurations for closed-loop alignment.

Figure A1 : Learning curve across the whole training pipeline. (a) Driving pretraining. (b) Open-loop alignment. (c) Closed-loop alignment with HRL. The dashed line denotes the performance of the extended driving-pretraining baseline.

Figure A2 : More qualitative results in the cut-in scenarios. In these cases, Senna [ 19 ] often reacts too late and leads to near-collision outcomes, while our policy maintains coherent decision–planning coupling and ensures safe avoidance.

Figure A3 : More qualitative results in diverse scenarios. (a) Our policy safely handles a hard-braking event, while the baseline causes a collision. (b) During a right turn, our method avoids the cyclist, whereas the baseline crashes. (c) Our policy completes the lane change, but the baseline drifts across lanes. (d) Our method aborts the lane change upon hazard detection, while the baseline is rear-ended.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving
Jun 29, 2026Chen Yang, Yuhao Wei, Ze Xu +6VLMs for Autonomous Driving End-to-End Autonomous Driving
Chongqing University
$AutoDrive\text{-}P^3$ : Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning
Mar 30, 2026Yuqi Ye, Zijian Zhang, Junhong Lin +3Autonomous Driving Planning VLMs for Autonomous Driving
School of Electronic and Computer Engineering, Peking University
CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving
Jul 3, 2026Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh +1VLMs for Autonomous Driving End-to-End Autonomous Driving
Qualcomm AI Research

Jun 29, 2026J/K move · Enter open · S save

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

Chen Yang, Yuhao Wei, Ze Xu, Ziheng Zou+5

Chongqing University

VLMs for Autonomous Driving End-to-End Autonomous Driving World Model-Based Planning Model-Based Planning World Models for Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning. However, trajectories directly generated by VLMs often encode only coarse driving intentions and remain insufficient for geometrically accurate, future-aware, and multi-view-grounded planning. To address these limitations, we develop the Layer-Wise World-Model-Guided Driving framework (LWDrive). LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance. Instead of treating the VLM output as the final trajectory, LWDrive uses it as an intent-aware coarse plan, expands a diverse candidate space around it, and progressively refines the candidates through a Foresight Cascade Planner (FCP). Specifically, we introduce future-frame generation supervision to encourage the VLM to learn forward-looking scene representations, thereby injecting planning-relevant predictive dynamics into its internal hidden states. Built upon these world-model-supervised representations, FCP exploits VLM features across multiple layers and integrates historical temporal states, Action-Query representations, and current-frame multi-view Bird's-Eye-View (BEV) features to refine candidate trajectories in a coarse-to-fine manner. This design enables progressive correction of spatial positions and motion trends while grounding trajectory refinement with multi-view scene cues and preserving the high-level driving intention produced by the large model. Finally, a score head evaluates the refined candidates and selects the best trajectory as the final planning output. Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2. Code and models will be made publicly available.

Abstract

End-to-End Autonomous Driving

Explore similar work

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

AutoDrive-P3AutoDrive\text{-}P^3AutoDrive-P3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

$AutoDrive\text{-}P^3$ : Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning