Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models

Canyu Chen; Yuguang Yang; Zhewen Tan; Yizhi Wang; Ruiyi Zhan; Haiyan Liu; Xuanyao Mao; Jason Bao

cs.CVMar 6, 2026

Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models

Authors: Canyu Chen, Yuguang Yang, Zhewen Tan, Yizhi Wang, Ruiyi Zhan, Haiyan Liu, Xuanyao Mao, Jason Bao,

Organizations: National Superior College for Engineers, Beihang University · Institute for AI Industry Research, Tsinghua University · School of Electronic Information Engineering, Beihang University · School of Computer Science and Engineering, Beihang University ·

Abstract

We identify a fundamental Narrow Policy limitation undermining the performance of autonomous VLA models, where driving Imitation Learning (IL) tends to collapse exploration and limit the potential of subsequent Reinforcement Learning (RL) stages, which often saturate prematurely due to insufficient feedback diversity. Thereby, we propose Curious-VLA, a framework that alleviates the exploit-explore dilemma through a two-stage design. During IL, we introduce a Feasible Trajectory Expansion (FTE) strategy to generate multiple physically valid trajectories and a step-wise normalized trajectory representation to adapt this diverse data. In the RL stage, we present Adaptive Diversity-Aware Sampling (ADAS) that prioritizes high-diversity samples and introduce Spanning Driving Reward (SDR) with a focal style weighting to amplify reward's value span for improving sensitivity to driving quality. On the Navsim benchmark, Curious-VLA achieves SoTA results (PDMS 90.3, EPDMS 85.4) and a Best-of-N PDMS of 94.8, demonstrating its effectiveness in unlocking the exploratory potential of VLA models. Code: https://github.com/Mashiroln/curious_vla.git.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of Narrow Policy. (a) shows quantitative comparison of exploration. It is obvious that baselines achieve low diversity (mean-pFDE: 0.20m for Qwen2.5-VL, 0.33m for ReCogDrive) and low quality (min-FDE: 1.05m for Qwen2.5-VL). In contrast, our Curious-VLA achieves best diversity, quality and performance. (b) compares the trajectories of Qwen2.5-VL and our Curious-VLA. Qwen2.5-VL trajectories perform exploration collapse with single mode or unsafe behaviors, while ours not.

Figure 2: Overall Pipeline of Curious-VLA. As identified in Sec. 3.2 , existing VLAs with the IL-RL pipeline suffer from “Narrow Policy” and tend to generate overlap and unsafe behaviors. In this case, we introduce Curious-VLA, which takes into account both the think process and planing waypoint, to improve the diversity, quality and performance (middle panel). Specifically, Curious-VLA alleviates the Narrow Policy in both the IL stage (left panel) and the RL stage (right panel). For IL, to improve the separability of trajectory patterns, we generate diverse trajectories, structure the driving reasoning process into a four-stage CoT, and normalize each prediction step (See Sec. 4.1 ). For RL, to address the advantage collapse problem and sustain exploration, we introduce adaptive diverse aware sampling and spanning driving reward (See Sec. 4.2 ).

Figure 3: Visualization of horizon physical scale mismatch by waypoints distribution.

Figure 4: Qualitative comparison with BEV and Camera. We can see that our Curious-VLA achieves more feasible trajectories.

	Base Model	Sensors	Trajectory	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	C $\uparrow$	PDMS $\uparrow$
Human GT
Human [ 10 ]	-	-	-	100.0	100.0	87.5	100.0	99.9	94.8
Classical E2E
Ego-MLP [ 29 ]	-	6x C	Continuous	93.1	78.3	63.2	84.0	100.0	66.4
UniAD [ 16 ]	-	3x C + L	Continuous	97.7	92.8	79.2	92.8	100.0	84.0
DiffusionDrive [ 30 ]	-	3x C + L	Continuous	98.2	96.2	82.2	94.7	100.0	88.1

Table 1: Comparison of different methods on Navsim V1 benchmark. The model with † indicates evaluation using Best-of-N sampling. The best and second best results are bolded and underlined , respectively. Continuous regresses the trajectory with an additional MLP/Transformer head. Discrete Action decodes action tokens that discretize each dimension of the trajectory into several bins. Text Waypoint decodes textual numbers that represent the trajectory.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	C $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Ego-MLP [ 29 ]	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0
VADv2 [ 20 ]	97.3	91.7	98.2	99.9	77.6	92.7	66.0	100.0	97.4	76.6
TransFuser [ 5 ]	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
HydraMDP++ [ 28 ]	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4
DriveSuprim [ 47 ]	97.5	96.5	99.4	99.6	88.4	96.6	95.5	98.3	77.0	83.1
ARTEMIS [ 12 ]	98.3	95.1	98.6	99.8	81.5	97.4	96.5	98.3	–	83.1

Table 2: Comparison of different methods with the Extended PDMS metric of NAVISM V2.

Method	ST-P3 metrics		UniAD metrics
Method	L2 (m) $\downarrow$	Collision (%) $\downarrow$	L2 (m) $\downarrow$	Collision (%) $\downarrow$
ST-P3 [ 15 ]	2.11	0.71	–	–
VAD [ 20 ]	0.37	0.14	–	–
UniAD [ 16 ]	0.69	0.12	1.03	0.31
EMMA [ 18 ]	0.32	–	–	–
OpenEMMA [ 45 ]	2.81	–	–	–

Table 3: Open-loop evaluation on the nuScenes Benchmark.

Method	Stage	Quality	Diversity	Perf.
ReCogDrive	IL+RL	0.295 / 0.621	0.148 / 0.325	90.95
Qwen2.5-VL	IL	0.481 / 1.052	0.090 / 0.200	90.69
+ FTE (w/o SN)	IL	0.513 / 1.129	0.170 / 0.381	90.65
+ FTE	IL	0.480 / 1.078	0.346 / 0.803	91.31
+ FTE + RL	IL+RL	0.269 / 0.547	0.641 / 1.415	91.55

Table 4: Exploration analysis (all metrics @k=8). We evaluate Quality (minADE/FDE

\downarrow

), Diversity (mean-pADE/FDE

\uparrow

), and Perf. (mean-PDMS

\uparrow

). In the Quality and Diversity columns, values correspond to ADE / FDE.

DE	CoT	SN	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	C $\uparrow$	PDMS $\uparrow$
$\times$	$\times$	$\times$	97.7	91.8	85.8	96.8	98.4	83.9
$\times$	✓	$\times$	98.2	93.2	85.8	97.3	98.4	85.6
✓	✓	$\times$	98.0	93.0	85.9	97.2	98.4	85.2
$\times$	✓	✓	98.2	94.3	86.7	97.3	98.4	86.9
✓	✓	✓	98.3	95.1	86.5	97.6	98.3	87.6

Table 5: Ablation study on Imitation Learning approaches. We evaluate the impact of three key Feasible Trajectory Expansion components: DE (Exploratory Data Expansion), CoT (Chain-of-Thought) and SN (Step-wise Normalization).

Sampling Strategy	SDR	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	C $\uparrow$	PDMS $\uparrow$
Human Difficulty	$\times$	73.9	43.7	94.9	70.3	97.1	35.2
Reject Unimodal	$\times$	98.4	96.0	87.0	97.8	98.4	88.8
ADAS(1x)	$\times$	98.2	96.3	88.6	97.6	98.2	89.6
ADAS(3x)	$\times$	98.4	96.8	88.5	97.8	98.1	90.1
ADAS(3x)	$\checkmark$	98.4	96.9	88.5	97.9	98.1	90.3

Table 6: Ablation study on RL approaches. We evaluate the effect of different Sampling strategies and the SDR (Spanning Driving Reward) on Navsim v1. The

i

x denotes a total of

i

outer-loop iterations of ADAS.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SFT
Base Model	Qwen2.5-VL-3B
Max Pixels	262144
Global Batch Size	128
Epochs	3(align) / 3(fine-tune)
Trainset Samples	103k(navtrain) + 39k(DE)
Learning Rate	4e-5 / 5e-6

Appendix

Table 7: Training configurations for SFT.

Setting	GRPO
Max Pixels	262144
Rollout Batch Size	256
Actor Global Batch Size	256
N Rollout(Group Size)	8
Outer Loops	3
Total Steps	130

Appendix

Table 8: Training configurations for GRPO.

Panel of Figure 5 — Figure 5: The prompt template and representative answer examples used to filter semantic-aware challenging scenarios with multiple feasible driving intents.

Method	Setting	Latency (s)
AutoVLA	Dual-Sys (Text)	9.31
AutoVLA	Dual-Sys (Action)	3.95
AutoVLA	Dual-Sys (Action + RFT)	1.31
Curious-VLA	Slow Think only (Text)	1.57

Appendix

Table 9: Inference Latency Comparison. Text : Text waypoint; Action : Action token. AutoVLA uses a Fast-Slow Dual-System.

Method	Output	Quality	Diversity	Perf.
DiffusionDrive	Diff.@all	0.218 / 0.430	0.571 / 1.175	87.60
DiffusionDrive	Diff.@1	0.350 / 0.720	0.037 / 0.076	88.10
Curious-VLA	AR@1	0.269 / 0.547	0.641 / 1.415	91.55

Appendix

Table 10: Extended exploration analysis. DiffusionDrive is evaluated with its 20 denoised candidates(@all) and the confidence Top-1 selection(@1). All metrics @k=8.

Figure 6: RL Training Curves. Val Reward and Test PDMS over 130 steps (ADAS 3x). The Random Sample baseline shows RL collapse.

Figure 7: Stability Analysis. $k=4$ training runs (ADAS 1x). Critic and Val Reward curves show consistent improvement with low variance.

Panel of Figure 8 — Figure 8: More visualization between Curious-VLA(top) and Qwen2.5-VL(bottom).

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving
Apr 3, 2026Zihao Sheng, Xin Ye, Jingru Luo +2World Model Learning VLMs for Autonomous Driving
Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · University of Wisconsin–Madison
Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures
Mar 1, 2026Yuechen Luo, Qimao Chen, Fang Li +5VLMs for Autonomous Driving RL for Autonomous Driving
Tsinghua University · University of Macau · Beijing Jiaotong University
DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving
Aug 11, 2026Zebin Xing, Yupeng Zheng, Qiang Chen +10Memory-Augmented VLMs End-to-End Autonomous Driving
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Chongqing Chang’an Technology Co., Ltd., Chongqing, China

Apr 3, 2026J/K move · Enter open · S save

ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving

Zihao Sheng, Xin Ye, Jingru Luo, Sikai Chen+1

Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · University of Wisconsin–Madison

World Model Learning VLMs for Autonomous Driving End-to-End Autonomous Driving RL Exploration Vision-Language-Action Models World Models for Autonomous Driving+1

Open paper AskPDF arXiv

End-to-end autonomous driving models based on Vision-Language-Action (VLA) architectures have shown promising results by learning driving policies through behavior cloning on expert demonstrations. However, imitation learning inherently limits the model to replicating observed behaviors without exploring diverse driving strategies, leaving it brittle in novel or out-of-distribution scenarios. Reinforcement learning (RL) offers a natural remedy by enabling policy exploration beyond the expert distribution. Yet VLA models, typically trained on offline datasets, lack directly observable state transitions, necessitating a learned world model to anticipate action consequences. In this work, we propose a unified understanding-and-generation framework that leverages world modeling to simultaneously enable meaningful exploration and provide dense supervision. Specifically, we augment trajectory prediction with future RGB and depth image generation as dense world modeling objectives, requiring the model to learn fine-grained visual and geometric representations that substantially enrich the planning backbone. Beyond serving as a supervisory signal, the world model further acts as a source of intrinsic reward for policy exploration: its image prediction uncertainty naturally measures a trajectory's novelty relative to the training distribution, where high uncertainty indicates out-of-distribution scenarios that, if safe, represent valuable learning opportunities. We incorporate this exploration signal into a safety-gated reward and optimize the policy via Group Relative Policy Optimization (GRPO). Experiments on the NAVSIM and nuScenes benchmarks demonstrate the effectiveness of our approach, achieving a state-of-the-art PDMS score of 93.7 and an EPDMS of 88.8 on NAVSIM. The code is available at https://zihaosheng.github.io/ExploreVLA/.

Abstract

Autonomous Driving

Explore similar work

ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving

Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures

DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving