Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving

Yuechen Luo; He Chong; Fang Li; Shaoqing Xu; Hexin Zhang; Ruiqi Zhang; Ziying Song; Lei Yang

cs.ROOct 9, 2026

Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving

Authors: Yuechen Luo, He Chong, Fang Li, Shaoqing Xu, Hexin Zhang, Ruiqi Zhang, Ziying Song, Lei Yang,

Organizations: Tsinghua University · University of Macau · University of Science and Technology of China · Beijing Jiaotong University ·

Abstract

Reinforcement Learning (RL) substantially improves planning performance in autonomous-driving Vision-Language-Action (VLA) models. However, existing RL methods rely on scalar trajectory rewards that measure overall quality but fail to identify specific planning errors or provide actionable correction signals. In long-tail scenarios, errors in scene understanding, behavior selection, and trajectory execution are compressed into a single score. We observe that Reflective Guidance turns trajectory evaluation into explicit diagnoses and corrective instructions, substantially improving planning accuracy. Yet such guidance requires an initial prediction and its evaluation results, making it unavailable for direct single-pass inference. We propose Reflect-VLA, which treats Reflective Guidance as training-time privileged information and internalizes its planning benefits through Dual-Path On-Policy Self-Distillation (Dual-OPSD). Unlike scalar-reward optimization, Dual-OPSD uses the driving score only to route student-generated trajectories. It distills dense token-level corrective distributions from the teacher for risky trajectories while applying self-imitation to preserve reliable ones. To provide reliable corrective distributions on student rollouts, the teacher is progressively constructed through Driving-Knowledge Pretraining, Planning-and-Refinement SFT, and Self-Refinement Rejection Sampling. At inference time, Reflect-VLA generates trajectories in a single pass without explicit guidance or iterative refinement. Experiments on NAVSIM v1/v2 and Bench2Drive demonstrate competitive planning performance while reducing training time by over 90% compared with RL.

Figures & tables

Figure 1: From Reflective Guidance to Internalized Planning. (a) Reflective Guidance corrects an unsafe initial trajectory during iterative inference, revealing substantial planning gains. (b) Reflect-VLA transfers these gains through Dual-OPSD by aligning teacher and student token distributions along student-generated trajectories. (c) Across different model scales, Reflect-VLA improves planning performance over SFT while requiring substantially less training time than RL, enabling guidance-free single-pass planning.

Figure 2: Overview of Reflect-VLA. Reflective Guidance is constructed from trajectory evaluation and used as privileged information to train a refinement-capable teacher. Dual-OPSD then transfers corrective distributions to the student on risky trajectories while preserving reliable student trajectories. Only the student is used for single-pass inference.

Method	Image	Lidar	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
End-to-End Methods
UniAD ( Hu et al. 2023 )	✓		97.8	91.9	92.9	100	78.8	83.4
TransFuser ( Chitta et al. 2022 )	✓	✓	97.7	92.8	92.9	100	84.0	84.0
DiffusionDrive ( Liao et al. 2025 )	✓	✓	98.2	96.2	94.7	100	82.2	88.1
WoTE ( Li et al. 2025b )	✓	✓	98.5	96.8	94.9	99.9	81.9	88.3
DVGT-2 ( Zuo et al. 2026 )	✓		98.7	97.9	95.8	100	84.3	90.3

Table 1: Comparison with state-of-the-art methods on NAVSIMv1 with PDMS.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Ego Status	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0
TransFuser ( Chitta et al. 2022 )	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
DiffusionDrive ( Liao et al. 2025 )	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
HydraMDP++ ( Li et al. 2024 )	98.5	98.5	99.5	99.7	87.4	97.9	95.8	98.2	75.7	85.6
DriveVLA-W0-7B ( Li et al. 2025a )	98.5	99.1	98.0	99.7	86.4	98.1	93.2	97.9	58.9	86.1
Reflect-VLA-0.8B	98.8	96.7	99.1	99.7	87.5	98.3	96.5	98.3	86.2	88.8

Table 2: Comparison with state-of-the-art methods on NAVSIMv2 with EPDMS.

End-to-End Methods
Method	DS $\uparrow$	SR (%) $\uparrow$	Effi $\uparrow$	Comf $\uparrow$
VAD ( Jiang et al. 2023 )	42.35	15.00	157.94	46.01
UniAD-Base ( Hu et al. 2023 )	45.81	16.36	129.21	43.58
DriveAdapter ( Jia et al. 2023 )	64.22	33.08	70.22	16.01
Vision-Language-Action Methods
ReasonPlan ( Liu et al. 2025 )	64.01	34.55	180.64	25.63

Table 3: Comparison with state-of-the-art methods on the Bench2Drive benchmark with DS (Driving Score), SR (Success Rate), Effi (Efficiency), and Comf (Comfortness).

Method	Student	Teacher	OPSD
Reflect-VLA-0.8B	86.3	92.4	89.3
Reflect-VLA-2B	87.4	92.7	90.0
Reflect-VLA-4B	88.4	93.2	91.5

Table 4: Planning performance of the student, teacher, and student after Dual-OPSD on NAVSIMv1 using PDMS, where student and teacher refer to the same model after three-stage SFT performing trajectory prediction without and with guidance, respectively.

Base Model	SFT	GRPO	OPSD
Qwen3.5-0.8B	0.11	5.3	1.3
Qwen3.5-2B	0.26	22.3	1.7
Qwen3.5-4B	0.43	33.4	2.4

Table 5: Comparison of training time (hours) per 10k samples for SFT, GRPO, and OPSD across different model scales.

Figure 3: Qualitative comparisons and token-level analysis. Left and middle: trajectories generated by the unguided student, guidance-conditioned teacher, and distilled student. Right: distributions over special trajectory tokens at critical decoding steps under the same student-generated prefix. Each token encodes a relative displacement quantized at 0.01 m resolution; e.g., <| $x\_208$ |> represents a 2.08 m forward displacement for the next waypoint.

Model	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
SFT	98.5	96.3	94.9	100	81.7	88.1
Pre+SFT	98.7	96.5	95.4	100	82.1	88.5
Pre+SFT+RS	98.6	96.4	95.3	100	82.2	88.4
Pre+SFT+OPSD	98.9	97.1	95.8	100	84.7	90.2
Pre+SFT+RS+OPSD	98.9	98.0	96.7	100	86.0	91.5

Table 6: Ablation study of training stages. We evaluate the effect of pretraining, supervised fine-tuning, rejection sampling and on-policy self-distillation on driving performance using NAVSIM evaluation metrics with the 4B model.

ID	Trajectory	Planning	Expert	Refinement	PDMS $\uparrow$
ID	Verdict	Diagnosis	Alignment	Directive	PDMS $\uparrow$
1		✓	✓	✓	93.1
2	✓		✓	✓	92.4
3	✓	✓		✓	92.1
4	✓	✓	✓		92.9
5	✓	✓	✓	✓	93.2

Table 7: Ablation study of reflective guidance components for the teacher model. We report the NAVSIM PDMS of the 4B teacher when conditioned on different combinations of guidance components.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Traditional End-to-end Methods
Method	VLM Model	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Method	VLM Model	1s	2s	3s	Avg.	1s	2s	3s	Avg.
UniAD* ( Hu et al. 2023 )	-	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37
Vision-Language-Action Methods
Doe-1 ( Zheng et al. 2024 )	Lumina-mGPT-7B	0.50	1.18	2.11	1.26	0.04	0.37	1.19	0.53
AutoVLA* ( Zhou et al. 2025b )	Qwen2.5-VL-3B	0.28	0.66	1.16	0.70	0.14	0.25	0.53	0.31

Appendix

Table 8: Comparison with state-of-the-art methods on nuScenes. * indicates that ego status is used.

Top-K	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Top-16	98.7	97.5	95.1	100	85.4	90.2
Top-64 (ours)	98.9	98.0	96.7	100	86.0	91.5
Top-256	98.9	97.9	96.5	100	86.0	91.4
Full Vocab	98.9	97.9	96.7	100	86.0	91.5

Appendix

Table 9: Ablation study of the top-

k

truncation used for token-level distillation on NAVSIM with the 4B model.

Direction	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Forward KL	98.9	97.8	95.9	100	85.8	91.0
Reverse KL(ours)	98.9	98.0	96.7	100	86.0	91.5

Appendix

Table 10: Ablation study of the KL-divergence direction used for token-level distillation on NAVSIM with the 4B model.

Threshold $s$	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
0.65	98.9	97.8	96.0	100	85.9	91.1
0.75	98.9	97.8	96.4	100	86.0	91.3
0.85 (ours)	98.9	98.0	96.7	100	86.0	91.5
0.95	98.9	97.8	96.2	100	86.0	91.2

Appendix

Table 11: Ablation study of the PDMS routing threshold

s

in Dual-OPSD on NAVSIM with the 4B model.

Stage	Hyper-parameter	Value
Pretraining	Epochs	2
	Batch size	1
	Learning rate	$1\times 10^{-5}$
	Gradient accumulation steps	2
	Optimizer	Adam
	Weight decay	0.05

Appendix

Table 12: Training hyperparameters of Reflect-VLA.

Figure 4: Detailed qualitative visualization of Reflect-VLA. We present three representative failure cases: Drivable-Area Violation , Insufficient TTC Margin , and At-Fault Collision Risk . For each scenario, we show the front-view image, the initial student trajectory, the guidance-conditioned teacher trajectory, and the distilled student trajectory, together with the constructed Reflective Guidance and the corresponding token-level distribution comparison at a critical waypoint under the same student-generated prefix. The results illustrate how Reflective Guidance shifts the teacher toward safer and more expert-aligned trajectory tokens, and how Dual-OPSD transfers these corrective preferences to the distilled student.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

RefineDrive: Reliable Failure-Guided Learning for Vision-Language-Action Driving
Sep 28, 2026Zhe Sun, Ziyi Luo, Yehao Lu +2VLMs for Autonomous Driving Autonomous Driving Planning
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Yinwang Intelligent Technology Co., Ltd.
Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures
Mar 1, 2026Yuechen Luo, Qimao Chen, Fang Li +5VLMs for Autonomous Driving RL for Autonomous Driving
Tsinghua University · University of Macau · Beijing Jiaotong University
Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Driving
Jun 22, 2026Zisheng Chen, Yuping Qiu, Jianhua Han +6VLMs for Autonomous Driving End-to-End Autonomous Driving
Sun Yat-sen University · HKUST(GZ) · Yinwang Intelligent Technology Co. Ltd.

Sep 28, 2026J/K move · Enter open · S save

RefineDrive: Reliable Failure-Guided Learning for Vision-Language-Action Driving

Zhe Sun, Ziyi Luo, Yehao Lu, Lei Zhou+1

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Yinwang Intelligent Technology Co., Ltd.

VLMs for Autonomous Driving Autonomous Driving Planning RL for Autonomous Driving Safe RL

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models for autonomous driving rely heavily on successful expert demonstrations, leaving model-specific failures underexploited. Learning from these failures is hindered by unreliable diagnoses, poorly matched correction targets, and coarse rewards. We propose RefineDrive, a failure-guided post-training framework that learns from self-generated failures through targeted supervision and safety-aware reinforcement learning. Reliable Diagnosis derives structured, verifiable feedback on collisions and drivable-area violations directly from simulator states. Minimum-Correction Target Retrieval searches a clustered human trajectory bank for nearby corrections that satisfy hard-safety constraints in the current scene, prioritizing preservation of the failed prediction's motion pattern. Conditioned on the driving context and failed trajectory, Correction SFT learns to generate the diagnosis followed by the retrieved correction as a training-only auxiliary task. We then apply GRPO with a Safety-Layered Reward that strictly prioritizes hard-safe trajectories, retains continuous safety feedback for both unsafe and hard-safe trajectories, and rewards driving progress only after hard safety is satisfied. At inference, the policy directly predicts trajectories from the driving context without an explicit diagnosis or repair stage. On NAVSIM v1, RefineDrive improves the 4B base SFT policy from 87.7 to 91.7 PDMS. Using the same checkpoint without additional training, RefineDrive achieves 89.4 EPDMS on the original NAVTEST scenes evaluated with NAVSIM v2 extended metrics. Controlled ablations support the benefits of structured diagnosis supervision, retrieved corrections, and safety-layered optimization for direct planning.

Abstract

Autonomous Driving Planning

Explore similar work

RefineDrive: Reliable Failure-Guided Learning for Vision-Language-Action Driving

Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures

Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Driving