Vision-language-action (VLA) models bring multimodal understanding and world knowledge to end-to-end autonomous driving. To further improve the quality of predicted trajectories, existing studies introduce candidate refinement, residual denoising, and token correction. However, these methods typically focus on unified refinement of the entire trajectory, paying limited attention to the distinct physical quantities and their interactions. We further investigate physical specialization and coordination in trajectory refinement, centering on position and heading for simpler and more intuitive correction modeling. To this end, we propose CARE-Drive (Coupled Alternating Refinement for Ego-trajectories), which decomposes trajectory refinement into waypoint residual correction and position-conditioned heading generation. Through imitation learning and a future-token auxiliary task, the backbone provides predictive representations and a base trajectory. The position expert predicts bounded metric waypoint residuals, while the heading expert reconstructs vehicle orientation from adjacent angular increments conditioned on the corrected path. With the backbone frozen, the two experts are optimized alternately through supervised initialization and reinforcement fine-tuning. This design simplifies refinement through clear physical specialization while preserving conditional coupling during prediction and optimization. On NAVSIM v1, CARE-Drive achieves 88.8 PDMS, improving by 2.4 points over the unrefined base trajectories generated by the same backbone. Ablation studies further support the effectiveness of specialized experts and staged training.
Figures & tables
Figure 1: CARE-Drive overview. (a) The predictive backbone produces shared action features and an initial trajectory; a frozen video teacher provides future-feature supervision during training. (b) The position expert predicts metric waypoint residuals, and the heading expert reconstructs orientation from adjacent angular increments conditioned on corrected positions and shared action features. Greedy decoding yields one eight-waypoint trajectory. (c) Dynamically weighted imitation learning trains the backbone, followed by alternating position and heading optimization with the backbone frozen. Each expert update holds its conditioning models fixed. Snowflakes mark frozen modules.
Family
Method
Sensors
Frames
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
End-to-end
UniAD [ 4 ]
C
N
97.8
91.9
92.9
100.0
78.8
83.4
Hydra-MDP [ 10 ]
C+L
N
98.3
96.0
94.6
100.0
78.7
86.5
DiffusionDrive [ 11 ]
C+L
N
98.2
96.2
94.7
100.0
82.2
88.1
World models
LAW [ 6 ]
C
N
96.4
95.4
88.7
99.9
81.7
84.6
Epona [ 18 ]
C
N
97.9
95.1
93.8
99.9
80.4
86.2
PWM [ 19 ]
C
N
98.6
95.9
95.4
100.0
81.8
88.1
Table 1: NAVSIM v1 navtest results. AutoVLA and CARE-Drive are locally evaluated on 12,146 scenes using the official CPU scorer; other results follow CoWorld-VLA [ 5 ] . C = camera, L = LiDAR; N denotes multiple frames when the exact count is not specified by the source. All scores use a 0–100 scale, where higher is better. Values are displayed to one decimal; trailing zeros for reported integers are formatting only.
Configuration
Full-test PDMS ↑
Gain over backbone
Backbone only
86.462
—
Add heading expert
87.562
+1.100
Add position expert; freeze heading expert
88.672
+2.210
Freeze position expert; adapt heading expert
88.837
+2.375
Table 2: Effect of progressively adding correction experts on the full test set. Gains are relative to the backbone-only configuration.
Variant
Future queries
Supervision
PDMS ↑
No future branch
0
No
74.310
Queries only
96
No
74.680
Auxiliary task
96
Yes
76.800
Table 3: Effect of the future-token auxiliary task.
Student vision tower
Test PDMS ↑
Frozen
74.198
Unfrozen (including merger)
69.097
Table 4: Effect of visual-tower freezing.
Training
Full-test PDMS
Uniform weighting
85.863
Dynamic weighting
86.462
Table 5: Difficulty-weighted imitation learning. Full-test PDMS of the backbone without correction experts.
Configuration
PDMS ↑
Backbone only
86.462
Staged expert training (CARE-Drive)
88.837
Joint SFT
85.651
Joint SFT + GRPO
83.671
Table 6: Staged versus joint expert training.
Configuration
Added parameters (M)
PDMS ↑
Backbone only
0
86.462
MLP decoder expansion
156.84
86.510
Separate experts (CARE-Drive)
156.84
88.837
Single expert, final RL
156.84
84.227
Single expert, after SFT
156.84
85.147
Table 7: Correction-module structure and added capacity on the full test set.
Reward coefficient λ
Test PDMS ↑
Δ PDMS
0 (original reward)
88.738
—
0.5
88.749
+0.010
1.0
88.840
+0.101
2.0
88.720
−0.018
Table 8: Safety-gated progress reward ablation on the full test set. Differences are relative to λ=0 .
Figure 2: Qualitative comparison of backbone and complete-system predictions. Columns show straight driving, a left turn, a right turn, and a lane change. Green denotes the reference trajectory, dashed blue the backbone, and red CARE-Drive. Orange marks related objects; dotted vehicle outlines denote TTC projections, and black regions mark overlaps. Crosses mark off-road corners and circles the corresponding corners of the other prediction. Local enlargements highlight the geometric changes in these selected improvement cases. Scene PDMS is shown on the [0,1] scale.
Figure 3: Geometry, magnitude, and safety outcomes of correction. (a) Vehicle-footprint examples near road boundaries, with local enlargements. (b,c) Median and 90th-percentile absolute position and heading changes relative to the backbone, by prediction time. (d) Improved and degraded scene counts for collision compliance (NC), drivable-area compliance (DA), and time-to-collision compliance (TTC). The statistics use a random sample from the test set.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Updated component
Objective / role
Fixed context
Updates
Val497 PDMS
A0
Backbone
Dynamic weighted L1 + future features
Frozen visual tower
3,625
87.611
B1
Heading
Token SFT
Backbone; base XY
3,620
87.527
B2
Heading
First official-reward RL stage
Backbone; base XY
3,620
89.581
B3
Heading
Second official-reward RL stage
Backbone; base XY
3,620
90.126
C1
Position
Residual-token SFT
Backbone; B3 heading helper
3,620
90.128
C2
Position
Official reward; global RMS
Backbone; B3 heading helper
3,620
90.924
Appendix
Table 9: Training lineage of the dynamic system. Every row is a five-epoch stage. Each stage retains its final-epoch checkpoint; the table reports the corresponding validation scores. R0 later repeats C3 and D1 from the recorded dynamic parents.
System
Test PDMS
L1 backbone, epoch 4
85.988
Historical experts (H)
88.810
Cross-attention
88.726
Dynamic / R0
88.837
Appendix
Table 10: Internal system comparison on the test set. H denotes the historical expert system.
Figure 4: Test-set performance of complete systems. (a) PDMS of each fixed system. (b) Paired log-bootstrap 95% intervals for differences from the historical expert system. Intervals are computed by resampling logs for fixed trained models.
Variant
Val497
Full PDMS
Δ vs. R0
R0: original recipe
90.675
88.837
—
R1: position penalty
90.819
88.682
−0.155
R2: heading penalty
90.949
88.655
−0.182
R1 interval: [−0.383,0.078]
R2 interval: [−0.379,0.014]
Appendix
Table 11: Controlled risk experiments. All use the same dynamic backbone. Intervals are Bonferroni-adjusted 97.5% paired log-bootstrap intervals for full-split differences from R0.
Figure 5: Planning scores and compliance under safety reward penalties. (a) Both full-split differences are negative in point estimate, with two-comparison-adjusted intervals crossing zero. (b) A small reduction in collision-compliance failures in R2 accompanies more road-compliance failures. Counts indicate components below one, including partial NC penalties.
Variant
NC <1
DA <1
Mean cost
R0
204
497
0.0567
R1
208
502
0.0575
R2
201
522
0.0585
Appendix
Table 12: Safety costs and component counts on the full split. NC below one includes partial penalties.
Figure 6: Additional low-speed and lane-change examples. The upper row includes one improvement case and two cases with unchanged maximum PDMS; the lower row shows lane-change improvements. For examples labeled “All waypoints outside camera view”, the trajectory positions are shown in the bird’s-eye-view panels.
Figure 7: Additional turning examples. Right turns appear in the upper row and left turns in the lower row. Local enlargements highlight road-boundary and TTC-projection differences associated with the annotated improvements.
Figure 8: Additional miscellaneous and straight-driving examples. The upper row retains the supplied “Other” category; the lower row shows straight-driving cases.
Vision-Language-Action (VLA) models for autonomous driving rely heavily on successful expert demonstrations, leaving model-specific failures underexploited. Learning from these failures is hindered by unreliable diagnoses, poorly matched correction targets, and coarse rewards. We propose RefineDrive, a failure-guided post-training framework that learns from self-generated failures through targeted supervision and safety-aware reinforcement learning. Reliable Diagnosis derives structured, verifiable feedback on collisions and drivable-area violations directly from simulator states. Minimum-Correction Target Retrieval searches a clustered human trajectory bank for nearby corrections that satisfy hard-safety constraints in the current scene, prioritizing preservation of the failed prediction's motion pattern. Conditioned on the driving context and failed trajectory, Correction SFT learns to generate the diagnosis followed by the retrieved correction as a training-only auxiliary task. We then apply GRPO with a Safety-Layered Reward that strictly prioritizes hard-safe trajectories, retains continuous safety feedback for both unsafe and hard-safe trajectories, and rewards driving progress only after hard safety is satisfied. At inference, the policy directly predicts trajectories from the driving context without an explicit diagnosis or repair stage. On NAVSIM v1, RefineDrive improves the 4B base SFT policy from 87.7 to 91.7 PDMS. Using the same checkpoint without additional training, RefineDrive achieves 89.4 EPDMS on the original NAVTEST scenes evaluated with NAVSIM v2 extended metrics. Controlled ablations support the benefits of structured diagnosis supervision, retrieved corrections, and safety-layered optimization for direct planning.
Zhe Sun, Ziyi Luo, Yehao Lu +2
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Yinwang Intelligent Technology Co., Ltd.
Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning. However, trajectories directly generated by VLMs often encode only coarse driving intentions and remain insufficient for geometrically accurate, future-aware, and multi-view-grounded planning. To address these limitations, we develop the Layer-Wise World-Model-Guided Driving framework (LWDrive). LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance. Instead of treating the VLM output as the final trajectory, LWDrive uses it as an intent-aware coarse plan, expands a diverse candidate space around it, and progressively refines the candidates through a Foresight Cascade Planner (FCP). Specifically, we introduce future-frame generation supervision to encourage the VLM to learn forward-looking scene representations, thereby injecting planning-relevant predictive dynamics into its internal hidden states. Built upon these world-model-supervised representations, FCP exploits VLM features across multiple layers and integrates historical temporal states, Action-Query representations, and current-frame multi-view Bird's-Eye-View (BEV) features to refine candidate trajectories in a coarse-to-fine manner. This design enables progressive correction of spatial positions and motion trends while grounding trajectory refinement with multi-view scene cues and preserving the high-level driving intention produced by the large model. Finally, a score head evaluates the refined candidates and selects the best trajectory as the final planning output. Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2. Code and models will be made publicly available.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Yuguang Yang, Canyu Chen, Zhewen Tan +10
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5