Current Vision-Language-Action (VLA) paradigms in end-to-end autonomous driving rely on offline training from static datasets, leaving them vulnerable to distribution shift. Recent post-training methods use takeover data to mitigate this by augmenting the dataset with high-quality expert takeover samples, yet they suffer from two key limitations: supervision restricted to the period after the takeover moments leads to policies with limited safety margins, and passive preference optimization lacks active exploration for optimal performance. In this paper, we propose TakeVLA, a novel VLA post-training framework that overcomes these shortcomings through two complementary innovations. First, we introduce pre-takeover language supervision, which allows the VLA to learn from mistakes proactively. By explicitly teaching the model about what to do in error-prone situations, we cultivate a precautionary mindset that anticipates hazards early and substantially enlarges safety margins. Second, we propose Scenario Dreaming, a reinforcement fine-tuning paradigm that operates in reconstruceted takeover scenarios, encouraging active exploration beyond mere preference fitting. Experiments on the Bench2Drive benchmark demonstrate that TakeVLA achieves state-of-the-art closed-loop performance, surpassing the strong VLA baseline SimLingo by 4.93 in driving score, with an enhanced safety margin as evidenced by an 11.76% increase in average TTC.
Figures & tables
Figure 1: We propose TakeVLA , a post-training framework for driving VLA with takeover data. (a) TakeVLA iteratively collects takeover data via online expert interventions and conducts post-training on the collected data. It introduces pre-takeover supervision for larger safety margins and Scenario Dreaming for active exploration toward optimal policy. (b) t-SNE visualization shows that the takeover dataset ( red ) provides valuable OOD samples compared to the pretrain dataset ( blue ). (c) Closed-loop evaluation on Bench2Drive demonstrates TakeVLA’s superior performance, with a higher driving score, success rate, and significantly improved safety margins.
Figure 2: Post-training pipiline for TakeVLA. Starting from a pre-trained VLA model, each round consists of three key steps: (1) online interaction with expert takeovers to construct pre-takeover and takeover datasets, (2) supervised fine-tuning on the constructed datasets, and (3) reinforcement fine-tuning via Scenario Dreaming in reconstructed takeover scenarios. Multiple rounds progressively enhance language-conditioned driving performance.
Figure 3: Architecture of the baseline VLA model.
Figure 4: Language label enhancement. For each takeover trigger type (Follow, Collision, Restart), original language labels (middle) are refined (bottom) based on the specific cause. Enhanced labels emphasize urgency and explicit causality, leading to more conservative and effective driving actions in critical scenarios.
Methods
Reference
Closed-loop Metric ↑
Multi-ability (%) ↑
DS
SR (%)
Efficiency
Comfortness
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
End-to-end Methods
UniAD [ 16 ]
CVPR 23
45.81
16.36
129.21
43.58
12.16
20.00
23.64
10.00
13.89
15.89
VAD [ 19 ]
ICCV 23
42.35
15.00
157.94
46.01
7.14
20.00
16.36
20.00
20.22
16.75
SparseDrive [ 39 ]
ICRA 25
44.54
16.71
170.21
48.63
12.18
23.19
17.91
20.00
20.98
17.45
DriveTrans.-L. [ 18 ]
ICLR 25
63.46
35.01
100.64
20.78
17.57
35.00
48.36
40.00
52.10
38.60
Table 1: Closed-loop and Multi-ability performance of SoTA methods on the Bench2Drive leaderboard.
Methods
Closed-loop Metric
DS ↑
SR (%) ↑
TR (%) ↓
SimLingo †
79.19
55.50
12.39
TakeVLA ( Ours ) †
86.44
68.20
9.22
Table 2: Closed-loop performance without rule-based post-process.
ID
Post-training Techniques
Closed-loop Metric ↑
Takeover Data
Label Enhancement
Pre-takeover
Sceniario Dreaming
DS
SR (%)
TTC (s)
1
✗
✗
✗
✗
84.79
64.84
7.23
2
✓
✗
✗
✗
86.22
68.52
7.56
3
✓
✓
✗
✗
86.73
68.04
7.53
4
✓
✓
✓
✗
87.25
68.35
7.99
5
✓
✓
✓
✓
87.79
68.81
8.16
Table 3: Ablation study on post-training techniques of TakeVLA.
ID
Group
Reward terms
Closed-loop Metric ↑
size G
Distance
Collision
DS
SR (%)
TTC (s)
1
2
✓
✓
86.88
68.35
8.05
2
4
✓
✓
87.33
67.28
8.01
3
6
✓
✓
87.48
69.41
8.19
4
8
✓
✓
87.79
68.81
8.16
5
8
✓
✗
87.22
68.81
7.54
Table 4: Effects of group size and reward design for GRPO training.
Figure 7: Qualitative visualization of SimLingo and TakeVLA . Each column shows a selected frame in temporal order. Driving action ad is drawn on the front-view image in the form of speed and path waypoints. The PID control signals are generated from ad . We omit the language input ol for brevity. SimLingo generates insufficient language outputs, leading to unsafe lane changes and rear-end collisions. TakeVLA generates more anticipatory and context-aware language outputs, enabling safer gap selection, timely acceleration, and proactive deceleration to avoid collisions.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for end-to-end autonomous driving by enabling unified reasoning across perception, language, and planning. However, existing approaches lack mechanisms to exploit past failures or adapt to distribution shifts, causing the model to persistently underperform on similar scenarios where it has previously failed. In this paper, we propose DriveVLA-M0, a retrieval-augmented VLA with failure-aware latent memory. We construct a latent memory pool that stores failure cases along with their structure scene representations and expert trajectory labels, and design a dedicated Retrieve Model that decouples static road structure and dynamic agent interactions to enable structurally grounded retrieval. At inference time, retrieved cases are injected into the model via a lightweight decoupled LoRA-based test-time training (TTT) mechanism, allowing targeted and scenario-specific correction without modifying the backbone. Extensive experiments on NAVSIMv1 and NAVSIMv2 benchmark demonstrate that our approach consistently outperforms prior methods, achieving 94.1 PDMS on Navtest and 47.0 EPDMS on Navhard with only 26.44 ms TTT backward latency overhead. Furthermore, we show that DriveVLA-M0 scales effectively with additional memory, enabling training-free performance gains through memory expansion. The code is available at https://github.com/ZebinX/DriveVLA-M0.
Zebin Xing, Yupeng Zheng, Qiang Chen +10
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Chongqing Chang’an Technology Co., Ltd., Chongqing, China
Vision-Language-Action (VLA) models for autonomous driving rely heavily on successful expert demonstrations, leaving model-specific failures underexploited. Learning from these failures is hindered by unreliable diagnoses, poorly matched correction targets, and coarse rewards. We propose RefineDrive, a failure-guided post-training framework that learns from self-generated failures through targeted supervision and safety-aware reinforcement learning. Reliable Diagnosis derives structured, verifiable feedback on collisions and drivable-area violations directly from simulator states. Minimum-Correction Target Retrieval searches a clustered human trajectory bank for nearby corrections that satisfy hard-safety constraints in the current scene, prioritizing preservation of the failed prediction's motion pattern. Conditioned on the driving context and failed trajectory, Correction SFT learns to generate the diagnosis followed by the retrieved correction as a training-only auxiliary task. We then apply GRPO with a Safety-Layered Reward that strictly prioritizes hard-safe trajectories, retains continuous safety feedback for both unsafe and hard-safe trajectories, and rewards driving progress only after hard safety is satisfied. At inference, the policy directly predicts trajectories from the driving context without an explicit diagnosis or repair stage. On NAVSIM v1, RefineDrive improves the 4B base SFT policy from 87.7 to 91.7 PDMS. Using the same checkpoint without additional training, RefineDrive achieves 89.4 EPDMS on the original NAVTEST scenes evaluated with NAVSIM v2 extended metrics. Controlled ablations support the benefits of structured diagnosis supervision, retrieved corrections, and safety-layered optimization for direct planning.
Zhe Sun, Ziyi Luo, Yehao Lu +2
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Yinwang Intelligent Technology Co., Ltd.
Recent advances in vision language action (VLA) models have shown remarkable potential for autonomous driving by directly mapping multimodal inputs to control signals. However, previous VLA-based methods have not explicitly exploited the critic capability of VLAs to refine driving decisions, even though such capability has been well demonstrated in other LLM-based domains, thereby limiting their performance in complex closed-loop scenarios. In this work, we present a theoretically inspired two-stage framework, CriticVLA, which extends the role of VLAs from acting to judging. CriticVLA first generates a rough trajectory and then refines it through multimodal evaluation and single-step optimization guided by a VLA-based critic, yielding higher-quality driving behaviors. To support this process, we construct a large-scale synthetic dataset of 12.9 million annotated trajectories covering diverse driving scenarios, which enhances the critic's reasoning and refinement abilities. Extensive closed-loop experiments on the Bench2Drive benchmark show that CriticVLA significantly surpasses state-of-the-art baselines, achieving a 73.33% total success rate and delivering about 30% improvement in challenging scenarios.