Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and underutilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.
Figures & tables
Figure 1: ( Left ) Closed-loop metrics (Driving Score and Success Rate) and open-loop metric (Trajectory L2 error) for different models. GeRo consistently improves both closed-loop and open-loop performance across baselines like Qwen2.5 [ 2 ] and ORION [ 11 ] . ( Right ) GeRo scenario rollout pipeline: given scenario description and multi-view observations, GeRo generates temporally consistent agent behaviors using a large language model (LLM) across time steps.
Figure 2: Overview of GeRo’s two stage framework. ( Left ) GeRo pretraining stage of the Vision–Language–Action (VLA) model. Multi-view images and scenario prompts are encoded into visual and text tokens and projected by the LLM head to compute ego and agent tokens. These tokens are passed to the generative planner and the motion prediction head . The planning, and motion and language prediction tasks are supervised by their respective loss functions ( Lplan,Lmot,LVLA ). ( Right ) GeRo’s autoregressive rollout stage. Given a scenario description and ego-action questions, GeRo performs scenario rollout to predict future tokens for T steps, decoding them into ego waypoints, multi-agent trajectories, and language outputs. Rollouts are optimized with consistency loss ( Lroll ) and reinforcement feedback ( LGRPO ) for robust scenario grounding.
Figure 3: Autoregressive scenario rollout in GeRo. At time t , GeRo consumes ego and agent tokens along with scenario description and ego-action questions, and predicts next-step tokens using the LLM head. These tokens are decoded into ego trajectories, multi-agent trajectories, and language answers. The process repeats for T steps, with updated scenario descriptions and ego-action questions at each rollout step, enabling consistent rollouts aligned with text and environment dynamics. Rollouts are optimized using the consistency loss Lroll and the reinforcement feedback loss LGRPO for robust planning and reasoning.
Method
Condition
Aux. Sup.
Closed-loop Metric
Open-loop Metric
DS ↑
SR(%) ↑
Efficiency ↑
Comfortness ↑
Avg. L2 ↓
TCP-traj [ 52 ]
TP
E+Speed
59.90
30.00
76.54
18.08
1.70
ThinkTwice [ 22 ]
TP
E+D+S+M
62.44
31.23
69.33
16.22
0.95
DriveAdapter [ 21 ]
TP
E+S
64.22
33.08
70.22
16.01
1.01
SimLingo [ 38 ] ‡
TP
-
85.07
67.27
259.23
33.67
-
iPad [ 14 ]
TP
-
65.02
35.91
161.31
28.21
0.97
Table 1: Comparison of closed-loop and open-loop performance of end-to-end autonomous driving methods on the Bench2Drive base set. Avg. L2 denotes the average trajectory error over 2 seconds at 2 Hz, following UniAD [ 18 ] . Abbreviations: NC = Navigation Command, TP = Target Point, DS = Driving Score, SR = Success Rate, M = Map, B = Bounding boxes, D = Depth, Mo = Motion prediction, O = Occupancy, S = Segmentation, E = Expert distillation. ‡ indicates models trained with geometric path waypoint supervision and evaluated using the PID controller from SimLingo.
Method
Condition
Aux. Sup.
Ability (%) ↑
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
TCP-traj [ 52 ]
TP
E+Speed
28.84
24.29
51.67
40
46.28
34.22
ThinkTwice [ 22 ]
TP
E+D+S+M
27.38
18.42
35.82
50
54.23
37.17
DriveAdapter [ 21 ]
TP
E+S
28.38
28.38
47.50
50
56.43
42.14
SimLingo [ 38 ] ‡
TP
-
54.01
57.04
88.33
53
82.45
67.03
UniAD-Base [ 18 ]
NC
M+B+Mo+O
8.89
9.33
20
20
14.21
14.49
Table 2: Multi-ability performance Comparison for E2E-AD methods on Bench2Drive base set. NC = Navigation Command, TP = Target Point, M = Map, B = Bounding boxes, D = Depth, Mo = Motion prediction, O = Occupancy, S = Segmentation, E = Expert distillation. ‡ indicates models trained with geometric path waypoint supervision and evaluated using the PID controller from SimLingo.
Figure 4: Qualitative examples of language-guided scenario rollouts using proposed GeRo on Bench2Drive. Top: Intersection scenarios with STOP signs and pedestrian interactions, where the ego vehicle demonstrates cautious deceleration, yielding, and safe left-turn. Bottom: Accident-handling scenarios under adverse weather, showing proactive hazard detection, lane-change planning, and smooth merging. Each frame includes generated textual reasoning aligned with the ego actions, highlighting temporal consistency and safety-aware decision.
Method
Inference Rollout
Closed-loop
Open-loop
Latency (second/frame) ↓
DS ↑
SR(%) ↑
Ego Avg. L2 ↓
Agent minADE ↓
Agent minFDE ↓
ORION [ 11 ]
No
76.9
53.9
0.63
4.16
4.85
0.292
ORION [ 11 ]
Yes
77.7
54.4
0.61
3.67
4.17
0.918
GeRo (ORION)
No
81.9
60.1
0.57
1.88
2.08
0.306
GeRo (ORION)
Yes
85.7
63.9
0.46
1.10
1.36
0.940
Table 3: Driving performance using inference-time autoregressive rollout on Bench2Drive. Driving Score (DS) and Success Rate (SR) measure closed-loop driving. Ego Avg. L2 and agent minADE/minFDE measure open-loop prediction errors. ‡ indicates using geometric path waypoint supervision and SimLingo PID controller.
Table 8
ID
Sc Desc
Q
Pretraining
Scenario Generation Rollout
Closed-loop metric
Lplan
Lmot
LVQA
Lroll
LGRPO
DS
SR
Ltc
Lplan
Lmot
Coll
TTC
LVLA
1
✗
✗
✓
58.2
27.4
2
✗
✗
✓
✓
61.1
28.2
3
✓
✗
✓
✓
✓
63.9
31.6
4
✓
✗
✓
✓
✓
✓
67.8
40.1
Table 6: Ablation study of GeRo (Qwen) on Bench2Drive. Baseline: Qwen2.5-3B. Metrics: Driving Score (DS) and Success Rate (SR).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Generative Planner
Closed-loop
Open-loop
Multi-Ability Avg.
DS ↑
SR(%) ↑
Avg. L2 (m) ↓
ORION w/ MLP
70.73
41.52
0.75
48.44
ORION w/ Diffusion
71.97
46.54
0.73
46.68
ORION w/ VAE
77.74
54.62
0.68
54.72
GeRo (ORION) w/ MLP
77.21
56.72
0.62
62.38
GeRo (ORION) w/ Diffusion
80.75
59.22
0.57
65.43
Appendix
Table 7: Ablation comparison with diverse planners. DS and SR denote Driving Score and Success Rate respectively.
Generative Planner
Closed-loop
Multi-Ability Avg.
DS ↑
SR(%) ↑
Qwen2.5
63.90
31.60
25.60
GeRo (Qwen2.5) w/o stage-1 pretraining
71.67
48.95
53.7
GeRo (Qwen2.5) w/ stage-1 pretraining
79.60
57.80
61.98
Appendix
Table 8: Ablation comparison on stage-1 pretraning. DS and SR denote Driving Score and Success Rate respectively.
Stage
Model / added objective
Avg. L2 ↓
Collision rate ↓
Lpre
Qwen2.5
0.96
0.60
Stage-2 Lroll
+Ltc
0.61
0.31
Stage-2 Lroll
+Lmot
0.54
0.28
Stage-2 Lroll
+Lplan
0.39
0.20
Stage-2 LGRPO
+LGRPO
0.31
0.14
Appendix
Table 9: Training ablation of GeRo (Qwen2.5) on nuScenes validation under the PARA-Drive/TOKEN evaluation protocol. Baseline: Qwen2.5-3B-VL. Objectives are added cumulatively; lower is better.
Metric
T=4
T=8
T=12
T=16
Driving Score ↑
78.31
79.60
80.15
79.08
Success Rate ↑
56.75
57.80
58.01
57.17
Appendix
Table 10: Ablation on T , the number of rollout steps in scenario generation.
Method
Ego-Action
Scenario Description
GPT score ↑
CIDEr ↑
BLEU ↑
GPT score ↑
CIDEr ↑
Qwen2.5
70.82
67.90
53.69
68.16
72.21
GeRo (Qwen2.5)
77.54
75.21
65.38
76.47
83.01
Appendix
Table 12: Language performance on Bench2Drive for ego-action QA and scenario description.
Figure 5: An example of the text-description of the numerical annotations of a scene from Bench2Drive. This is appended to the user prompt for Qwen3-VL-32B-VL.
Figure 6: Some examples of generated QA pairs. The perception, prediction, and planning pairs are generated with Llama-3. The behavior QA is created using the future motion of the ego vehicle.
Figure 7: Qualitative visualization of two complex urban driving scenarios under adverse weather conditions. Top: illustrates a left-turn maneuver at an intersection with limited visibility, requiring cautious execution and yielding to oncoming traffic. Bottom: depicts a pedestrian-crossing scenario where the ego vehicle detects a fast-approaching pedestrian, performs an emergency stop, and resumes motion safely after the pedestrian clears the road. These examples highlight the GeRo’s ability to handle dynamic and safety-critical situations.
Figure 8: Failure case under extreme fog conditions. The ego vehicle initially performs a successful lane change to prepare for a highway exit (top row). However, due to low visibility, it confuses the shoulder lane with the exit lane and deviates from its route for approximately 23.476 meters (bottom row). The vehicle later corrects its trajectory and merges back into the exit lane.
Method
Inference
Time (s) ↓
TFLOPs ↓
ORION
0.292
0.796
GeRo (ORION)
0.306
0.798
Appendix
Table 13: Per-frame inference cost on Bench2Drive (validation). Time is measured in seconds per frame; compute is reported in TFLOPs.
Method
Meta ↑
Kinematic ↑
Interactive ↑
ORION
0.391
0.263
0.518
SimLingo
0.421
0.368
0.573
DeepSight [ 56 ]
0.490
0.413
0.608
GeRo
0.553
0.454
0.620
Appendix
Table 14: Latent rollout realism on Bench2Drive using adapted Waymo Sim Agents metrics. Higher values indicate better reference-distribution agreement.
Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io
Guoqing Wang, Pin Tang, Xiangxuan Ren +3
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei
Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for vision-language understanding and reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal evolution of driving scenes, allowing agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-language instructions, LMGenDrive generates both future driving videos and control signals. This design provides complementary benefits: video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying multimodal understanding and generation is a promising direction for more generalizable and robust embodied decision-making systems.
Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.
Shuzheng Wang, Haoxuan Li, Shengzhen Li +5
Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.