Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and underutilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.
Figures & tables
Figure 1: ( Left ) Closed-loop metrics (Driving Score and Success Rate) and open-loop metric (Trajectory L2 error) for different models. GeRo consistently improves both closed-loop and open-loop performance across baselines like Qwen2.5 [ 2 ] and ORION [ 11 ] . ( Right ) GeRo scenario rollout pipeline: given scenario description and multi-view observations, GeRo generates temporally consistent agent behaviors using a large language model (LLM) across time steps.
Figure 2: Overview of GeRo’s two stage framework. ( Left ) GeRo pretraining stage of the Vision–Language–Action (VLA) model. Multi-view images and scenario prompts are encoded into visual and text tokens and projected by the LLM head to compute ego and agent tokens. These tokens are passed to the generative planner and the motion prediction head . The planning, and motion and language prediction tasks are supervised by their respective loss functions ( Lplan,Lmot,LVLA ). ( Right ) GeRo’s autoregressive rollout stage. Given a scenario description and ego-action questions, GeRo performs scenario rollout to predict future tokens for T steps, decoding them into ego waypoints, multi-agent trajectories, and language outputs. Rollouts are optimized with consistency loss ( Lroll ) and reinforcement feedback ( LGRPO ) for robust scenario grounding.
Figure 3: Autoregressive scenario rollout in GeRo. At time t , GeRo consumes ego and agent tokens along with scenario description and ego-action questions, and predicts next-step tokens using the LLM head. These tokens are decoded into ego trajectories, multi-agent trajectories, and language answers. The process repeats for T steps, with updated scenario descriptions and ego-action questions at each rollout step, enabling consistent rollouts aligned with text and environment dynamics. Rollouts are optimized using the consistency loss Lroll and the reinforcement feedback loss LGRPO for robust planning and reasoning.
Method
Condition
Aux. Sup.
Closed-loop Metric
Open-loop Metric
DS ↑
SR(%) ↑
Efficiency ↑
Comfortness ↑
Avg. L2 ↓
TCP-traj [ 52 ]
TP
E+Speed
59.90
30.00
76.54
18.08
1.70
ThinkTwice [ 22 ]
TP
E+D+S+M
62.44
31.23
69.33
16.22
0.95
DriveAdapter [ 21 ]
TP
E+S
64.22
33.08
70.22
16.01
1.01
SimLingo [ 38 ] ‡
TP
-
85.07
67.27
259.23
33.67
-
iPad [ 14 ]
TP
-
65.02
35.91
161.31
28.21
0.97
Table 1: Comparison of closed-loop and open-loop performance of end-to-end autonomous driving methods on the Bench2Drive base set. Avg. L2 denotes the average trajectory error over 2 seconds at 2 Hz, following UniAD [ 18 ] . Abbreviations: NC = Navigation Command, TP = Target Point, DS = Driving Score, SR = Success Rate, M = Map, B = Bounding boxes, D = Depth, Mo = Motion prediction, O = Occupancy, S = Segmentation, E = Expert distillation. ‡ indicates models trained with geometric path waypoint supervision and evaluated using the PID controller from SimLingo.
Method
Condition
Aux. Sup.
Ability (%) ↑
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
TCP-traj [ 52 ]
TP
E+Speed
28.84
24.29
51.67
40
46.28
34.22
ThinkTwice [ 22 ]
TP
E+D+S+M
27.38
18.42
35.82
50
54.23
37.17
DriveAdapter [ 21 ]
TP
E+S
28.38
28.38
47.50
50
56.43
42.14
SimLingo [ 38 ] ‡
TP
-
54.01
57.04
88.33
53
82.45
67.03
UniAD-Base [ 18 ]
NC
M+B+Mo+O
8.89
9.33
20
20
14.21
14.49
Table 2: Multi-ability performance Comparison for E2E-AD methods on Bench2Drive base set. NC = Navigation Command, TP = Target Point, M = Map, B = Bounding boxes, D = Depth, Mo = Motion prediction, O = Occupancy, S = Segmentation, E = Expert distillation. ‡ indicates models trained with geometric path waypoint supervision and evaluated using the PID controller from SimLingo.
Figure 4: Qualitative examples of language-guided scenario rollouts using proposed GeRo on Bench2Drive. Top: Intersection scenarios with STOP signs and pedestrian interactions, where the ego vehicle demonstrates cautious deceleration, yielding, and safe left-turn. Bottom: Accident-handling scenarios under adverse weather, showing proactive hazard detection, lane-change planning, and smooth merging. Each frame includes generated textual reasoning aligned with the ego actions, highlighting temporal consistency and safety-aware decision.
Method
Inference Rollout
Closed-loop
Open-loop
Latency (second/frame) ↓
DS ↑
SR(%) ↑
Ego Avg. L2 ↓
Agent minADE ↓
Agent minFDE ↓
ORION [ 11 ]
No
76.9
53.9
0.63
4.16
4.85
0.292
ORION [ 11 ]
Yes
77.7
54.4
0.61
3.67
4.17
0.918
GeRo (ORION)
No
81.9
60.1
0.57
1.88
2.08
0.306
GeRo (ORION)
Yes
85.7
63.9
0.46
1.10
1.36
0.940
Table 3: Driving performance using inference-time autoregressive rollout on Bench2Drive. Driving Score (DS) and Success Rate (SR) measure closed-loop driving. Ego Avg. L2 and agent minADE/minFDE measure open-loop prediction errors. ‡ indicates using geometric path waypoint supervision and SimLingo PID controller.
Table 8
ID
Sc Desc
Q
Pretraining
Scenario Generation Rollout
Closed-loop metric
Lplan
Lmot
LVQA
Lroll
LGRPO
DS
SR
Ltc
Lplan
Lmot
Coll
TTC
LVLA
1
✗
✗
✓
58.2
27.4
2
✗
✗
✓
✓
61.1
28.2
3
✓
✗
✓
✓
✓
63.9
31.6
4
✓
✗
✓
✓
✓
✓
67.8
40.1
Table 6: Ablation study of GeRo (Qwen) on Bench2Drive. Baseline: Qwen2.5-3B. Metrics: Driving Score (DS) and Success Rate (SR).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Generative Planner
Closed-loop
Open-loop
Multi-Ability Avg.
DS ↑
SR(%) ↑
Avg. L2 (m) ↓
ORION w/ MLP
70.73
41.52
0.75
48.44
ORION w/ Diffusion
71.97
46.54
0.73
46.68
ORION w/ VAE
77.74
54.62
0.68
54.72
GeRo (ORION) w/ MLP
77.21
56.72
0.62
62.38
GeRo (ORION) w/ Diffusion
80.75
59.22
0.57
65.43
Appendix
Table 7: Ablation comparison with diverse planners. DS and SR denote Driving Score and Success Rate respectively.
Generative Planner
Closed-loop
Multi-Ability Avg.
DS ↑
SR(%) ↑
Qwen2.5
63.90
31.60
25.60
GeRo (Qwen2.5) w/o stage-1 pretraining
71.67
48.95
53.7
GeRo (Qwen2.5) w/ stage-1 pretraining
79.60
57.80
61.98
Appendix
Table 8: Ablation comparison on stage-1 pretraning. DS and SR denote Driving Score and Success Rate respectively.
Stage
Model / added objective
Avg. L2 ↓
Collision rate ↓
Lpre
Qwen2.5
0.96
0.60
Stage-2 Lroll
+Ltc
0.61
0.31
Stage-2 Lroll
+Lmot
0.54
0.28
Stage-2 Lroll
+Lplan
0.39
0.20
Stage-2 LGRPO
+LGRPO
0.31
0.14
Appendix
Table 9: Training ablation of GeRo (Qwen2.5) on nuScenes validation under the PARA-Drive/TOKEN evaluation protocol. Baseline: Qwen2.5-3B-VL. Objectives are added cumulatively; lower is better.
Metric
T=4
T=8
T=12
T=16
Driving Score ↑
78.31
79.60
80.15
79.08
Success Rate ↑
56.75
57.80
58.01
57.17
Appendix
Table 10: Ablation on T , the number of rollout steps in scenario generation.
Method
Ego-Action
Scenario Description
GPT score ↑
CIDEr ↑
BLEU ↑
GPT score ↑
CIDEr ↑
Qwen2.5
70.82
67.90
53.69
68.16
72.21
GeRo (Qwen2.5)
77.54
75.21
65.38
76.47
83.01
Appendix
Table 12: Language performance on Bench2Drive for ego-action QA and scenario description.
Figure 5: An example of the text-description of the numerical annotations of a scene from Bench2Drive. This is appended to the user prompt for Qwen3-VL-32B-VL.
Figure 6: Some examples of generated QA pairs. The perception, prediction, and planning pairs are generated with Llama-3. The behavior QA is created using the future motion of the ego vehicle.
Figure 7: Qualitative visualization of two complex urban driving scenarios under adverse weather conditions. Top: illustrates a left-turn maneuver at an intersection with limited visibility, requiring cautious execution and yielding to oncoming traffic. Bottom: depicts a pedestrian-crossing scenario where the ego vehicle detects a fast-approaching pedestrian, performs an emergency stop, and resumes motion safely after the pedestrian clears the road. These examples highlight the GeRo’s ability to handle dynamic and safety-critical situations.
Figure 8: Failure case under extreme fog conditions. The ego vehicle initially performs a successful lane change to prepare for a highway exit (top row). However, due to low visibility, it confuses the shoulder lane with the exit lane and deviates from its route for approximately 23.476 meters (bottom row). The vehicle later corrects its trajectory and merges back into the exit lane.
Method
Inference
Time (s) ↓
TFLOPs ↓
ORION
0.292
0.796
GeRo (ORION)
0.306
0.798
Appendix
Table 13: Per-frame inference cost on Bench2Drive (validation). Time is measured in seconds per frame; compute is reported in TFLOPs.
Method
Meta ↑
Kinematic ↑
Interactive ↑
ORION
0.391
0.263
0.518
SimLingo
0.421
0.368
0.573
DeepSight [ 56 ]
0.490
0.413
0.608
GeRo
0.553
0.454
0.620
Appendix
Table 14: Latent rollout realism on Bench2Drive using adapted Waymo Sim Agents metrics. Higher values indicate better reference-distribution agreement.