Recent progress in integrating vision-language-action (VLA) models with world modeling has significantly advanced end-to-end autonomous driving by allowing systems to reason about imagined futures instead of merely reacting to current observations. However, most world VLA models generate future frames at the pixel level with plenty of dense visual tokens. Although such generated futures can be visually plausible, they preserve many driving-irrelevant appearance details, i.e., only a small portion of the tokens directly encode critical planning-relevant factors, such as agent geometry, motion dynamics, and interactions. This introduces a significant performance bottleneck. An important question thus arises: what world state should a world VLA model reason over to support planning beyond pixel-level prediction? We argue that planning-oriented world modeling should focus on a compact and structured agent-centric latent state that captures the interactions between objects in the driving scene over time. As such, we introduce AgentWorld, an agent-centric VLA world model that uses discrete agent tokens jointly for reasoning and trajectory planning. Each token is dynamically grounded to a surrounding agent associated with its object's state, such as semantic identity, 3D geometry, motion, and interaction context. AgentWorld learns these tokens through three key processes: explicit geometric grounding, reasoning-oriented supervised fine-tuning, and agent-oriented reinforcement learning. Extensive open-loop and closed-loop experiments under diverse driving scenarios show that AgentWorld enhances planning capabilities, fully exploits visual evidence and agent dynamics, and yields interpretable agent-centric world states.
Figures & tables
Figure 1: (a) VLA models output actions from observations and prompts, lacking explicit spatiotemporal modeling and interaction awareness. (b) World models learn predictive latent futures, yet their imagined states fail to align with causal reasoning. (c) VLA World models integrate imagination and language reasoning, but reason over generated frames rather than interactive agents for driving decisions. (d) By contrast, AgentWorld models futures via discrete latent agent states, enabling VLA reasoning over physically grounded 3D geometry, motion dynamics and agent interactions.
Figure 2: Visual grounding results (UniAD metric) and token efficiency of VLA models.
Figure 3: Overview of AgentWorld. Given multi-view observations, ego-status, navigation goals, and language prompts, AgentWorld constructs an agent-centric world state by grounding discrete agent tokens with 4D geometry and 2D visual features through multimodal routed anchoring. The grounded agent tokens are then used by the policy model for reasoning, action, and planning through grounding, supervised fine-tuning, and reinforcement learning stages.
Figure 4: The pipeline of the multimodal routed anchoring. Agent queries are routed to geometry and vision branches, and fused through cross-attention to produce grounded agent tokens.
Method
ST-P3 metrics
UniAD metrics
VLM
L2 (m) ↓
Collision (%) ↓
L2 (m) ↓
Collision (%) ↓
1s
2s
3s
Avg.
1s
2s
3s
Avg.
1s
2s
3s
Avg.
1s
2s
3s
Avg.
ELM [ 94 ]
-
-
-
-
-
-
-
-
0.34
1.23
2.57
1.38
0.12
0.50
2.36
0.99
BLIP2-2.7B
FeD* [ 95 ]
-
-
-
-
-
-
-
-
0.27
0.53
0.94
0.58
0.00
0.04
0.52
0.19
LLaVA-7B
OccWorld [ 56 ]
0.39
0.73
1.18
0.77
0.11
0.19
0.67
0.32
0.52
1.27
2.41
1.40
0.12
0.40
2.08
0.87
GPT3-like
Doe-1 [ 55 ]
0.37
0.67
1.07
0.70
0.02
0.14
0.47
0.21
0.50
1.18
2.11
1.26
0.04
0.37
1.19
0.53
Lumina-mGPT-7B
Table 1: End-to-end trajectory planning results on nuScenes [ 92 ] . We evaluate L2 error and collision rate following the respective protocols of ST-P3 [ 93 ] and UniAD [ 18 ] . The ∗ indicates the use of additional ego-state information.
Table 6
Figure 5: Qualitative comparison between VLA-World [ 30 ] and AgentWorld. For each scene, the left panels visualize the future ego trajectories of VLA-World (top) and AgentWorld (bottom), with AgentWorld additionally showing the traffic participants aligned with the special tokens ⟨agent_i⟩ . The right panels present the corresponding perception, reasoning, and planning results.
Method
Driving Score ↑
Success Rate (%) ↑
Efficiency ↑
Comfortness ↑
AD-MLP [ 103 ]
18.05
0.00
48.45
22.63
UniAD-Base [ 18 ]
45.81
16.36
129.21
43.58
VAD [ 19 ]
42.35
15.00
157.94
46.01
TCP-traj [ 104 ]
59.90
30.00
76.54
18.08
DriveAdapter [ 105 ]
64.22
33.08
70.22
16.01
Orion [ 106 ]
77.74
54.62
151.48
17.38
Table 4: Closed-loop evaluation results on Bench2Drive (CARLA) with SOTA methods.
Table 9
Figure 6: Ablation studies on agent-centric world states. (a) Perception and (b) planning performance with varying numbers of agent tokens. (c) Performance comparisons of w/ and w/o ego-status.
Table 11
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Example of the Stage-1 agent grounding QA template. Given multi-view driving observations and a grounding instruction, AgentWorld outputs a fixed set of <agent_i> tokens inside the <perception> field. These tokens act as latent agent slots: active slots are supervised to align with physical traffic participants through 3D box and future trajectory targets, while unused slots are trained as no-object.
Figure 8: Prompt design for reasoning with agent data generation.
Figure 9: Example of the Stage-2 reasoning-oriented SFT template. Given multi-view driving observations, ego-state instructions, and the fixed <agent_i> slots in the <perception> field, AgentWorld learns to generate a structured response consisting of concise agent-grounded reasoning in <think> , a high-level driving command in <action> , and future ego waypoints in <answer> . The rationale explicitly references decision-relevant agent tokens, encouraging the policy to plan based on surrounding-agent interactions rather than shortcut cues.
Table 15
Figure 10: Distribution of object counts.
Figure 11: Visualization of trajectories across VLA-World and our AgentWorld on the nuScenes validation dataset. The predicted and ground-truth trajectories are depicted in red and green, respectively.
Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io
Guoqing Wang, Pin Tang, Xiangxuan Ren +3
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei
Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving. While pixel-level future prediction enables fine-grained spatiotemporal reasoning, it compromises robustness in noisy driving scenarios. Conversely, latent-based world models alleviate this sensitivity but often incur limited interpretability and representational degradation due to absent pixel-level grounding. To reconcile this trade-off, we propose HyWorldVLA, a hybrid world-VLA framework that unifies pixel-level supervision and latent representation learning. In the pre-training stage, HyWorldVLA predicts video latents encoded by a pre-trained video VAE, while simultaneously reconstructing video frames to provide precise pixel-level grounding. During the subsequent co-fine-tuning phase, the model exclusively predicts latent features, which are fed into an action expert to generate trajectories. Extensive experiments on NAVSIM v1 and v2 benchmarks demonstrate that HyWorldVLA significantly outperforms both pixel-based and latent-based world model baselines. Notably, we present the first comprehensive qualitative and quantitative analysis of world model noise robustness in autonomous driving, establishing a new benchmark for evaluating future architectures.
Quanfu Yu, Xian Wu, Hao Xu +1
Automotive New Technology Research Institute, BYD Company Limited
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing reasoning mechanisms still struggle to provide planning-oriented intermediate representations: textual Chain-of-Thought (CoT) fails to preserve continuous spatiotemporal structure, while latent world reasoning remains difficult to use as a direct condition for action generation. In this paper, we propose CoWorld-VLA, a multi-expert world reasoning framework for autonomous driving, where world representations serve as explicit conditions to guide action planning. CoWorld-VLA extracts complementary world information through multi-source supervision and encodes it into expert tokens within the VLA, thereby providing planner-accessible conditioning signals. Specifically, we construct four types of tokens: semantic interaction, geometric structure, dynamic evolution, and ego trajectory tokens, which respectively model interaction intent, spatial structure, future temporal dynamics, and behavioral goals. During action generation, CoWorld-VLA employs a diffusion-based hierarchical multi-expert fusion planner, which is coupled with scene context throughout the joint denoising process to generate continuous ego trajectories. Experiments show that CoWorld-VLA achieves competitive results in both future scene generation and planning on the NAVSIM v1 benchmark, demonstrating strong performance in collision avoidance and trajectory accuracy. Ablation studies further validate the complementarity of expert tokens and their effectiveness as planning conditions for action generation. Code will be available at https://github.com/AFARI-Research/CoWorld-VLA.
Minqing Huang, Yujiao Xiang, Zihan Liang +7
Afari Intelligent Drive · University of Electronic Science and Technology of China · Shanghai Jiao Tong University +2