PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving

Jinchang Xu; Hongda Yu; Fengwei Dong; Wenhui Huang; Xi Wei; Yongzhi Liu; Sunan Zhang; Jirao Wang

cs.ROOct 8, 2026

PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving

Authors: Jinchang Xu, Hongda Yu, Fengwei Dong, Wenhui Huang, Xi Wei, Yongzhi Liu, Sunan Zhang, Jirao Wang, +4 more

Organizations: Southeast University · NIO · Nanyang Technological University · Tongji University

Abstract

World models in end-to-end autonomous driving predict future scene evolution to provide foresight for trajectory planning. Existing methods mainly study how to predict the future and how to use it, but less often ask which future representation is actually most useful for planning. To this end, we propose PlanWAM, a Planning-Shaped World Action Model. The key idea is to let the planning task shape the future-state representation, so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning. Specifically, we first use a Temporal Register Pyramid to compress multi-frame historical information in a recency-aware manner, learning a compact history representation oriented toward future reasoning and planning. We then introduce a privileged future posterior branch that observes ground-truth future frames, and shape its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation. Hindsight-to-Foresight Distillation trains a prior branch that depends only on history to predict this future latent representation. The predicted future latent representation serves as planning context and guides trajectory generation and selection. PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations. Extensive experiments further demonstrate that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.

Figures & tables

Figure 1: Four future-modeling paradigms in driving World Action Models (WAMs). (a) Pixel reconstruction. (b) Latent prediction. (c) Future-conditioned planning. (d) PlanWAM (ours).

Figure 2: Overview of PlanWAM . Historical multi-view observations are compressed into a compact representation for future reasoning. The privileged posterior branch uses ground-truth future observations and planning objectives to learn planning-shaped future latents, which are distilled into a prior branch conditioned only on history. The predicted future latents then serve as foresight for trajectory generation and selection. The posterior is training-only; inference uses only the prior.

Figure 3: Temporal Register Pyramid for Recency-Aware History Representation.

Method	Reference	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Human Agent	-	100.0	100.0	99.8	100.0	87.4	100.0	100.0	98.1	90.1	90.3
E2E-based planners
TransFuser Chitta et al. (2023)	TPAMI’23	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
DiffusionDrive Liao et al. (2025)	CVPR’25	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
MeanFuser Wang et al. (2026a)	CVPR’26	98.3	97.2	99.6	99.8	87.6	97.4	97.3	98.3	88.2	89.5
VLM-based planners

Table 1: Performance on the NAVSIM-v2 navtest benchmark. The best learned result in each column is shown in bold .

Table 5

Figure 4: Qualitative planning results. (a) Comparison with DrivoR on representative NAVSIM scenarios. (b) Zero-shot closed-loop rollouts on HUGSIM across different driving domains.

ID	TRP	H2F	FCP	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
0	$\times$	$\times$	$\times$	98.3	98.9	94.6	100	90.5	93.1
1	$\checkmark$	$\times$	$\times$	98.8	98.8	95.3	100	90.9	93.4
2	$\checkmark$	$\checkmark$	$\times$	98.6	98.6	95.5	100	91.5	93.5
3	$\checkmark$	$\checkmark$	$\checkmark$	98.9	98.8	96.3	100	91.0	93.8

Table 4: Overall component ablation of PlanWAM.

Table 8

Table 9

Figure 5: Analysis of failure reduction enabled by planning-shaped foresight. (a) Failure Type Distribution. (b) Attribution of Failure Reduction.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Additional NAVSIM results (set 1). Front view, ground-truth trajectory, and PlanWAM candidates.

Figure 8: Additional NAVSIM results (set 2). Curved roads and intersections.

Figure 9: Additional NAVSIM results (set 3). Urban scenes with parked and moving traffic.

Figure 10: Additional NAVSIM results (set 4). Dense multi-lane traffic. Gray curves are candidates; red is the selected plan.

Figure 11: Zero-shot HUGSIM rollouts on nuScenes. Two closed-loop cases are shown over six consecutive planning steps.

Figure 12: Zero-shot HUGSIM rollouts on KITTI-360. Two closed-loop cases are shown over six consecutive planning steps.

Figure 13: Zero-shot HUGSIM rollouts on Waymo. Two closed-loop cases are shown over six consecutive planning steps.

Figure 14: Zero-shot HUGSIM rollouts on PandaSet. Two closed-loop cases are shown over six consecutive planning steps.

Explore similar work

ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model
Sep 22, 2026Sinuo Wang, Zichong Gu, Yuhan Huang +10End-to-End Autonomous Driving Diffusion-Based Planning
Huazhong University of Science and Technology · Shanghai Zaofu Intelligent Technology Co., Ltd. · Tongji University
DriveFuture: Future-Aware Latent World Models for Autonomous Driving
May 10, 2026Yufeng Hong, Xiaotian Zhou, Yingyan Li +6World Models for Autonomous Driving Latent World Models
Beijing Institute of Technology · Institute of Automation, Chinese Academy of Sciences · Beihang University +5
ReDrive: Shaping Representations with World Modeling for End-to-End Driving
Sep 27, 2026Yueting Zhu, Shaoyu Chen, Yuehao Song +4World Model Learning End-to-End Autonomous Driving
Huazhong University of Science & Technology · Horizon Robotics

Sep 22, 2026, cs.CVJ/K move · Enter open · S save

ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model

Sinuo Wang, Zichong Gu, Yuhan Huang, Wenxin Wen+9

Huazhong University of Science and Technology · Shanghai Zaofu Intelligent Technology Co., Ltd. · Tongji University

End-to-End Autonomous Driving Diffusion-Based Planning World Models Autonomous Driving Planning

Open paper AskPDF arXiv

Existing latent world models are typically optimized for future predictability, yet the resulting representations are not necessarily useful for planning in autonomous driving. Predictions are commonly used for pretraining or auxiliary supervision rather than as direct conditioning signals for trajectory generation. We propose ForeDrive, which learns a planning-relevant latent representation and couples it asymmetrically to a Diffusion Transformer (DiT) planner. The planner consumes multi-horizon latent future representations learned with a JEPA-style world model; planning gradients update the shared online encoder, while stop-gradient routing trains the latent predictor with forecasting losses only. Because predicted futures have varying reliability across horizons and BEV trajectories are misaligned with image tokens, we use gated visual fusion, future-status injection, and Trajectory-Adaptive Bias (TAB) to inject future latents as guidance without overriding the current observation. Trained with pure imitation learning and using only the current front-view image as visual input at inference, ForeDrive attains 89.9 PDMS on NAVSIM v1 and 90.0 one-stage EPDMS on NAVSIM v2, without reinforcement learning or an external trajectory scorer.

Abstract

Explore similar work

ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model

DriveFuture: Future-Aware Latent World Models for Autonomous Driving

ReDrive: Shaping Representations with World Modeling for End-to-End Driving