Learned world models for autonomous driving operate in unconstrained latent spaces, yet the physical systems they represent are governed by strict non-holonomic constraints: ground vehicles cannot realize instantaneous pure lateral motion. We study long-horizon latent rollout modeling for autonomous driving and ask whether rollout quality can be improved by assigning a motion-aware geometry to latent transitions, rather than by redesigning the predictor architecture. Starting from an unconstrained latent rollout baseline, we introduce Directional Pullback Regularization (DPR), a training time method that defines a kinematics-aware anisotropic Riemannian metric in decoded state space and pulls it back through the decoder Jacobian to the latent manifold. The resulting latent-space metric assigns higher cost to latent directions that decode to lateral vehicle motion, while preserving the original encoder-dynamics-decoder backbone. This yields a clean three-way comparison among an Unconstrained Latent Rollout model (ULR), an Euclidean Pullback Regularization ablation (EPR), and the full directional method (DPR). We evaluate all models on open-loop ego-motion rollout over a common 10-second test horizon, using training rollout lengths of 1, 3, 5, and 10 steps. Across all four training regimes, DPR achieves lower 10-second final displacement error than both ULR and EPR. The strongest relative accuracy gain appears for models trained with 3-step rollout supervision, where DPR reduces the 10-second final displacement error by approximately 44 percent relative to ULR. The strongest overall control-relevant tradeoff is obtained for longer rollout-trained models, especially at 10 steps, where DPR simultaneously improves displacement error, lateral motion consistency, and forward motion preservation relative to both ULR and EPR.
Figures & tables
Fig. 1: Method overview of the proposed directional pullback regularization (DPR) framework. The upper row shows the baseline open-loop rollout pipeline: the motion history, current state, and road curvature are assembled into the observation ot , encoded into the latent state zt , propagated by the latent dynamics model to produce Δzt , decoded back to state space, converted to the predicted increment Δxt , and integrated to obtain the next rollout state. The lower row shows the regularization path. The latent update and decoder Jacobian define the decoded tangent ut=Jψ(zt)Δzt/Δt , which is evaluated under the kinematics-aware Riemannian metric Mβ(x^t) . Pulling this metric back through the decoder Jacobian yields the latent-space metric Gβ(zt) , and the resulting metric loss is combined with the prediction loss to form the total training objective. The dashed arrow indicates the backpropagated training signal through which the pullback regularizer shapes latent transitions.
Fig. 2: Average displacement error over the evaluation horizon for the three compared models: the unconstrained baseline (ULR), the Euclidean pullback ablation (EPR), and the proposed directional pullback model (DPR). Each subplot corresponds to a different training rollout length U∈{1,3,5,10} , and DPR is shown with the validation-selected directional weight β∗ for that training regime. Across all training rollout lengths, DPR reduces long-horizon displacement error relative to both ULR and EPR, with the strongest gains appearing for longer self-fed rollout training.
Fig. 3: Key 10-second results across training rollout lengths for the unconstrained baseline (ULR), the Euclidean pullback ablation (EPR), and the proposed directional pullback model (DPR). The panels show final displacement error, ground-truth-frame lateral error, and the fraction of rollout steps with forward velocity above 0.5m/s . DPR is the only variant that consistently improves 10-second final displacement error across all training rollout lengths, and in the longer-rollout regimes it also maintains or improves the forward-valid fraction while reducing lateral-motion error.
Train rollout
Model
β
10 s mean displacement error (m)
10 s final displacement error (m)
95th lateral error (m/s)
95th curvature error (rad/s)
Median forward velocity (m/s)
Forward-valid fraction
1
ULR
–
31.366
80.646
18.370
0.683
16.345
0.953
1
EPR
0
23.165
50.927
13.689
0.687
16.983
0.692
1
DPR
100
19.254
47.172
9.395
0.773
0.946
0.531
3
ULR
–
33.913
84.163
32.377
1.868
1.712
0.530
3
EPR
0
24.067
65.300
29.309
1.716
-0.623
0.483
3
DPR
50
17.057
47.160
13.748
0.902
3.115
0.599
TABLE I: Ten-second open-loop rollout results for ULR (Unconstrained Latent Rollout), EPR (Euclidean Pullback Regularization), and DPR (Directional Pullback Regularization).
Long-horizon planning enables autonomous vehicles to anticipate scene evolution and potential risks, supporting safe and stable decisions in complex interactions. However, existing methods struggle to propagate motion trends from observed history into the future. Long rollouts based on a single latent state may further attenuate useful dynamics, retain stale motion patterns, and disrupt reliable near-term plans. We introduce MomWorld, a momentum-aware latent world model for long-horizon planning. MomWorld extracts scene motion trends from historical-to-current observations and propagates latent momentum into future horizons, jointly predicting future configuration and momentum states. A learnable momentum persistence mechanism preserves stable trends, scene-conditioned momentum updates adapt future dynamics, and a scene-adaptive reset gate suppresses stale momentum under abrupt changes. We further propose MoFlow, a momentum-conditioned flow-matching module that refines a base trajectory to align with the predicted future scene evolution in only a few integration steps, with a horizon-aware residual fusion that preserves near-term planning stability while permitting stronger long-range corrections. Extensive experiments on NAVSIM, nuScenes and Bench2Drive demonstrate that MomWorld improves long-horizon planning consistency and reduces the average collision rate by 12.2% relative to MomAD over a 6-second planning horizon.
Ziying Song, Shengkai Zhang, Lei Yang +6
Nanyang Technological University · Beijing Jiaotong University · North University of China +2
We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level diffusion world models enable safe imagination-based training, they suffer from multi-step diffusion inference latency (2s/frame) that prevents high-frequency RL interaction. Our approach leverages denoised latent features from video generation models through three key mechanisms: (1) shortcut forcing that reduces sampling complexity via recursive multi-resolution step compression, (2) an autoregressive dense reward model operating directly on latent representations for fine-grained credit assignment, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 87.7 EPDMS on NavSim v2, establishing state-of-the-art performance and demonstrating that latent-space RL is effective for autonomous driving.
Pengxuan Yang, Yupeng Zheng, Deheng Qian +11
Institute of Automation, CAS · Chongqing Chang’an Technology Co., Ltd · School of Advanced Interdisciplinary Sciences, UCAS +1
Data-efficient learning remains a central challenge in autonomous driving due to the high cost and safety risks of large-scale real-world interaction. Although world-model-based reinforcement learning enables policy optimization through latent imagination, existing approaches often lack explicit mechanisms to encode spatial and kinematic structure essential for driving tasks. In this work, we build upon the Recurrent State-Space Model (RSSM) and propose a kinematics-aware latent world model framework for autonomous driving. Vehicle kinematic information is incorporated into the observation encoder to ground latent transitions in physically meaningful motion dynamics, while geometry-aware supervision regularizes the RSSM latent state to capture task-relevant spatial structure beyond pixel reconstruction. The resulting structured latent dynamics improve long-horizon imagination fidelity and stabilize policy optimization. Experiments in a driving simulation benchmark demonstrate consistent gains over both model-free and pixel-based world-model baselines in terms of sample efficiency and driving performance. Ablation studies further verify that the proposed design enhances spatial representation quality within the latent space. These results suggest that integrating kinematic grounding into RSSM-based world models provides a scalable and physically grounded paradigm for autonomous driving policy learning.
Jiazhuo Li, Linjiang Cao, Qi Liu +1
Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, Shanghai, China