Organizations: Institute of Automation, CAS · Chongqing Chang’an Technology Co., Ltd · School of Advanced Interdisciplinary Sciences, UCAS · School of Artificial Intelligence, UCAS
We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level diffusion world models enable safe imagination-based training, they suffer from multi-step diffusion inference latency (2s/frame) that prevents high-frequency RL interaction. Our approach leverages denoised latent features from video generation models through three key mechanisms: (1) shortcut forcing that reduces sampling complexity via recursive multi-resolution step compression, (2) an autoregressive dense reward model operating directly on latent representations for fine-grained credit assignment, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 87.7 EPDMS on NavSim v2, establishing state-of-the-art performance and demonstrating that latent-space RL is effective for autonomous driving.
Figures & tables
Figure 1: World model imagination training guided by diverse trajectories. Each row shows a driving scenario where the world model imagines future outcomes for candidate trajectories. RGB sequences display predicted frames with reward model scores (red: collision risk, green: safe). BEV maps (right) visualize trajectories: hazardous paths (left, red-highlighted) versus safe alternatives (right, green-highlighted).
Figure 2: PCA visualization of denoised latent features, demonstrating strong spatial and semantic coherence.
Figure 3: Overview of the DreamerAD RL training architecture. The RL training pipeline consists of three main stages: 1) Policy Generation and Sampling (yellow): Generates a base policy from historical inputs and samples a set of candidate trajectories based on a predefined vocabulary. 2) RL Training via World Model (green): Performs latent rollouts for the sampled trajectories to imagine future states. Step-wise rewards are decoded from these latent features and aggregated into a time-aware dense reward. Notably, our latent representations can be losslessly decoded into RGB frames for accident analysis or visualization, though decoding is bypassed during training for efficiency. 3) Policy Optimization (blue): Computes group advantages from the dense rewards to optimize the policy network using the GRPO algorithm.
Figure 4: Visualization of one-step inference for Epona and Shortcut Forcing World Model.
Method
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
TransFuser [ 24 ]
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7
hydramdp++ [ 20 ]
97.2
97.5
99.4
99.6
83.1
96.5
94.4
98.2
70.9
81.4
DriveSuprim [ 22 ]
97.5
96.5
99.4
99.6
88.4
96.6
95.5
98.3
77.0
83.1
ipad [ 35 ]
98.7
97.8
99.1
99.8
83.5
98.0
96.2
98.1
85.6
84.1
ReCogDrive [ 38 ]
98.3
95.2
99.5
99.8
87.1
97.5
96.6
98.3
86.5
83.6
DiffusionDrive [ 21 ]
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
Table 1: Comparison with state-of-the-art methods on the NAVSIM v2 [ 5 ] with extended metrics. + and - denote improvement/degradation relative to Epona.
Method
Venue
Input
NC ↑
DAC ↑
TTC ↑
Comf ↑
EP ↑
PDMS ↑
VADV2 [ 4 ]
arXiv 2024
C&L
97.2
89.1
91.6
100.0
76.0
80.9
UniAD [ 14 ]
CVPR 2023
C&L
97.8
91.9
92.9
100.0
78.8
83.4
TransFuser [ 24 ]
IEEE TPAMI
C&L
97.7
92.8
92.8
100.0
79.2
84.0
PARA-Drive [ 27 ]
CVPR 2024
C&L
97.9
92.4
93.0
99.8
79.3
84.0
DRAMA [ 33 ]
arXiv 2024
C&L
98.0
93.1
94.8
100.0
80.1
85.5
Hydra-MDP [ 20 ]
arXiv 2024
C&L
98.3
96.0
94.6
100.0
78.7
86.5
Table 2: Comparison with state-of-the-art methods on the NAVSIM v1 [ 5 ] . + and - denote improvement/degradation relative to Epona.
ID
SF-WM
AD-RM
RL-SM
EPDMS ↑
Vocab Sampling
WorldRFT
Flow-GRPO
1
85.1
2
√
√
86.4
3
√
√
87.0
4
√
√
√
87.7
5
√
√
√
86.6
Table 3: Ablation study of each component. SF-WM: Shortcut Forcing World Model, AD-RM: Autoregressive Dense Reward Model, RL-SM: RL Sampling Method. One-step inference is used by default during training unless otherwise specified.
Table 8
Figure 5: Comparison before and after RL training. The leftmost column displays the front-view camera image at the current timestep. The right columns show the BEV planning results from SFT and RL, respectively. The red trajectory represents the SFT output, while the blue represents the RL output. Red highlights in the SFT BEV maps indicate collisions, whereas green highlights in the RL BEV maps denote safe passage.