DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving

Yiyao Zhu; Ying Xue; Haiming Zhang; Guangfeng Jiang; Wending Zhou; Xu Yan; Jiantao Gao; Yingjie Cai

cs.CVApr 1, 2026

DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving

Authors: Yiyao Zhu, Ying Xue, Haiming Zhang, Guangfeng Jiang, Wending Zhou, Xu Yan, Jiantao Gao, Yingjie Cai, +3 more

Organizations: HKUST · CUHK-SZ · USTC · Huawei Foundation Model Department

Abstract

Vision-based autonomous driving has gained much attention due to its low costs and excellent performance. Compared with dense BEV (Bird's Eye View) or sparse query models, Gaussian-centric method is a comprehensive yet sparse representation by describing scene with 3D semantic Gaussians. In this paper, we introduce DLWM, a novel paradigm with Dual Latent World Models specifically designed to enable holistic gaussian-centric pre-training in autonomous driving using two stages. In the first stage, DLWM predicts 3D Gaussians from queries by self-supervised reconstructing multi-view semantic and depth images. Equipped with fine-grained contextual features, in the second stage, two latent world models are trained separately for temporal feature learning, including Gaussian-flow-guided latent prediction for downstream occupancy perception and forecasting tasks, and ego-planning-guided latent prediction for motion planning. Extensive experiments in SurroundOcc and nuScenes benchmarks demonstrate that DLWM shows significant performance gains across Gaussian-centric 3D occupancy perception, 4D occupancy forecasting and motion planning tasks.

Figures & tables

Figure 1: Illustration of our DLWM for pre-training and performance improvements for downstream tasks .

Figure 2: Overall pipeline of DLWM. Stage 1 (Sec. 3.1 ) focuses on learning robust 3D Gaussian scene representations from multi-view videos using self-supervised reconstruction on depth and semantic maps. Stage 2 introduces dual latent world models. a. Gaussian-flow-guided model (Sec. 3.2 ) explicitly predicts 3D Gaussian flow, propagating the current Gaussian states to the future frame for latent prediction. b. Ego-planning-guided model (Sec. 3.3 ) conditions the future scene forecasting on the predicted ego trajectory. All predicted latents are supervised by the perceived features from next multi-view image using a frozen Gaussian perception module.

Figure 3: 4D occupancy forecasting in a stream manner. We transform current 3D Gaussians to the next frame with ego motion alignment and complete new area with random Gaussians.

Method	IoU	mIoU	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ const. veh.	$\blacksquare$ motorcycle	$\blacksquare$ pedestrian	$\blacksquare$ traffic cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drive. suf.	$\blacksquare$ other flat	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation
MonoScene [ 3 ]	23.96	7.31	4.03	0.35	8.00	8.04	2.90	0.28	1.16	0.67	4.01	4.35	27.72	5.20	15.13	11.29	9.03	14.86
Atlas [ 31 ]	28.66	15.00	10.64	5.68	19.66	24.94	8.90	8.84	6.47	3.28	10.42	16.21	34.86	15.46	21.89	20.95	11.21	20.54
BEVFormer [ 23 ]	30.50	16.75	14.22	6.58	23.46	28.28	8.66	10.77	6.64	4.05	11.20	17.78	37.28	18.00	22.88	22.17	13.80	22.21
TPVFormer [ 12 ]	30.86	17.10	15.96	5.31	23.86	27.32	9.79	8.74	7.09	5.20	10.97	19.22	38.87	21.25	24.26	23.15	11.73	20.81
OccFormer [ 51 ]	31.39	19.03	18.65	10.41	23.92	30.29	10.31	14.19	13.59	10.13	12.49	20.77	38.78	19.79	24.19	22.21	13.48	21.35
SurroundOcc [ 39 ]	31.49	20.30	20.59	11.68	28.06	30.86	10.70	15.14	14.09	12.06	14.38	22.26	37.29	23.70	24.49	22.77	14.89	21.86

Table 1: 3D occupancy perception results on the SurroundOcc-nuScenes validation set [ 39 ] . *: We re-evaluate the checkpoints released by GaussianWorld because they repeatedly calculated the metrics for intermediate time-interval frames within each video.

Method	Input	Aux. Sup.	mIoU (%) $\uparrow$					IoU (%) $\uparrow$
Method	Input	Aux. Sup.	0s	1s	2s	3s	Avg.	0s	1s	2s	3s	Avg.
Copy&Paste	3D-Occ	None	66.38	14.91	10.54	8.52	11.33	62.29	24.47	19.77	17.31	20.52
OccWorld-O [ 52 ]	3D-Occ	None	66.38	25.78	15.14	10.51	17.14	62.29	34.63	25.07	20.18	26.63
OccWorld-T [ 52 ]	Camera	Semantic LiDAR	7.21	4.68	3.36	2.63	3.56	10.66	9.32	8.23	7.47	8.34
OccWorld-S [ 52 ]	Camera	None	0.27	0.28	0.26	0.24	0.26	4.32	5.05	5.01	4.95	5.00
OccWorld-D [ 52 ]	Camera	3D-Occ	18.63	11.55	8.10	6.22	8.62	22.88	18.90	16.26	14.43	16.53

Table 2: 4D occupancy forecasting results on the SurroundOcc-nuScenes validation set [ 39 ] . Aux. Sup. represents auxiliary supervision. Avg. computes the average result of 1s, 2s, and 3s.

Method	Aux. Task	L2 (m) $\downarrow$				Collision Rate (%) $\downarrow$
Method	Aux. Task	1s	2s	3s	Avg.	1s	2s	3s	Avg.
NMP $\diamond$ [ 47 ]	Det & Motion	0.53	1.25	2.67	1.48	0.04	0.12	0.87	0.34
FF $\diamond$ [ 6 ]	FreeSpace	0.55	1.20	2.54	1.43	0.06	0.17	1.07	0.43
EO $\diamond$ [ 19 ]	FreeSpace	0.67	1.36	2.78	1.60	0.04	0.09	0.88	0.33
ST-P3 [ 7 ]	Det & Map & Depth	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
UniAD [ 9 ]	Det&Track&Map&Motion&Occ	0.44	0.67	0.96	0.69	0.04	0.08	0.23	0.12

Table 3: Motion Planning Results on the nuScenes validation set. The metrics are computed by the way in VAD [ 15 ] .

\diamond

: Lidar-based methods. We do not utilize ego status in the planning module.

Exp.	Settings				Perception
Exp.	Sup- $D_{L}$	Sup- $D$	Sup- $S$	Gaussian Flow	IoU (%) $\uparrow$	mIoU (%) $\uparrow$
Exp1	✓				18.40	29.83
Exp2	✓	✓			18.74	30.05
Exp3	✓	✓	✓		18.99	30.23
Exp4	✓	✓	✓	✓	19.30	30.56

Table 4: Ablation study on render supervision and latent world model guided by Gaussian flow. “Sup-

D_{L}

”: sparse depth supervision derived from LiDAR points. “Sup-

D

”: dense pseudo-depth map supervision. “Sup-

S

”: semantics supervision. “Gaussian Flow”: denotes the inclusion of the Stage 2 latent world model guided by Gaussian flow.

Exp.	Settings			L2 (m) $\downarrow$				Col. (%) $\downarrow$
Exp.	Latent BEV	BEV Forecast	Stage-1 Pre-train	1s	2s	3s	Avg	1s	2s	3s	Avg
Exp1				0.27	0.51	0.87	0.55	0.11	0.17	0.46	0.24
Exp2	✓			0.26	0.48	0.82	0.52	0.12	0.16	0.38	0.22
Exp3	✓	✓		0.23	0.47	0.81	0.50	0.10	0.13	0.36	0.20
Exp4	✓	✓	✓	0.21	0.43	0.76	0.46	0.11	0.14	0.32	0.19

Table 5: Ablation study on latent world model guided by ego planning. Latent BEV: planning with latent BEV rendered from 3D Gaussians. BEV Forecasting: future latent BEV forecasting integrated into planning. Stage-1 Pretrain: image reconstruction.

Dataset fraction	mIoU (%) $\uparrow$	IoU (%) $\uparrow$
10%	18.24	28.59
50%	19.30	30.56
100%	19.98	31.17

Table 6: Ablation study on data scale. 3D occupancy perception performance under different pre-training data scales.

Figure 4: Qualitative results of depth and semantic reconstruction and forecasting during the pre-training. The left column shows the depth and semantic reconstruction at the current frame while the right column illustrates the depth and semantic images rendered by predicted Gaussians at the next frame compared with the GT RGB images.

Figure 5: Qualitative comparison of 3D Occupancy Perception over GT, baseline without pre-training and ours with pre-training.

Model	mIoU (%) $\uparrow$	L2 (m) $\downarrow$	Col. (%) $\downarrow$
Unified	18.9	0.58	0.22
Dual	19.3	0.46	0.19

Table 7: Ablation study on “Dual” and “Unified”.

Nums	mIoU (%) $\uparrow$	IoU (%) $\uparrow$
7500	18.35	28.83
25600	19.30	30.56
51200	19.43	29.79

Table 8: Ablation study on the number of 3D Gaussians.

Nums	mIoU (%) $\uparrow$	IoU (%) $\uparrow$
0	18.83	30.12
1	19.30	30.56
2	19.23	30.52
4	19.06	29.20

Table 9: Ablation study on the number of future frames.

Task	Memory	Latency
Perception	4.4GB	327ms
Forecasting	6.8GB	684ms
Planning	4.2GB	274ms

Table 10: The computation cost analysis of all three tasks.

Figure 6: Qualitative comparison of 4D Occupancy Forecasting over GTs, baseline without pre-training and ours with pre-training.

Explore similar work

GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning
May 17, 2026Cheng Chen, Hao Huang, Saurabh Bagchi3D Gaussian Splatting World Model Learning
Purdue University · New York University Abu Dhabi
Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving
Mar 25, 2026Linbo Wang, Yupeng Zheng, Qiang Chen +13End-to-End Autonomous Driving World Models for Autonomous Driving
Institute of Automation, Chinese Academy of Sciences · Chongqing Chang’an Technology Co., Ltd · Zhongguancun Academy +2
GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation
Aug 17, 2026Tianchen Deng, Xuefeng Chen, Shuang Wu +5World Models for Autonomous Driving 4D Scene Generation
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University; the State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis; and the Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai 200240, China · Tsinghua University, Shenzhen, China · Chongqing Afari Intelligent Drive Co., Ltd., China +1

May 17, 2026J/K move · Enter open · S save

GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning

Cheng Chen, Hao Huang, Saurabh Bagchi

Purdue University · New York University Abu Dhabi

3D Gaussian Splatting World Model Learning Autonomous Driving Semantic Occupancy Mapping 3D Occupancy Prediction World Models for Autonomous Driving+1

Open paper AskPDF arXiv

Future 3D semantic occupancy forecasting and motion planning are central to autonomous driving, as they require models to reason about how surrounding scenes evolve and how the ego vehicle should act. Existing occupancy world models commonly discretize scenes into latent embeddings, volumetric features, or quantized tokens, and forecast future states through fixed-step autoregressive generation. This limits temporal flexibility, obscures scene evolution, accumulates errors over long horizons, and poorly matches the continuous-time dynamics of real driving scenes. We propose GEM, a Gaussian Evolution Model for non-autoregressive occupancy world modeling, where driving scenes are represented as explicit continuous 4D Gaussian primitives with learned dynamics. Instead of rolling out future occupancy states step by step, GEM directly queries the Gaussian world representation at arbitrary timestamps and splats the corresponding conditional 3D Gaussians into semantic occupancy volumes. This enables efficient forecasting over the full horizon while retaining a compact and interpretable scene representation. By decoupling spatial geometry, temporal support, and primitive motion, GEM makes the predicted world easier to inspect, as each primitive's evolution can be followed continuously over time. The same representation also supports motion planning by predicting future ego trajectories from the learned Gaussian world. Extensive experiments show that GEM achieves state-of-the-art future semantic occupancy forecasting and strong motion planning performance, while providing flexible temporal querying.

Abstract

Explore similar work

GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning

Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving

GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation