GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation
Authors: Tianchen Deng, Xuefeng Chen, Shuang Wu, Qu Chen, Jiajun Zhu, Bo Dai, Jianfei Yang, Hesheng Wang
Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University; the State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis; and the Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai 200240, China · Tsinghua University, Shenzhen, China · Chongqing Afari Intelligent Drive Co., Ltd., China · Nanyang Technological University, Singapore
Driving World Models (DWMs) have recently advanced rapidly with generative models, yet most existing methods mainly focus on conditional scene generation and lack explicit 3D scene understanding, language-grounded reasoning, and controllable 4D editing capabilities. Moreover, commonly used point cloud, occupancy, or BEV representations make it difficult to achieve fine-grained alignment between textual information and the underlying 3D scene structure. To address these limitations, we propose a foundation-feature Gaussian driving world model that unifies scene understanding, language-grounded reasoning, controllable 4D editing, and multi-modal generation within a single framework. Specifically, we introduce a foundation-feature Gaussian tokenizer that directly distills Qwen/SigLIP visual-language features into 3D Gaussian primitives, building a compact open-vocabulary Gaussian semantic field. We further design a geometry-aware Gaussian adapter that combines importance-aware hierarchical selection with text-conditioned Perceiver-style cross-attention to aggregate dense Gaussian primitives into compact world tokens. To improve representation compatibility, we introduce a KL-based Gaussian--image distribution alignment objective that aligns Gaussian world tokens with foundation image tokens. Based on the aligned Gaussian representation, our framework further supports instruction-controllable scene editing, including weather-conditioned generation and dynamic vehicle manipulation. Extensive experiments on broader driving benchmarks demonstrate that our method achieves state-of-the-art performance across scene understanding, visual grounding, planning-oriented reasoning, and controllable 4D generation tasks. We will release the code and datasets publicly on Github.
Figures & tables
Fig. 1: Overall performance on understanding and generation benchmarks. GaussianDWM++ achieves state-of-the-art performance in both scene understanding and scene generation. Radar radii are normalized as Metrici/Metricmax for understanding and Metricmin/Metrici for generation, so that larger radii indicate better performance. Boxed values show the original GaussianDWM++ results.
Fig. 2: We propose the unified 3D Gaussian-based world model framework that achieves comprehensive scene understanding and scene generation for driving scenarios. It efficiently encodes complex scenes, samples task-relevant information, and handles diverse question-answering tasks. Moreover, by leveraging the extracted world knowledge, our framework guides the generative model to perform accurate spatial and temporal scene generation.
Fig. 3: System Overview. GaussianDWM++ builds a foundation-feature Gaussian field by distilling Qwen visual features into explicit 3D Gaussian primitives. A geometry-aware adapter first performs driving-aware coarse selection and attribute projection. It then combines a pooled text embedding with learned queries and aggregates the selected Gaussian tokens through Perceiver-style cross-attention. A Gaussian–image distribution alignment objective regularizes the resulting world tokens toward the foundation image-token distribution. The compact Gaussian world tokens are shared by the LLM-based scene understanding branch and the dual-condition scene generation branch for high-level world knowledge.
Fig. 4: Qualitative results for scene understanding and scene generation. From top to bottom, we display the multi-view input of the current scene and the 3D Gaussian ellipsoids, the scene understanding results, and the results of temporal generation, spatial generation, weather-conditioned generation, and dynamic vehicle manipulation with varying motion speeds.
Fig. 5: Qualitative comparison of RGB-D NVS with a 4 m camera shift. Compared with GaussianDWM and reconstruction-based baselines [ 74 , 75 , 44 , 45 ] , GaussianDWM++ better preserves broad, continuous scene regions such as road surfaces, with fewer large-area artifacts and more coherent depth structure under large viewpoint changes.
RD&P ↑
2D VG ↑
3D VG ↑
Plan ↑
Model
Year
LLM
BLEU
ROUGE-L
CIDEr
mAP
F1
mIoU
Pr
mAP
F1
Acc
Avg. ↑
LLaVA1.5 [ 76 ]
2024
Vicuna-7B
64.23
76.69
74.82
0.10
0.16
14.31
6.51
5.33
3.12
36.20
28.15
MiniCPM-V 2 [ 77 ]
2024
MiniCPM-2B
47.43
63.16
69.88
0.11
0.13
13.34
0.97
1.55
0.86
36.69
23.41
MiniCPM-V 2.6 [ 77 ]
2024
Qwen2-7B
47.92
69.11
70.20
0.36
0.49
18.74
1.97
1.61
0.93
36.42
24.78
InternVL1.5-2B [ 78 ]
2024
InternLM2-7B
67.14
81.10
79.83
14.74
17.64
55.43
28.05
21.73
12.92
53.96
43.25
InternVL1.5-4B [ 78 ]
2024
Phi3-4B
66.63
80.64
79.24
14.27
17.60
53.52
25.14
19.46
11.63
40.25
40.84
TABLE I: Comparison on NuInteract [ 23 ] . Results cover region description and perception (RD&P), 2D/3D visual grounding (VG), and closed-set planning. Avg. is the arithmetic mean over the ten task metrics. All metrics are higher-is-better; best and second-best results are shown in bold and underlined , respectively.
Zero-shot Models
Fine-tuned Models
Dataset
Metrics
GPT-4o [ 82 ]
LLaVA-OV [ 83 ]
RoboTron [ 84 ]
Qwen2.5-VL [ 85 ]
Qwen2.5-VL [ 85 ]
InMLLM [ 68 ]
VGGT-Dist [ 86 ]
VGGT-Add [ 86 ]
VGGDrive [ 86 ]
GaussianDWM++
NuInstruct
MAE ↓
9.93
87.04
19.36
24.10
4.35
9.08
3.73
3.63
3.08
2.41
Accuracy ↑
10.64
3.75
2.57
0.63
47.71
32.48
56.21
56.35
56.37
58.02
mAP ↑
0
0
0
0
6.15
21.93
28.51
30.12
37.49
42.59
BLEU ↑
7.08
8.55
8.06
5.56
75.75
35.20
79.23
79.45
81.13
76.40
Average* ↑
1.95
0
0
0
31.32
20.13
40.06
40.57
42.98
43.65
TABLE II: Comparison on NuInstruct [ 68 ] under the VGGDrive evaluation protocol. MAE is lower-is-better, while the remaining metrics are higher-is-better. Average* is (Accuracy+mAP+BLEU−MAE)/4 . Best and second-best results are shown in bold and underlined , respectively.
Zero-shot Models
Fine-tuned Models
Dataset
Metrics
GPT-4o [ 82 ]
LLaVA-OV [ 83 ]
RoboTron [ 84 ]
OmniDrive [ 69 ]
HERMES [ 8 ]
Qwen2.5-VL [ 85 ]
VGGT-Dist [ 86 ]
VGGT-Add [ 86 ]
VGGDrive [ 86 ]
GaussianDWM++
OmniDrive
BLEU ↑
10.91
16.14
20.30
38.00
–
37.29
37.28
37.23
37.58
42.11
CIDEr ↑
24.42
28.41
34.33
68.60
74.10
86.29
86.41
86.38
86.57
100.00
ROUGE-L ↑
22.34
22.14
23.67
32.60
32.70
34.33
34.32
34.26
34.40
38.13
Average ↑
19.22
22.23
26.10
46.40
–
52.64
52.67
52.62
52.85
60.08
TABLE III: Comparison on OmniDrive [ 69 ] under the VGGDrive evaluation protocol. Average is the arithmetic mean of BLEU, CIDEr, and ROUGE-L. All metrics are higher-is-better. Best and second-best results are shown in bold and underlined , respectively.
Fig. 6: Long-horizon generation from 0–9 s.
±1 m
±2 m
±4 m
Method
FID ↓
FVD ↓
FID ↓
FVD ↓
FID ↓
FVD ↓
PVG [ 44 ]
48.15
246.74
60.44
356.23
84.50
501.16
EmerNeRF [ 42 ]
37.57
171.47
52.03
294.55
76.11
497.85
StreetGaussian [ 45 ]
32.12
153.45
43.24
256.91
67.44
429.98
OmniRe [ 74 ]
31.48
152.01
43.31
254.52
67.36
428.20
FreeVS [ 87 ]
51.26
431.99
62.04
497.37
77.14
556.14
TABLE IV: Spatial novel-view generation on nuScenes. FID and FVD under ±1 m, ±2 m, and ±4 m camera shifts using the six-camera DiST-S protocol; lower is better. Best and second-best results are shown in bold and underlined , respectively.
Method
6-View
Video
FID ↓
FVD ↓
BEVGen
✓
✗
25.54
–
BEVControl
✓
✗
24.85
–
DriveGAN [ 89 ]
✗
✓
73.40
502.30
DriveDreamer
✗
✓
52.60
452.00
Vista [ 1 ]
✗
✓
6.90
89.40
WoVoGen
✓
✓
27.60
417.70
TABLE V: Temporal video generation on the nuScenes validation set. FID and FVD are lower-is-better. Best and second-best results are shown in bold and underlined , respectively.
Fine-tuning
Gaussian
Sampling
RD&P ↑
2D VG ↑
3D VG ↑
Plan ↑
Avg. ↑
BLEU
ROUGE-L
CIDEr
mAP
F1
mIoU
Pr
mAP
F1
Acc
Zero-shot
No
–
2.91
12.68
0.59
0.00
0.00
12.24
48.75
47.59
29.12
0.00
15.39
Fine-tuned
No
–
65.09
78.35
76.56
30.01
35.45
69.01
50.36
51.88
31.43
45.09
53.32
Fine-tuned
Yes
Random
66.19
79.00
76.97
33.94
39.37
71.40
50.94
52.85
32.03
49.43
55.21
Fine-tuned
Yes
Top- K + Uniform
68.78
81.06
78.82
33.89
39.31
71.37
51.16
52.87
32.05
80.95
58.93
Fine-tuned
Yes
Top- K + Uniform + Similarity
68.78
81.06
78.82
34.95
40.49
71.85
50.66
52.78
32.05
80.95
59.23
TABLE VI: Ablation of the Gaussian scene representation and token sampling on NuInteract. The settings and results follow the conference-stage evaluation protocol. Best and second-best results are shown in bold and underlined , respectively.
Foundation-Feature Distillation
Coarse Selection
Fine Adapter
Distribution Alignment
RD&P ↑
2D VG ↑
3D VG ↑
Plan ↑
Avg. ↑
BLEU
ROUGE-L
CIDEr
mAP
F1
mIoU
Pr
mAP
F1
Acc
✗
Voxel Top- K
Similarity
✗
68.78
81.06
78.72
34.95
40.49
71.85
50.66
52.78
32.05
80.95
59.23
✗
Voxel Top- K
Similarity
✓
69.97
81.47
80.05
48.59
41.95
77.56
53.39
57.18
35.22
67.02
61.24
✗
Voxel Top- K
Query-Aware Cross-Attn
✓
70.19
81.77
80.11
48.10
41.83
77.67
53.59
57.67
35.51
67.21
61.37
✗
Driving-Aware
Query-Aware Cross-Attn
✓
69.76
81.88
79.98
49.07
42.03
77.76
53.30
57.10
35.18
68.06
61.41
✓
Driving-Aware
Query-Aware Cross-Attn
✓
69.81
81.75
80.42
48.22
41.92
77.83
53.32
57.19
35.13
81.52
62.71
TABLE VII: Ablation of the foundation-feature Gaussian tokenizer and geometry-aware Gaussian adapter on NuInteract. Rows progressively introduce KL-based Gaussian–image distribution alignment, text-conditioned cross-attention, and driving-aware coarse selection. The last row additionally enables foundation-feature distillation. Avg. is the arithmetic mean over the ten task metrics; higher is better.
Low-Level
High-level
FID ↓
FVD ↓
✗
✓
-
-
✓
✗
10.12
45.14
✓
✓
8.36
44.5
TABLE VIII: Ablation study of the dual-condition generation mechanism under a ±1 m camera shift. FID and FVD are lower-is-better. “–” denotes failure under the corresponding setting.
Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation, yet action-imitation objectives provide only weak supervision for metric 3D structure and short-horizon physical evolution. Geometry-enhanced policies mainly improve current-scene grounding, whereas predictive policies often model future dynamics in RGB or latent spaces and may incur substantial deployment cost. GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information. We present \textbf{\methodname}, a compact, policy-native extension that inserts \textbf{World State Tokens} and \textbf{World Prediction Tokens} directly into the VLA backbone. A training-only \textbf{World Representation Head} decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, while static--dynamic factorization preserves persistent structure and focuses residual motion on interaction-relevant regions. At inference, the head, renderer, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only 20 world tokens without online Gaussian decoding or rollout. \method achieves \textbf{98.6%} on LIBERO and \textbf{87.8%} on LIBERO-Plus, with clear gains under Camera and Layout shifts. Real-robot experiments further improve average success from 29.2% to 52.5% over reproduced π0.5 while maintaining efficient closed-loop control.
Yuqing Jiang, Zijian Zhang, Weitao Zhou +8
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tuojing Intelligence +7
Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/
Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso +4
Driving world models provide a promising route toward scalable counterfactual data generation and interactive simulation beyond recorded driving logs. Realizing this potential requires a system that can generalize across diverse scenes, respond faithfully to prescribed controls, generate coherent multi-sensor observations, and operate efficiently under repeated inference. We present \textbf{HelloWorld}, a 2B driving world model system designed around these requirements. HelloWorld progressively specializes broad visual and motion priors from heterogeneous video data into controllable driving generation using ego pose, HD maps, and 3D boxes. A block-causal generation interface, together with adaptation to self-generated context, aligns the model with sequential simulation. The system further supports synchronized seven-camera RGB generation and conditional LiDAR synthesis, and is distilled toward few-step inference for efficient deployment. Experiments evaluate visual quality, control fidelity, cross-view consistency, robustness under repeated generation, inference efficiency, and LiDAR synthesis. Together, HelloWorld provides a unified framework for scalable driving data generation and interactive simulation.