Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos

Tong Zhao; Zhiyuan Han; Lin Chen; Bowen Xie; Yunxi Qiao; Bohan Li; Weiqing Xiao; Gen Li

cs.CVOct 19, 2026

Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos

Authors: Tong Zhao, Zhiyuan Han, Lin Chen, Bowen Xie, Yunxi Qiao, Bohan Li, Weiqing Xiao, Gen Li,

Organizations: Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · Tongji University, Shanghai, China. · Beijing Technology and Business University, Beijing, China. · Fuzhou University, Fuzhou, China. ·

Abstract

Generating reusable driving environments requires both controllable future observations and an explicit representation that supports rendering from different viewpoints, yet existing methods still recover explicit scenes from generated RGB videos through optimization for each sequence. To overcome these limitations, we present Drive2Gauss, a feedforward framework that converts jointly generated RGB, metric depth, and dynamic flow into dynamic 4D Gaussian scenes. A Multimodal Video Generator predicts the three modalities from reference multiview images and future control signals. Furthermore, we introduce a Gaussian Query Decoder that uses depth to initialize sparse 3D queries and flow to associate the queries across time, and integrates appearance information through spatiotemporal feature aggregation and iterative query refinement. The reconstruction stage operates in a single feedforward pass without optimization for each scene. Extensive results on nuScenes demonstrate that our approach attains an FID of 6.38 and an FVD of 12.43 for future RGB generation, a PSNR of 29.91 dB when reconstructing its generated observations, and supports both direct rendering under lateral camera shifts and downstream video fixing. Code and more visualizations are available at https://zttt04. github.io/Drive2Gauss_page/.

Figures & tables

Fig. 1: Schematic comparison with RGB-first methods that use per-scene optimization. Drive2Gauss instead jointly generates RGB, metric depth, and dynamic flow, then constructs a 4D Gaussian scene with a feedforward decoder. The feedforward property refers to the reconstruction stage.

Fig. 2: Overview of Drive2Gauss. Reference multiview images and control conditions are processed by the Multimodal Video Generator to predict RGB, metric depth, and dynamic flow. Depth and flow initialize 3D queries, while RGB features from multiple views and context frames are aggregated by the Spatiotemporal Feature Aggregation module. The updated queries are decoded into dynamic Gaussian primitives for rendering.

Fig. 3: Details of the spatiotemporal feature aggregation. Each query is projected into valid camera views and context frames to sample features from the generated observations, along with the corresponding key and value features. Cross-attention (Q, K, V) fuses the sampled features to update the current query location, and the update is repeated $k$ times, after which the refined queries are passed to the Gaussian decoding stage for rendering.

Method	RGB	Depth	Flow	FID $\downarrow$	FVD $\downarrow$
Panacea [ 1 ]	$\checkmark$	$\times$	$\times$	16.96	139.00
MagicDrive [ 2 ]	$\checkmark$	$\times$	$\times$	16.20	217.94
Drive-WM [ 3 ]	$\checkmark$	$\times$	$\times$	15.80	122.70
WoVoGen [ 40 ]	$\checkmark$	$\times$	$\times$	27.60	417.70
Vista* [ 41 ]	$\checkmark$	$\times$	$\times$	13.97	112.65
DriveDreamer-2 [ 42 ]	$\checkmark$	$\times$	$\times$	11.20	55.70

TABLE I: Comparison of future video generation quality and output modalities. Checkmarks indicate the modalities generated by each method. Lower FID and FVD values are better.

Fig. 4: Qualitative results of joint multimodal future generation. For each example, we visualize the generated RGB, metric depth, and dynamic flow from the front three camera views of each sample. The three modalities together provide complementary appearance, geometric, and motion information for the subsequent 4D Gaussian construction and rendering.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FID $\downarrow$	FVD $\downarrow$
Panacea + DGGT [ 1 , 36 ]	26.23	0.261	0.790	50.24	181.51
MagicDrive-V2 + DGGT [ 7 , 36 ]	28.12	0.230	0.830	68.09	370.59
DiST-4D + DGGT [ 28 , 36 ]	27.76	0.200	0.825	30.17	90.70
Drive2Gauss	29.91	0.101	0.884	12.85	32.01

TABLE II: Quantitative comparison of feedforward 4D Gaussian reconstruction. PSNR, LPIPS, and SSIM are computed against the generated RGB observations, while FID and FVD are computed against the original nuScenes videos.

Fig. 5: Qualitative comparison of feedforward 4D Gaussian reconstruction. For each method, the upper row shows the generated RGB observation used as input, while the lower row shows the corresponding rendering from the reconstructed 4D Gaussian scene. The comparison highlights how well each method preserves scene layout, object appearance, and local geometric details between inference and rendering.

Method	$\pm 1$ m		$\pm 2$ m		$\pm 4$ m
	FID $\downarrow$	FVD $\downarrow$	FID $\downarrow$	FVD $\downarrow$	FID $\downarrow$	FVD $\downarrow$
DreamDrive [ 26 ]	72.20	381.93	143.15	780.71	233.15	1271.42
Panacea + DGGT [ 1 , 36 ]	142.32	664.29	190.57	844.70	251.93	1085.29
MagicDrive-V2 + DGGT [ 7 , 36 ]	86.40	1032.22	109.34	1241.29	150.05	1481.17
DiST-T + DGGT [ 28 , 36 ]	53.44	562.22	73.55	744.32	107.93	956.98
Drive2Gauss	38.38	281.27	59.60	605.65	101.23	1025.44

TABLE III: Direct novel-view synthesis under lateral camera shifts from generated future observations. Lower FID and FVD are better.

Fig. 6: Qualitative comparison of novel-view rendering under lateral camera shifts. For each scene, the upper row shows the generated or ground-truth observation, while the lower row shows the corresponding rendering from the reconstructed 4D Gaussian scene.

Method	$\pm 1$ m		$\pm 2$ m		$\pm 4$ m
	FID $\downarrow$	FVD $\downarrow$	FID $\downarrow$	FVD $\downarrow$	FID $\downarrow$	FVD $\downarrow$
DiST-S [ 28 ]	10.12	45.14	12.97	68.80	17.57	105.29
GaussianDWM [ 31 ]	8.36	44.50	11.27	68.17	18.81	116.40
WorldSplat [ 39 ]	8.25	40.17	11.26	47.41	13.38	64.07
RGB-D + SPVC [ 104 ]	10.37	25.99	11.19	30.68	12.82	41.29
Drive2Gauss + SPVC [ 104 ]	6.68	21.25	7.85	24.28	9.00	35.73

TABLE IV: View-fixing performance under lateral camera shifts. Lower FID and FVD are better.

Fig. 7: Qualitative example of the downstream view fixing pipeline in our framework. Applying SPVC to the Drive2Gauss reconstruction substantially reduces the artifacts introduced by the lateral camera shift.

Variant	PSNR $\uparrow$	SSIM $\uparrow$
w/o Init Flow	27.89	0.814
w/o Iterative Refinement	28.03	0.827
w/o Cross-Attention	28.76	0.849
Drive2Gauss (Full)	29.91	0.884

TABLE V: Ablation study of the Gaussian Query Decoder. Metrics follow the reconstruction protocol of Table II .

Queries	PSNR $\uparrow$	SSIM $\uparrow$	Train Peak Memory (GB)	Inference Time (s)
480K	28.87	0.811	45	0.96
720K	29.62	0.836	67	1.11
960K	29.91	0.884	92	1.29
1200K	/	/	OOM	/

TABLE VI: Sensitivity to the number of Gaussian queries. Training peak memory is reported in GB, while inference time is reported in seconds.

Multimodal Video Generation

See all 29 papers

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction
Mar 8, 2026Haibao Yu, Kuntao Xiao, Jiahang Wang +7Dynamic Scene Reconstruction 3D Gaussian Splatting
The University of Hong Kong · Tuojing Intelligence · King’s College London +2
SplatFlux: Feed-Forward 4D Gaussian Reconstruction and Generation from Video Latents
Oct 20, 2026Runmin Zhang, Binglin Ren, Zhu Yu +64D Reconstruction Dynamic Scene Reconstruction
Zhejiang University · Ant Group
GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation
Aug 17, 2026Tianchen Deng, Xuefeng Chen, Shuang Wu +5World Models for Autonomous Driving 4D Scene Generation
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University; the State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis; and the Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai 200240, China · Tsinghua University, Shenzhen, China · Chongqing Afari Intelligent Drive Co., Ltd., China +1

Mar 8, 2026J/K move · Enter open · S save

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction

Haibao Yu, Kuntao Xiao, Jiahang Wang, Ruiyang Hao+6

The University of Hong Kong · Tuojing Intelligence · King’s College London+2

Dynamic Scene Reconstruction 3D Gaussian Splatting Novel View Synthesis Feed-Forward 3D Reconstruction Autonomous Driving Simulation

Open paper AskPDF arXiv

High-fidelity visual reconstruction and novel-view synthesis are essential for realistic closed-loop evaluation in autonomous driving. While 4D Gaussian Splatting (4DGS) offers a promising balance of accuracy and efficiency, existing per-scene optimization methods require costly iterative refinement, rendering them unscalable for extensive urban environments. Conversely, current feed-forward approaches often suffer from degraded photometric quality. To address these limitations, we propose ReconDrive, a feed-forward framework that leverages and extends the 3D foundation model VGGT for rapid, high-fidelity 4DGS generation. Our architecture introduces two core adaptations to tailor the foundation model to dynamic driving scenes: (1) Hybrid Gaussian Prediction Heads, which decouple the regression of spatial coordinates and appearance attributes to overcome the photometric deficiencies inherent in generalized foundation features; and (2) a Static-Dynamic 4D Composition strategy that explicitly captures temporal motion via velocity modeling to represent complex dynamic environments. Benchmarked on nuScenes, ReconDrive significantly outperforms existing feed-forward baselines in reconstruction, novel-view synthesis, and 3D perception. It achieves performance competitive with per-scene optimization while being orders of magnitude faster, providing a scalable and practical solution for realistic driving simulation.

Abstract

Multimodal Video Generation

Explore similar work

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction

SplatFlux: Feed-Forward 4D Gaussian Reconstruction and Generation from Video Latents

GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation