Dynamic driving scene reconstruction is critical for autonomous driving simulation and closed-loop learning. While recent feed-forward methods have shown promise for 3D reconstruction, they struggle with long-range driving sequences due to quadratic complexity in sequence length and challenges in modeling dynamic objects over extended durations. We propose UFO, a novel recurrent paradigm that combines the benefits of optimization-based and feed-forward methods for efficient long-range 4D reconstruction. Our approach maintains a 4D scene representation that is iteratively refined as new observations arrive, using a visibility-based filtering mechanism to select informative scene tokens and enable efficient processing of long sequences. For dynamic objects, we introduce an object pose-guided modeling approach that supports accurate long-range motion capture. Experiments on the Waymo Open Dataset demonstrate that our method significantly outperforms both per-scene optimization and existing feed-forward methods across various sequence lengths. Notably, our approach can reconstruct 16-second driving logs within 0.5 second while maintaining superior visual quality and geometric accuracy.
Figures & tables
Figure 1: (a) Per-scene optimization methods rely on complex update pipelines to iteratively refine scene representations. (b) Feed-forward methods directly predict 3D representations from image pixels. (c) Our UFO integrates the strengths of both paradigms by abstracting the render-supervise-update process into a single holistic transformer, enabling efficient long-range 4D reconstruction in a recurrent manner.
Figure 2: Overview of our proposed framework. Given a long sequence of multi-view images, we reconstruct the 4D scene in a recurrent manner. (A) At each time step, we update the scene representation by refining previous scene tokens based on the new observation and adding new information from the current frame. (B) To efficiently handle long sequences, we employ a visibility-based filtering mechanism to select relevant scene tokens for updating. A unified transformer model learns to update the scene in a feed-forward manner. (C) Dynamic objects are modeled using 3D bounding boxes and per-Gaussian lifespans, enabling complex motion modeling over time.
Figure 3: Qualitative comparison on novel view synthesis.
Methods
2s Sequences
8s Sequences
16s Sequences ( zero-shot )
PSNR ↑
SSIM ↑
D-RMSE ↓
PSNR ↑
SSIM ↑
D-RMSE ↓
PSNR ↑
SSIM ↑
D-RMSE ↓
3DGS [ 19 ]
21.07
0.578
13.52
19.57
0.517
14.42
17.18
0.454
17.01
PVG [ 5 ]
23.81
0.649
13.82
22.90
0.619
18.24
21.79
0.599
17.21
Street Gaussians [ 46 ]
22.96
0.652
12.15
21.69
0.609
13.17
22.67
0.675
14.88
DeformableGS [ 51 ]
23.40
0.669
11.55
21.47
0.611
13.12
19.79
0.600
15.82
GS-LRM [ 57 ]
25.18
0.753
7.94
21.81
0.584
7.37
16.98
0.500
9.81
Table 1: Comparison to state-of-the-art methods on the Waymo Open Dataset. We compare photorealism and geometric metrics against both per-scene optimization methods and feed-forward methods. We report PSNR, SSIM, and Depth RMSE (D-RMSE).
Figure 4: Inference Time and Memory Usage Comparison. We compare inference time and memory usage for different sequence lengths between STORM and our method. Results are reported excluding Gaussian rendering stage.
Figure 5: Visualization of dynamic object modeling. Lifespan and motion assignment enable accurate modeling of dynamic objects. Top: rendered RGB at a novel timestep. Middle: rendered lifespan map, where blue indicates transient objects with short lifespan. Bottom: rendered motion-assignment map; different colors indicate different objects. We leverage object poses to transform Gaussians accordingly.
Figure 6: Dynamic Object Modeling Performance. We compare novel view synthesis performance on dynamic objects with different number of input frames within a fixed timespan of 2 seconds.
Method
PSNR ↑
SSIM ↑
D-RMSE ↓
(1) w/o iterative training
20.67
0.625
11.81
(2) w/o scene tokens
26.31
0.787
5.31
(3) w/o refinement
27.08
0.824
5.13
(4) w/o lifespan
26.47
0.790
5.21
(5) w/o bounding box
26.88
0.818
5.12
Ours (full)
27.39
0.830
5.10
Table 2: Ablation study on key design components.
Figure 7: Results under Adverse Lighting Conditions.
Figure 8: Additional Qualitative Results on Waymo Open Dataset.
Dynamic scene reconstruction in autonomous driving remains a fundamental challenge due to significant temporal variations, moving objects, and complex scene dynamics. Existing feed-forward 3D models have demonstrated strong performance in static reconstruction but still struggle to capture dynamic motion. To address these limitations, we propose DynamicVGGT, a unified feed-forward framework that extends VGGT from static 3D perception to dynamic 4D reconstruction. Our goal is to model point motion within feed-forward 3D models in a dynamic and temporally coherent manner. To this end, we jointly predict the current and future point maps within a shared reference coordinate system, allowing the model to implicitly learn dynamic point representations through temporal correspondence. To efficiently capture temporal dependencies, we introduce a Motion-aware Temporal Attention (MTA) module that learns motion continuity. Furthermore, we design a Dynamic 3D Gaussian Splatting Head that explicitly models point motion by predicting Gaussian velocities using learnable motion tokens under scene flow supervision. It refines dynamic geometry through continuous 3D Gaussian optimization. Extensive experiments on autonomous driving datasets demonstrate that DynamicVGGT significantly outperforms existing methods in reconstruction accuracy, achieving robust feed-forward 4D dynamic scene reconstruction under complex driving scenarios.
Zhuolin He, Jing Li, Guanghao Li +9
Fudan University · Yinwang Intelligent Technology · Huawei +2
High-fidelity visual reconstruction and novel-view synthesis are essential for realistic closed-loop evaluation in autonomous driving. While 4D Gaussian Splatting (4DGS) offers a promising balance of accuracy and efficiency, existing per-scene optimization methods require costly iterative refinement, rendering them unscalable for extensive urban environments. Conversely, current feed-forward approaches often suffer from degraded photometric quality. To address these limitations, we propose ReconDrive, a feed-forward framework that leverages and extends the 3D foundation model VGGT for rapid, high-fidelity 4DGS generation. Our architecture introduces two core adaptations to tailor the foundation model to dynamic driving scenes: (1) Hybrid Gaussian Prediction Heads, which decouple the regression of spatial coordinates and appearance attributes to overcome the photometric deficiencies inherent in generalized foundation features; and (2) a Static-Dynamic 4D Composition strategy that explicitly captures temporal motion via velocity modeling to represent complex dynamic environments. Benchmarked on nuScenes, ReconDrive significantly outperforms existing feed-forward baselines in reconstruction, novel-view synthesis, and 3D perception. It achieves performance competitive with per-scene optimization while being orders of magnitude faster, providing a scalable and practical solution for realistic driving simulation.
Haibao Yu, Kuntao Xiao, Jiahang Wang +7
The University of Hong Kong · Tuojing Intelligence · King’s College London +2
High-fidelity reconstruction of driving scenes is crucial for autonomous driving. While recent feedforward 3D Gaussian Splatting (3DGS) methods enable fast reconstruction, their per-pixel Gaussian prediction paradigm often suffers from multi-view inconsistency and layering artifacts. Moreover, existing methods often model dynamic instances via dense flow prediction, which lacks explicit cross-view correspondence and instance-level consistency. In this paper, we propose PointForward, a feedforward driving reconstruction framework through point-aligned representations. Unlike pixel-aligned methods, we initialize sparse 3D queries in world space and aggregate multi-view image information via spatial-temporal fusion onto these queries, enforcing explicit cross-view consistency in a single feedforward pass. To handle scene dynamics, we introduce scene graphs that explicitly organize moving instances during reconstruction. By leveraging 3D bounding boxes, our method enables instance-level motion propagation and temporally consistent dynamic representations. Extensive experiments demonstrate that PointForward achieves state-of-the-art performance on large-scale driving benchmarks. The code will be available upon the publication of the paper.
Cheng Chi, Xianqi Wang, Hongcheng Luo +9
Xiaomi EV · Huazhong University of Science and Technology · Zhejiang University