UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling

Kaiyuan Tan; Yingying Shen; Mingfei Tu; Haohui Zhu; Bing Wang; Guang Chen; Hangjun Ye; Haiyang Sun

cs.CVFeb 24, 2026

UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling

Authors: Kaiyuan Tan, Yingying Shen, Mingfei Tu, Haohui Zhu, Bing Wang, Guang Chen, Hangjun Ye, Haiyang Sun

Organizations: Xiaomi EV · UIUC

Abstract

Dynamic driving scene reconstruction is critical for autonomous driving simulation and closed-loop learning. While recent feed-forward methods have shown promise for 3D reconstruction, they struggle with long-range driving sequences due to quadratic complexity in sequence length and challenges in modeling dynamic objects over extended durations. We propose UFO, a novel recurrent paradigm that combines the benefits of optimization-based and feed-forward methods for efficient long-range 4D reconstruction. Our approach maintains a 4D scene representation that is iteratively refined as new observations arrive, using a visibility-based filtering mechanism to select informative scene tokens and enable efficient processing of long sequences. For dynamic objects, we introduce an object pose-guided modeling approach that supports accurate long-range motion capture. Experiments on the Waymo Open Dataset demonstrate that our method significantly outperforms both per-scene optimization and existing feed-forward methods across various sequence lengths. Notably, our approach can reconstruct 16-second driving logs within 0.5 second while maintaining superior visual quality and geometric accuracy.

Figures & tables

Figure 1: (a) Per-scene optimization methods rely on complex update pipelines to iteratively refine scene representations. (b) Feed-forward methods directly predict 3D representations from image pixels. (c) Our UFO integrates the strengths of both paradigms by abstracting the render-supervise-update process into a single holistic transformer, enabling efficient long-range 4D reconstruction in a recurrent manner.

Figure 2: Overview of our proposed framework. Given a long sequence of multi-view images, we reconstruct the 4D scene in a recurrent manner. (A) At each time step, we update the scene representation by refining previous scene tokens based on the new observation and adding new information from the current frame. (B) To efficiently handle long sequences, we employ a visibility-based filtering mechanism to select relevant scene tokens for updating. A unified transformer model learns to update the scene in a feed-forward manner. (C) Dynamic objects are modeled using 3D bounding boxes and per-Gaussian lifespans, enabling complex motion modeling over time.

Figure 3: Qualitative comparison on novel view synthesis.

Methods	2s Sequences			8s Sequences			16s Sequences ( zero-shot )
Methods	PSNR $\uparrow$	SSIM $\uparrow$	D-RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	D-RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	D-RMSE $\downarrow$
3DGS [ 19 ]	21.07	0.578	13.52	19.57	0.517	14.42	17.18	0.454	17.01
PVG [ 5 ]	23.81	0.649	13.82	22.90	0.619	18.24	21.79	0.599	17.21
Street Gaussians [ 46 ]	22.96	0.652	12.15	21.69	0.609	13.17	22.67	0.675	14.88
DeformableGS [ 51 ]	23.40	0.669	11.55	21.47	0.611	13.12	19.79	0.600	15.82
GS-LRM [ 57 ]	25.18	0.753	7.94	21.81	0.584	7.37	16.98	0.500	9.81

Table 1: Comparison to state-of-the-art methods on the Waymo Open Dataset. We compare photorealism and geometric metrics against both per-scene optimization methods and feed-forward methods. We report PSNR, SSIM, and Depth RMSE (D-RMSE).

Figure 4: Inference Time and Memory Usage Comparison. We compare inference time and memory usage for different sequence lengths between STORM and our method. Results are reported excluding Gaussian rendering stage.

Figure 5: Visualization of dynamic object modeling. Lifespan and motion assignment enable accurate modeling of dynamic objects. Top: rendered RGB at a novel timestep. Middle: rendered lifespan map, where blue indicates transient objects with short lifespan. Bottom: rendered motion-assignment map; different colors indicate different objects. We leverage object poses to transform Gaussians accordingly.

Figure 6: Dynamic Object Modeling Performance. We compare novel view synthesis performance on dynamic objects with different number of input frames within a fixed timespan of 2 seconds.

Method	PSNR $\uparrow$	SSIM $\uparrow$	D-RMSE $\downarrow$
(1) w/o iterative training	20.67	0.625	11.81
(2) w/o scene tokens	26.31	0.787	5.31
(3) w/o refinement	27.08	0.824	5.13
(4) w/o lifespan	26.47	0.790	5.21
(5) w/o bounding box	26.88	0.818	5.12
Ours (full)	27.39	0.830	5.10

Table 2: Ablation study on key design components.

Figure 7: Results under Adverse Lighting Conditions.

Figure 8: Additional Qualitative Results on Waymo Open Dataset.

Explore similar work

DynamicVGGT: Learning Dynamic Point Maps for 4D Scene Reconstruction in Autonomous Driving
Mar 9, 2026Zhuolin He, Jing Li, Guanghao Li +9Dynamic Scene Reconstruction Autonomous Driving Perception
Fudan University · Yinwang Intelligent Technology · Huawei +2
ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction
Mar 8, 2026Haibao Yu, Kuntao Xiao, Jiahang Wang +7Dynamic Scene Reconstruction 3D Gaussian Splatting
The University of Hong Kong · Tuojing Intelligence · King’s College London +2
PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations
May 12, 2026Cheng Chi, Xianqi Wang, Hongcheng Luo +93D Reconstruction Multi-View 3D Reconstruction
Xiaomi EV · Huazhong University of Science and Technology · Zhejiang University

Mar 9, 2026J/K move · Enter open · S save

DynamicVGGT: Learning Dynamic Point Maps for 4D Scene Reconstruction in Autonomous Driving

Zhuolin He, Jing Li, Guanghao Li, Xiaolei Chen+8

Fudan University · Yinwang Intelligent Technology · Huawei+2

Dynamic Scene Reconstruction Autonomous Driving Perception Feed-Forward 3D Reconstruction 3D Gaussian Splatting

Open paper AskPDF arXiv

Dynamic scene reconstruction in autonomous driving remains a fundamental challenge due to significant temporal variations, moving objects, and complex scene dynamics. Existing feed-forward 3D models have demonstrated strong performance in static reconstruction but still struggle to capture dynamic motion. To address these limitations, we propose DynamicVGGT, a unified feed-forward framework that extends VGGT from static 3D perception to dynamic 4D reconstruction. Our goal is to model point motion within feed-forward 3D models in a dynamic and temporally coherent manner. To this end, we jointly predict the current and future point maps within a shared reference coordinate system, allowing the model to implicitly learn dynamic point representations through temporal correspondence. To efficiently capture temporal dependencies, we introduce a Motion-aware Temporal Attention (MTA) module that learns motion continuity. Furthermore, we design a Dynamic 3D Gaussian Splatting Head that explicitly models point motion by predicting Gaussian velocities using learnable motion tokens under scene flow supervision. It refines dynamic geometry through continuous 3D Gaussian optimization. Extensive experiments on autonomous driving datasets demonstrate that DynamicVGGT significantly outperforms existing methods in reconstruction accuracy, achieving robust feed-forward 4D dynamic scene reconstruction under complex driving scenarios.

Abstract

Explore similar work

DynamicVGGT: Learning Dynamic Point Maps for 4D Scene Reconstruction in Autonomous Driving

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction

PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations