Feed-forward 4D reconstruction enables efficient creation of renderable driving scenes for autonomous driving simulation. However, most existing methods rely heavily on dataset-provided cameras, LiDAR depth, or 3D box annotations, limiting their applicability to unannotated driving videos. Pretrained visual models can predict cameras, depth, and object tracks from RGB alone, but their geometry is mutually inconsistent: multi-view predictions come in short windows with independent scales and frames, while monocular depth is metric only per image. Our key insight is that, once reconciled into a single metric frame, these priors can replace dataset annotations for feed-forward 4D reconstruction. We present FreeForward4D, which calibrates overlapping prediction windows against monocular metric depth and registers them into one frame across time. On this geometry, 2D tracks are associated across cameras in 3D, and objects move through hierarchical motion that combines a shared object velocity with local Gaussian corrections, so that both training and inference use only RGB videos. On Waymo, FreeForward4D achieves 27.66 dB PSNR and 0.836 SSIM while recovering metric-scale cameras and depth without scale alignment. It surpasses STORM, SLARM-F, UFO, DGGT, and StreetForward, which rely on dataset annotations, in full-image PSNR and attains the best RGB fidelity on 8-second windows. RGB-only fine-tuning further adapts the framework to nuScenes and Argoverse 2. Code will be released.
Figures & tables
Figure 2: FreeForward4D pipeline. (a) Frozen visual priors provide windowed cameras, pseudo-depth, and 2D object tracks. (b) Geometric prior integration calibrates the windows against monocular metric depth and aligns them into a single metric frame (before vs. after alignment). (c) This geometry enables cross-view object association, object motion is modeled by a shared velocity and local Gaussian trajectories, and temporal weights aggregate Gaussians from different context times. The dashed line carries 2D tracks.
Methods
Annotation-free
Dynamic regions
Full image
Depth
Train
Test
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
D-RMSE (m) ↓
Pose-conditioned feed-forward methods
STORM †
×
×
22.04
0.623
25.85
0.803
5.47
UFO
×
×
–
–
27.26
0.825
5.45
SLARM-F †
×
×
22.95
0.672
26.98
0.827
4.78
PointForward
×
×
25.01
0.768
28.48
0.861
3.35
Table 1: Waymo novel-time synthesis. Each clip contains four context timestamps from three cameras and 16 target timestamps (12 interpolated and 4 extrapolated; per-type results in supplementary Sec. D). Train/Test indicate whether the method avoids dataset-provided cameras, LiDAR depth, and 3D box annotations in the respective phase.
Figure 3: Waymo qualitative comparison at extrapolated timestamps. Each example shows the left, front, and right cameras; rows correspond to ground truth, STORM, SLARM-F, and ours. Enlarged regions illustrate vehicle boundaries, temporal ghosting, and fine scene details. The baselines require dataset cameras; ours renders at cameras recovered by its own front end.
Figure 4: Novel-view rendering and Gaussian trajectories on Waymo. Top: renderings from virtual cameras elevated by 0.5 m relative to the input views. Bottom: time-colored trajectories of selected Gaussians on moving vehicles over 0–1.5 s.
Rendering and depth quality
Feed-
Annot.-
Method
PSNR ↑
SSIM ↑
D-RMSE (m) ↓
forward
free
Waymo Open
3DGS [ 15 ]
19.57
0.517
14.42
×
×
PVG [ 5 ]
22.90
0.619
18.24
×
×
Street Gaussians [ 45 ]
21.69
0.609
13.17
×
×
DeformableGS [ 50 ]
21.47
0.611
13.12
×
×
Table 2: Extended 8s reconstruction. Each Waymo window uses 16 context frames and 64 target timestamps. Baseline results are reported by UFO [ 30 ] . Results on nuScenes and Argoverse2 are obtained through RGB-only fine-tuning of our Waymo-trained 8s model.
Methods
Annotation-free
nuScenes
Argoverse2
Train
Test
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
STORM [ 47 ]
×
×
24.54
0.784
24.97
0.791
DGGT [ 3 ]
×
✓
26.63
0.813
26.96
0.831
Ours
✓
✓
26.90
0.813
26.52
0.829
Table 3: Cross-dataset adaptation on nuScenes and Argoverse2. Full-image novel-time synthesis after adaptation to each target dataset. Train/Test indicate whether the method avoids dataset-provided cameras, LiDAR depth, and 3D box annotations in the respective phase. Ours is fine-tuned from the Waymo model using only RGB-derived conditions, whereas the baselines are trained on each target dataset with dataset annotations; baseline results are from their publications, with STORM as reported by DGGT [ 3 ] .
Full image
Dynamic regions
Depth
Variant
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
D-RMSE (m) ↓
w/o geom. integ.
23.73
0.677
22.57
0.608
7.99
w/o object motion
27.26
0.829
22.77
0.651
7.22
w/o cross-view assoc.
27.62
0.835
24.26
0.726
7.06
w/o local motion
27.55
0.833
24.12
0.719
7.06
w/o accel. & jerk
27.64
0.835
24.30
0.727
7.05
Table 4: Ablation study on Waymo 2s. w/o geom. integ. uses, for each clip, the uncalibrated VGGT- Ω cameras of the single window covering it; w/o object motion removes grouping, foreground refinement, and shared velocity; w/o local motion retains shared object velocity only; w/o accel. & jerk retains first-order residual motion; w/o normalization removes per-object normalization of temporal weights.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Mean
Median
P90
ATE, 2s clip (m)
1.08
0.58
2.55
RPE, 0.5 s (m)
0.53
0.27
1.20
Rotation error ( ∘ )
1.52
1.20
2.47
Appendix
Table S5: Camera accuracy on Waymo validation. 1,818 2s clips across 202 scenes.
Full image
Dynamic regions
Depth
Targets
Images
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
D-RMSE ↓
Interp.
72,132
28.09
0.848
24.88
0.755
6.96
Extrap.
24,228
26.39
0.798
22.96
0.660
7.35
All
96,360
27.66
0.836
24.36
0.729
7.06
Appendix
Table S6 : Waymo 2s results by target type. Full-image and dynamic-region RGB fidelity and depth error.
Method
Frames
PSNR ↑
SSIM ↑
DGGT [ 3 ]
see text
27.41
0.846
Ours
0 – 15
28.27
0.850
Ours
intermediate only
27.57
0.829
Appendix
Table S7 : Waymo 2s under the protocol of the official DGGT code. Frames: scored frames of each clip. DGGT as reported in its paper.
Figure S5 : Additional Waymo comparisons. Rows show the ground truth (GT), STORM, SLARM-F, and ours, illustrating nearby vehicles, occlusion, intersections, and lateral traffic.
Figure S6 : Additional Waymo comparisons. Same layout as Fig. S5 , illustrating urban structure, road traffic, and complex visibility.
Figure S7 : Additional novel-view renderings and Gaussian trajectories. Renderings from a raised camera and predicted trajectories of Gaussians on moving vehicles, with the same 2s three-view model and 0–1.5 s interval as Fig. 4 of the main paper.
Figure S8 : Scene-level static reconstruction and local views. Each BEV and zoom-in pair shares a static Gaussian representation merged from approximately 20 s of RGB. Orange arrows locate the virtual cameras.
Figure S9 : Wider static reconstructions. Two driving scenes and zoom-ins, each merged from approximately 20 s of static Gaussians. The top row is a wider view of the scene in Fig. S8 (a,b).
Figure S10 : Additional static reconstructions. Zoom-ins under the same conditions as Fig. S9 . The bottom row is a wider view of the scene in Fig. S8 (c,d).
Feed-forward 4D driving-scene reconstruction enables efficient modeling of dynamic autonomous-driving scenes, but current methods suffer from two complementary failure modes: semantic leakage of traffic participants into the static representation and geometric drift of Gaussian locations across viewpoints. We propose IGRecon4D, a feed-forward dynamic scene reconstruction framework that addresses these failures with training-time instance-prior distillation and LiDAR-anchored Gaussian positioning. Specifically, an instance-level semantic branch provides a potentially dynamic participant prior, which is distilled into dynamic-mask learning through the Instance-Dynamic Consistency Constraint to encourage high-recall localization of traffic-participant regions prone to dynamic-static leakage. In parallel, sparse LiDAR measurements guide dense depth estimation, and the resulting depths are back-projected to anchor Gaussian positions for improved multi-view geometric consistency. Experiments on the Waymo Open Dataset and zero-shot evaluation on nuScenes demonstrate improved full-image reconstruction on the main benchmark and stronger traffic-participant-region reconstruction under cross-dataset evaluation.
Yifei Tong, Ziyang Huang, Yizhao Han +2
Department of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China.
High-fidelity visual reconstruction and novel-view synthesis are essential for realistic closed-loop evaluation in autonomous driving. While 4D Gaussian Splatting (4DGS) offers a promising balance of accuracy and efficiency, existing per-scene optimization methods require costly iterative refinement, rendering them unscalable for extensive urban environments. Conversely, current feed-forward approaches often suffer from degraded photometric quality. To address these limitations, we propose ReconDrive, a feed-forward framework that leverages and extends the 3D foundation model VGGT for rapid, high-fidelity 4DGS generation. Our architecture introduces two core adaptations to tailor the foundation model to dynamic driving scenes: (1) Hybrid Gaussian Prediction Heads, which decouple the regression of spatial coordinates and appearance attributes to overcome the photometric deficiencies inherent in generalized foundation features; and (2) a Static-Dynamic 4D Composition strategy that explicitly captures temporal motion via velocity modeling to represent complex dynamic environments. Benchmarked on nuScenes, ReconDrive significantly outperforms existing feed-forward baselines in reconstruction, novel-view synthesis, and 3D perception. It achieves performance competitive with per-scene optimization while being orders of magnitude faster, providing a scalable and practical solution for realistic driving simulation.
Haibao Yu, Kuntao Xiao, Jiahang Wang +7
The University of Hong Kong · Tuojing Intelligence · King’s College London +2
We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by 13.4% relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.