Feed-forward 4D reconstruction enables efficient creation of renderable driving scenes for autonomous driving simulation. However, most existing methods rely heavily on dataset-provided cameras, LiDAR depth, or 3D box annotations, limiting their applicability to unannotated driving videos. Pretrained visual models can predict cameras, depth, and object tracks from RGB alone, but their geometry is mutually inconsistent: multi-view predictions come in short windows with independent scales and frames, while monocular depth is metric only per image. Our key insight is that, once reconciled into a single metric frame, these priors can replace dataset annotations for feed-forward 4D reconstruction. We present FreeForward4D, which calibrates overlapping prediction windows against monocular metric depth and registers them into one frame across time. On this geometry, 2D tracks are associated across cameras in 3D, and objects move through hierarchical motion that combines a shared object velocity with local Gaussian corrections, so that both training and inference use only RGB videos. On Waymo, FreeForward4D achieves 27.66 dB PSNR and 0.836 SSIM while recovering metric-scale cameras and depth without scale alignment. It surpasses STORM, SLARM-F, UFO, DGGT, and StreetForward, which rely on dataset annotations, in full-image PSNR and attains the best RGB fidelity on 8-second windows. RGB-only fine-tuning further adapts the framework to nuScenes and Argoverse 2. Code will be released.
Figures & tables
Figure 2: FreeForward4D pipeline. (a) Frozen visual priors provide windowed cameras, pseudo-depth, and 2D object tracks. (b) Geometric prior integration calibrates the windows against monocular metric depth and aligns them into a single metric frame (before vs. after alignment). (c) This geometry enables cross-view object association, object motion is modeled by a shared velocity and local Gaussian trajectories, and temporal weights aggregate Gaussians from different context times. The dashed line carries 2D tracks.
Methods
Annotation-free
Dynamic regions
Full image
Depth
Train
Test
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
D-RMSE (m) ↓
Pose-conditioned feed-forward methods
STORM †
×
×
22.04
0.623
25.85
0.803
5.47
UFO
×
×
–
–
27.26
0.825
5.45
SLARM-F †
×
×
22.95
0.672
26.98
0.827
4.78
PointForward
×
×
25.01
0.768
28.48
0.861
3.35
Table 1: Waymo novel-time synthesis. Each clip contains four context timestamps from three cameras and 16 target timestamps (12 interpolated and 4 extrapolated; per-type results in supplementary Sec. D). Train/Test indicate whether the method avoids dataset-provided cameras, LiDAR depth, and 3D box annotations in the respective phase.
Figure 3: Waymo qualitative comparison at extrapolated timestamps. Each example shows the left, front, and right cameras; rows correspond to ground truth, STORM, SLARM-F, and ours. Enlarged regions illustrate vehicle boundaries, temporal ghosting, and fine scene details. The baselines require dataset cameras; ours renders at cameras recovered by its own front end.
Figure 4: Novel-view rendering and Gaussian trajectories on Waymo. Top: renderings from virtual cameras elevated by 0.5 m relative to the input views. Bottom: time-colored trajectories of selected Gaussians on moving vehicles over 0–1.5 s.
Rendering and depth quality
Feed-
Annot.-
Method
PSNR ↑
SSIM ↑
D-RMSE (m) ↓
forward
free
Waymo Open
3DGS [ 15 ]
19.57
0.517
14.42
×
×
PVG [ 5 ]
22.90
0.619
18.24
×
×
Street Gaussians [ 45 ]
21.69
0.609
13.17
×
×
DeformableGS [ 50 ]
21.47
0.611
13.12
×
×
Table 2: Extended 8s reconstruction. Each Waymo window uses 16 context frames and 64 target timestamps. Baseline results are reported by UFO [ 30 ] . Results on nuScenes and Argoverse2 are obtained through RGB-only fine-tuning of our Waymo-trained 8s model.
Methods
Annotation-free
nuScenes
Argoverse2
Train
Test
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
STORM [ 47 ]
×
×
24.54
0.784
24.97
0.791
DGGT [ 3 ]
×
✓
26.63
0.813
26.96
0.831
Ours
✓
✓
26.90
0.813
26.52
0.829
Table 3: Cross-dataset adaptation on nuScenes and Argoverse2. Full-image novel-time synthesis after adaptation to each target dataset. Train/Test indicate whether the method avoids dataset-provided cameras, LiDAR depth, and 3D box annotations in the respective phase. Ours is fine-tuned from the Waymo model using only RGB-derived conditions, whereas the baselines are trained on each target dataset with dataset annotations; baseline results are from their publications, with STORM as reported by DGGT [ 3 ] .
Full image
Dynamic regions
Depth
Variant
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
D-RMSE (m) ↓
w/o geom. integ.
23.73
0.677
22.57
0.608
7.99
w/o object motion
27.26
0.829
22.77
0.651
7.22
w/o cross-view assoc.
27.62
0.835
24.26
0.726
7.06
w/o local motion
27.55
0.833
24.12
0.719
7.06
w/o accel. & jerk
27.64
0.835
24.30
0.727
7.05
Table 4: Ablation study on Waymo 2s. w/o geom. integ. uses, for each clip, the uncalibrated VGGT- Ω cameras of the single window covering it; w/o object motion removes grouping, foreground refinement, and shared velocity; w/o local motion retains shared object velocity only; w/o accel. & jerk retains first-order residual motion; w/o normalization removes per-object normalization of temporal weights.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Mean
Median
P90
ATE, 2s clip (m)
1.08
0.58
2.55
RPE, 0.5 s (m)
0.53
0.27
1.20
Rotation error ( ∘ )
1.52
1.20
2.47
Appendix
Table S5: Camera accuracy on Waymo validation. 1,818 2s clips across 202 scenes.
Full image
Dynamic regions
Depth
Targets
Images
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
D-RMSE ↓
Interp.
72,132
28.09
0.848
24.88
0.755
6.96
Extrap.
24,228
26.39
0.798
22.96
0.660
7.35
All
96,360
27.66
0.836
24.36
0.729
7.06
Appendix
Table S6 : Waymo 2s results by target type. Full-image and dynamic-region RGB fidelity and depth error.
Method
Frames
PSNR ↑
SSIM ↑
DGGT [ 3 ]
see text
27.41
0.846
Ours
0 – 15
28.27
0.850
Ours
intermediate only
27.57
0.829
Appendix
Table S7 : Waymo 2s under the protocol of the official DGGT code. Frames: scored frames of each clip. DGGT as reported in its paper.
Figure S5 : Additional Waymo comparisons. Rows show the ground truth (GT), STORM, SLARM-F, and ours, illustrating nearby vehicles, occlusion, intersections, and lateral traffic.
Figure S6 : Additional Waymo comparisons. Same layout as Fig. S5 , illustrating urban structure, road traffic, and complex visibility.
Figure S7 : Additional novel-view renderings and Gaussian trajectories. Renderings from a raised camera and predicted trajectories of Gaussians on moving vehicles, with the same 2s three-view model and 0–1.5 s interval as Fig. 4 of the main paper.
Figure S8 : Scene-level static reconstruction and local views. Each BEV and zoom-in pair shares a static Gaussian representation merged from approximately 20 s of RGB. Orange arrows locate the virtual cameras.
Figure S9 : Wider static reconstructions. Two driving scenes and zoom-ins, each merged from approximately 20 s of static Gaussians. The top row is a wider view of the scene in Fig. S8 (a,b).
Figure S10 : Additional static reconstructions. Zoom-ins under the same conditions as Fig. S9 . The bottom row is a wider view of the scene in Fig. S8 (c,d).