Recent advancements in 4D scene reconstruction, particularly those leveraging diffusion priors, have shown promise for novel view synthesis in autonomous driving. However, these methods often process frames independently or in a view-by-view manner, leading to a critical lack of spatio-temporal synergy. This results in spatial misalignment across cameras and temporal drift in sequences. We propose DriveFix, a novel multi-view restoration framework that ensures spatio-temporal coherence for driving scenes. Our approach employs an interleaved diffusion transformer architecture with specialized blocks to explicitly model both temporal dependencies and cross-camera spatial consistency. By conditioning the generation on historical context and integrating geometry-aware training losses, DriveFix enforces that the restored views adhere to a unified 3D geometry. This enables the consistent propagation of high-fidelity textures and significantly reduces artifacts. Extensive evaluations on the Waymo, nuScenes, and PandaSet datasets demonstrate that DriveFix achieves state-of-the-art performance in both reconstruction and novel view synthesis, marking a substantial step toward robust 4D world modeling for real-world deployment.
Figures & tables
Figure 1: Comparison of novel view synthesis results between baseline reconstruction methods and our DriveFix. DriveFix produces significantly cleaner renderings with sharper textures and fewer artifacts, particularly in distant regions and overlapping camera views.
Figure 2: Overview of the DriveFix framework. Given corrupted multi-view renderings from a base 4D simulator, our approach first constructs hybrid historical context by combining degraded and ground-truth frames from previous time steps. This context, along with the current corrupted frames and structural guidance such as depth and semantics, is fed into an interleaved diffusion transformer architecture. The model alternates between temporal attention blocks, which propagate high-fidelity textures from history to suppress flickering, and spatial attention blocks with camera-geometry embeddings to enforce cross-view geometric consistency across all synchronized cameras. The output is a set of spatio-temporally coherent surround-view frames, which serve as pseudo-ground-truth to re-optimize the underlying 4D scene representation for photorealistic novel view synthesis.
Method
Reconstruction
Interpolation
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
S-NeRF [ 41 ]
19.67
0.528
0.387
19.22
0.515
0.400
StreetSurf [ 9 ]
26.70
0.846
0.372
23.78
0.822
0.401
3DGS [ 17 ]
27.99
0.866
0.293
25.08
0.822
0.319
NSG [ 4 ]
24.08
0.656
0.441
21.01
0.571
0.487
Mars [ 20 ]
21.81
0.681
0.430
20.69
0.636
0.453
Table 1: Comparison with existing methods on Waymo Open Dataset
Table 4Table 5
Figure 3: Qualitative comparison of scene restoration and novel view synthesis. We evaluate DriveFix against several state-of-the-art baselines, including HUGSIM, SplatAD, and Difix3D+, with ground truth provided as a reference. Red bounding boxes highlight challenging regions where baseline methods frequently exhibit severe artifacts, such as blurring, ghosting, and geometric distortions, particularly in distant objects and overlapping camera views. In contrast, DriveFix produces significantly cleaner renderings with sharper textures and maintains spatio-temporal continuity.
Figure 4: Qualitative ablation study of DriveFix on the Waymo Open Dataset. Each column presents the restoration results of a variant of our framework with a specific component removed. From left to right:(a) without cross-view spatial attention, (b) without temporal attention, (c) without historical context conditioning, (d) without geometry-aware alignment loss, and (e) without depth and semantic conditioning, (f) Full DriveFix model. The removal of any component leads to visible degradation, such as ghosting, blurring, or temporal inconsistency, highlighting the necessity of each module in achieving spatio-temporally coherent restoration.
Method
Reconstruction
Interpolation
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Cross-View Attention
30.61
0.914
0.204
29.60
0.902
0.211
w/o Temporal Attention
32.23
0.918
0.201
30.42
0.909
0.187
w/o Historical Context
31.68
0.910
0.211
28.20
0.868
0.235
w/o Alignment Loss
33.57
0.928
0.181
30.47
0.913
0.185
w/o Depth and Semantic
33.73
0.927
0.190
30.90
0.895
0.195
Table 6: Ablation study of key components on the Waymo Open Dataset.
Figure 5: Ablation study on the number of fine-tuning steps for the alignment loss. All metrics get optimal around 3000 iterations, after which further fine-tuning yields only marginal gains or slight degradation. This suggests that 3,000 iterations offer the optimal trade-off between geometric fidelity and generative diversity.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Quantitative comparisons of view interpolation on nuScenes.
Figure 7: Quantitative comparison on the Pandaset. The camera is laterally shifted for 2 meters to left or right.
High-fidelity visual reconstruction and novel-view synthesis are essential for realistic closed-loop evaluation in autonomous driving. While 4D Gaussian Splatting (4DGS) offers a promising balance of accuracy and efficiency, existing per-scene optimization methods require costly iterative refinement, rendering them unscalable for extensive urban environments. Conversely, current feed-forward approaches often suffer from degraded photometric quality. To address these limitations, we propose ReconDrive, a feed-forward framework that leverages and extends the 3D foundation model VGGT for rapid, high-fidelity 4DGS generation. Our architecture introduces two core adaptations to tailor the foundation model to dynamic driving scenes: (1) Hybrid Gaussian Prediction Heads, which decouple the regression of spatial coordinates and appearance attributes to overcome the photometric deficiencies inherent in generalized foundation features; and (2) a Static-Dynamic 4D Composition strategy that explicitly captures temporal motion via velocity modeling to represent complex dynamic environments. Benchmarked on nuScenes, ReconDrive significantly outperforms existing feed-forward baselines in reconstruction, novel-view synthesis, and 3D perception. It achieves performance competitive with per-scene optimization while being orders of magnitude faster, providing a scalable and practical solution for realistic driving simulation.
Haibao Yu, Kuntao Xiao, Jiahang Wang +7
The University of Hong Kong · Tuojing Intelligence · King’s College London +2
High-fidelity street scene reconstruction is pivotal for end-to-end autonomous driving simulation, where novel-view synthesis (NVS) and time-varying information modeling are two fundamental capabilities to facilitate closed-loop training. However, existing 3DGS methods and their 4D extensions fail to simultaneously achieve both. To bridge this gap, we establish an information-geometric diagnostic framework, revealing that this limitation stems from a credit assignment dilemma between spatial and temporal parameters. Specifically, the deterministic coupling between viewpoint and time in single-source observation creates a low-rank structure that induces massive null-space ambiguity between static view-dependent and dynamic time-varying components. Temporal information overshadows spatial cues, causing the estimation variance of spatial parameters to diverge. To address this issue, we propose Orthogonal Projected Gradient (OPG), a hierarchical training method designed to restore spatial identifiability. OPG prioritizes the integrity of spatial representations by securing them in an initial stage, then restricts temporal updates to the spatial null space, enabling proactive credit assignment. While OPG isolates temporal updates algebraically, Temporal Regularization Strategy is proposed to further refine the temporal solution space by imposing a smoothness constraint based on the physical prior of consistent appearance evolution, ensuring that the reconstructed scene remains physically consistent in closed-loop simulation. Extensive experiments demonstrate that our method not only maintains stable NVS capabilities but also demonstrates superior performance in traditional observation-reproducing metrics, which indirectly reflect the capability of modeling temporal dynamics.
Bowyn Tan, Yutong Xie, Bai Huang +5
Tsinghua University · Meituan · Central University of Finance and Economics
The synthesis of spatiotemporally coherent 4D content presents fundamental challenges in computer vision, requiring simultaneous modeling of high-fidelity spatial representations and physically plausible temporal dynamics. Current approaches often struggle to maintain view consistency while handling complex scene dynamics, particularly in large-scale environments with multiple interacting elements. This work introduces Dream4D, a novel framework that bridges this gap through a synergy of controllable video generation and neural 4D reconstruction. Our approach seamlessly combines a two-stage architecture: it first predicts optimal camera trajectories from a single image using few-shot learning, then generates geometrically consistent multi-view sequences via a specialized pose-conditioned diffusion process, which are finally converted into a persistent 4D representation. This framework is the first to leverage both rich temporal priors from video diffusion models and geometric awareness of the reconstruction models, which significantly facilitates 4D generation and shows higher quality (e.g., mPSNR, mSSIM) over existing methods.
Xiaoyan Liu, Kangrui Li, Jiaxin Liu +2
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales +1