Simulation is essential to the development and evaluation of autonomous robots such as self-driving vehicles. Neural reconstruction is emerging as a promising solution as it enables simulating a wide variety of scenarios from real-world data alone in an automated and scalable way. However, while methods such as NeRF and 3D Gaussian Splatting can produce visually compelling results, they often exhibit artifacts particularly when rendering novel views, and fail to realistically integrate inserted dynamic objects, especially when they were captured from different scenes. To overcome these limitations, we introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings from such imperfect scenes into temporally consistent outputs while improving their realism. At its core is a single-step temporally-conditioned enhancer that is converted from a pretrained multi-step image diffusion model, capable of running in online simulators on a single GPU. The key to training it effectively is a custom data curation pipeline that constructs synthetic-real pairs emphasizing appearance harmonization, artifact correction, and lighting realism. The result is a scalable system that significantly elevates simulation fidelity in both research and production environments.
Figures & tables
Figure 2: Overview of the data curation pipeline ( top ) and model architecture ( bottom ) of DiffusionHarmonizer. We use a single-step temporally conditioned enhancement model, that is converted from a pretrained multi-step image diffusion model. To train it effectively, we develop a data curation pipeline to construct synthetic–real pairs emphasizing harmonization, artifact correction and lighting realism.
Figure 3: Comparison with Image and Video Editing Baselines on Out-of-Domain Testing Data. Our method harmonizes color tone and synthesizes realistic lighting and shadows, while editing baselines often fail to produce physically plausible shadowing. Although both can reduce reconstruction artifacts, baselines tend to hallucinate inconsistent content and over-edit well-reconstructed regions, whereas our method preserves scene geometry and input structure. Moreover, image-editing baselines introduce frame-to-frame jitter, whereas our model maintains strong temporal coherence.
Figure 4: Comparison with Harmonization Baselines Methods. While both our method and harmonization baselines adjust foreground appearance, the baselines fail to synthesize realistic shadows, resulting in less coherent composites.
Novel Trajectory Simulation (In-domain)
Object Insertion Simulation (Out-of-domain)
Method
Inference Speed ↓
FID ↓
FVD ↓
DINO Struct. ↑
Temporal Cons. ↑
FID ↓
FVD ↓
DINO Struct. ↑
Temporal Cons. ↑
SDEdit (SD 3)
398ms
129.92
753.39
0.7075
0.9661
106.17
506.91
0.6345
0.9524
InstructPix2Pix
555ms
153.94
680.92
0.6339
0.9203
128.72
573.20
0.5793
0.9085
Wan-Video V2V
2827ms
134.98
506.86
0.8289
0.9828
104.42
474.96
0.8226
0.9675
Our Model
212ms
120.23
470.11
0.9215
0.9827
101.27
453.17
0. 9096
0.9670
Table 1: Quantitative Comparison on Novel Trajectory Simulation (In-domain Test) and Object Insertion Simulation (Out-of-domain Test) Datasets. Our method outperforms all editing baselines in perceptual quality (lower FID/FVD) and preserves scene structure more faithfully (lower DINO-Struct-Dist). It also achieves strong temporal consistency (measured by VBench++ temporal flickering score), surpassing image-editing methods and matching video diffusion models, with only a marginal gap to WAN V2V.
Relighting Data (Images)
PBR Shadow Data (Videos)
ISP Modification Data (Videos)
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
SDEdit (SD 3)
15.17
0.9803
0.0098
15.65
0.9891
0.0095
17.46
0.9850
0.0112
InstructPix2Pix
15.35
0.9812
0.0112
16.46
0.9905
0.0083
16.71
0.9856
0.0105
Wan-Video V2V
N/A
N/A
N/A
14.21
0.9867
0.098
15.81
0.9814
0.1335
Our Model
23.93
0.9938
0.0042
26.31
0.9966
0.0028
28.10
0.9974
0.0020
Table 2: Quantitative Results on Relighting , PBR Shadow , and ISP Modification Holdout Sets. Our method achieves substantially better PSNR, SSIM, and LPIPS, closely matching real-world references.
Method
ISP Modification Data
Speed
PSNR
SSIM
LPIPS
FID
FVD
Temporal Cons.
VHTT
63ms
20.96
0.9921
0.0049
46.23
168.83
0.9838
Ke et al . [ 17 ]
10ms
25.98
0.9956
0.0037
61.51
170.16
0.9872
Ours
212ms
28.58
0.9971
0.0021
42.03
158.27
0.9805
Table 3: Quantitative Comparison with Harmonization Baselines. Evaluated on the harmonization subset (ISP Modification), our method outperforms all baselines. Inference speed reported at 1024×576 resolution for Ours and Ke et al . [ 17 ] , and at 576×320 resolution for VHTT.
Evaluation
Baselines
SDEdit (SD 3)
InstructPix2Pix
Wan-Video V2V
Human Eval. (%)
84.28%±10.92%
90.10%±14.54%
90.11%±14.13%
VLM Eval. (%)
79.18%
75.41%
88.57%
Table 4: User Study. Both human participants and VLM evaluators were asked to compare our results against each individual baseline and select the better one. We report the percentage of samples where our method is preferred over the baselines. A preference rate above 50% indicates that our method is preferred.
Figure 5: Ablation on Loss Design. Removing perceptual supervision leads to oversmoothed outputs, while using a conventional LPIPS loss produces high-frequency artifacts. Our multi-scale formulation mitigates these artifacts and yields perceptually better results.
Model Variant
Temporal Consistency Score
In-domain Test
Out-of-domain Test
Our Model
0.9827
0.9670
Our Model w/o Temporal Loss
0.9806
0.9618
Our Model w/o Temporal Modules
0.9714
0.9502
Table 5: Ablation on Temporal Components. Adding temporal loss and temporal modules effectively improves temporal consistency.
Metric
Full
w/o ISP
w/o
w/o Shadow
w/o Asset
w/o Artifacts
Model
Modif.
Relight.
Simul.
Re-ins.
Correc.
FID ↓
101.27
104.63
102.15
104.28
103.42
105.29
FVD ↓
453.17
465.7
457.92
462.31
459.34
476.82
Table 6: Quantitative ablation on curated data sources. Model trained on all data sources provides the best performance.
Figure 6: Ablation on Curated Data Sources. Removing any curated data source degrades performance: without artifact-correction data the model fails to fix reconstruction errors; without shadow data it cannot synthesize plausible shadows; and without appearance data it produces color-tone inconsistencies. Each data source provides complementary and essential supervision.
Figure 7: Visualization of Curated Training Dataset. We show representative paired samples from our curated training data: Relighting, ISP Modification, Asset Re-insertion, Artifacts Correction, and PBR Simulation.
Figure 8: User Study Interface. We show our study instructions and interface. Evaluators are shown the input image and two predictions (ours and a baseline) and asked to select the more realistic result, with prediction order randomized to avoid bias.
Figure 9: Additional Comparison with Image and Video Editing Baselines on Out-of-Domain Testing Data (Part 1 of 2). Our method harmonizes color tone and synthesizes realistic lighting and shadows, while editing baselines often fail to produce physically plausible shadowing. Although both can reduce reconstruction artifacts, baselines tend to hallucinate inconsistent content and over-edit well-reconstructed regions, whereas our method preserves scene geometry and input structure. Moreover, image-editing baselines introduce frame-to-frame jitter, whereas our model maintains strong temporal coherence.
Figure 10: Additional Comparison with Image and Video Editing Baselines on Out-of-Domain Testing Data (Part 2 of 2). Our method harmonizes color tone and synthesizes realistic lighting and shadows, while editing baselines often fail to produce physically plausible shadowing. Although both can reduce reconstruction artifacts, baselines tend to hallucinate inconsistent content and over-edit well-reconstructed regions, whereas our method preserves scene geometry and input structure. Moreover, image-editing baselines introduce frame-to-frame jitter, whereas our model maintains strong temporal coherence.
Figure 11: Additional Comparison with Video Harmonization Baselines on ISP modification held out test set.
Figure 12: Additional Comparison with Video Harmonization Baselines on ISP modification held out test set.
Figure 13: Comparison with Ground Truth on Holdout Datasets. Our model’s predictions closely match the ground-truth real-world captures, producing faithful, physically plausible results suitable for online simulation systems.
Urban scene reconstruction from real-world observations has emerged as a powerful tool for self-driving development and testing. While current neural rendering approaches achieve high-fidelity rendering along the recorded trajectories, their quality degrades significantly under large viewpoint shifts, limiting the applicability for closed-loop simulation. Recent works have shown promising results in using diffusion models to enhance quality at these challenging viewpoints and distill improvements back into 3D representations. However, they often require costly per-scene optimization, and the distilled representations remain fragile and fail to generalize beyond limited synthesized views. To address these limitations, we propose GenRe, a novel diffusion-guided generalizable enhancer for urban scene reconstruction. GenRe takes as input any pretrained 3D Gaussian representation and fixes the deficiencies within a few minutes. By learning to distill generative priors across diverse scenes, GenRe produces robust and high-fidelity representation efficiently that generalizes reliably to challenging unseen viewpoints (e.g., lane change). Experiments show that GenRe outperforms existing methods in both quality and efficiency and benefits various downstream tasks, enabling robust and scalable sensor simulation for autonomous driving.
Henry Che, Jingkang Wang, Yun Chen +3
1Waabi · University of Illinois Urbana-Champaign · University of Toronto
Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .
Yifan Wang, Francesco Pittaluga, Zaid Tasneem +3
NEC Labs America · Stony Brook University · UC San Diego
Autoregressive video diffusion is a promising render-time fixer for 3D Gaussian Splatting (3DGS) in autonomous-driving simulation, but deployment demands high visual quality and temporal consistency at low latency. This is especially hard for one-step causal generation, where each imperfect prediction immediately becomes context for subsequent frames. Existing approaches stabilize rollouts through staged training with multiple modules and rollout-aware regularization, yet one-step quality still falls short of what deployment requires. We introduce OneFixer, a one-step autoregressive video-diffusion fixer trained in a single task-specific adaptation stage. Our key idea is a deployment-matched shared rollout: the model's own one-step predictions serve as the causal context for flow matching, exposing training to deployment-time errors, while the same rollout receives direct pixel-space perceptual supervision to preserve fine detail. Because the predictions optimized for current-frame quality are exactly those reused as future context, fidelity and autoregressive robustness are learned jointly, without bidirectional-to-causal conversion or teacher-student distillation. OneFixer further exploits cues that driving simulation readily provides, lane geometry and dynamic-agent states, to improve geometric fidelity. On Waymo and proprietary driving scenes with 900-frame rollouts, OneFixer achieves the lowest FVD, LPIPS, and DISTS among all baselines at one step, with temporal consistency matching or exceeding multi-stage DMD pipelines. Under identical backbone and conditioning, it matches a multi-stage DMD-with-Self-Forcing pipeline in under half the GPU-hours and keeps improving beyond its plateau. In closed-loop simulation with a driving policy, OneFixer reduces the collision rate by a third relative to raw 3DGS rendering. Project page: https://onefixer-web.vercel.app/