DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer

Yuxuan Zhang; Katarína Tóthová; Zian Wang; Kangxue Yin; Haithem Turki; Riccardo de Lutio; Yen-Yu Chang; Or Litany

cs.CVFeb 27, 2026

DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer

Authors: Yuxuan Zhang, Katarína Tóthová, Zian Wang, Kangxue Yin, Haithem Turki, Riccardo de Lutio, Yen-Yu Chang, Or Litany, +2 more

Organizations: NVIDIA · University of Toronto · Cornell University · Technion

Abstract

Simulation is essential to the development and evaluation of autonomous robots such as self-driving vehicles. Neural reconstruction is emerging as a promising solution as it enables simulating a wide variety of scenarios from real-world data alone in an automated and scalable way. However, while methods such as NeRF and 3D Gaussian Splatting can produce visually compelling results, they often exhibit artifacts particularly when rendering novel views, and fail to realistically integrate inserted dynamic objects, especially when they were captured from different scenes. To overcome these limitations, we introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings from such imperfect scenes into temporally consistent outputs while improving their realism. At its core is a single-step temporally-conditioned enhancer that is converted from a pretrained multi-step image diffusion model, capable of running in online simulators on a single GPU. The key to training it effectively is a custom data curation pipeline that constructs synthetic-real pairs emphasizing appearance harmonization, artifact correction, and lighting realism. The result is a scalable system that significantly elevates simulation fidelity in both research and production environments.

Figures & tables

Figure 2: Overview of the data curation pipeline ( top ) and model architecture ( bottom ) of DiffusionHarmonizer. We use a single-step temporally conditioned enhancement model, that is converted from a pretrained multi-step image diffusion model. To train it effectively, we develop a data curation pipeline to construct synthetic–real pairs emphasizing harmonization, artifact correction and lighting realism.

Figure 3: Comparison with Image and Video Editing Baselines on Out-of-Domain Testing Data. Our method harmonizes color tone and synthesizes realistic lighting and shadows, while editing baselines often fail to produce physically plausible shadowing. Although both can reduce reconstruction artifacts, baselines tend to hallucinate inconsistent content and over-edit well-reconstructed regions, whereas our method preserves scene geometry and input structure. Moreover, image-editing baselines introduce frame-to-frame jitter, whereas our model maintains strong temporal coherence.

Figure 4: Comparison with Harmonization Baselines Methods. While both our method and harmonization baselines adjust foreground appearance, the baselines fail to synthesize realistic shadows, resulting in less coherent composites.

		Novel Trajectory Simulation (In-domain)				Object Insertion Simulation (Out-of-domain)
Method	Inference Speed $\downarrow$	FID $\downarrow$	FVD $\downarrow$	DINO Struct. $\uparrow$	Temporal Cons. $\uparrow$	FID $\downarrow$	FVD $\downarrow$	DINO Struct. $\uparrow$	Temporal Cons. $\uparrow$
SDEdit (SD 3)	398ms	129.92	753.39	0.7075	0.9661	106.17	506.91	0.6345	0.9524
InstructPix2Pix	555ms	153.94	680.92	0.6339	0.9203	128.72	573.20	0.5793	0.9085
Wan-Video V2V	2827ms	134.98	506.86	0.8289	0.9828	104.42	474.96	0.8226	0.9675
Our Model	212ms	120.23	470.11	0.9215	0.9827	101.27	453.17	0. 9096	0.9670

Table 1: Quantitative Comparison on Novel Trajectory Simulation (In-domain Test) and Object Insertion Simulation (Out-of-domain Test) Datasets. Our method outperforms all editing baselines in perceptual quality (lower FID/FVD) and preserves scene structure more faithfully (lower DINO-Struct-Dist). It also achieves strong temporal consistency (measured by VBench++ temporal flickering score), surpassing image-editing methods and matching video diffusion models, with only a marginal gap to WAN V2V.

	Relighting Data (Images)			PBR Shadow Data (Videos)			ISP Modification Data (Videos)
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SDEdit (SD 3)	15.17	0.9803	0.0098	15.65	0.9891	0.0095	17.46	0.9850	0.0112
InstructPix2Pix	15.35	0.9812	0.0112	16.46	0.9905	0.0083	16.71	0.9856	0.0105
Wan-Video V2V	N/A	N/A	N/A	14.21	0.9867	0.098	15.81	0.9814	0.1335
Our Model	23.93	0.9938	0.0042	26.31	0.9966	0.0028	28.10	0.9974	0.0020

Table 2: Quantitative Results on Relighting , PBR Shadow , and ISP Modification Holdout Sets. Our method achieves substantially better PSNR, SSIM, and LPIPS, closely matching real-world references.

Method	ISP Modification Data
Method	Speed	PSNR	SSIM	LPIPS	FID	FVD	Temporal Cons.
VHTT	63ms	20.96	0.9921	0.0049	46.23	168.83	0.9838
Ke et al . [ 17 ]	10ms	25.98	0.9956	0.0037	61.51	170.16	0.9872
Ours	212ms	28.58	0.9971	0.0021	42.03	158.27	0.9805

Table 3: Quantitative Comparison with Harmonization Baselines. Evaluated on the harmonization subset (ISP Modification), our method outperforms all baselines. Inference speed reported at

1024\times 576

resolution for Ours and Ke et al . [ 17 ] , and at

576\times 320

resolution for VHTT.

Evaluation	Baselines
Evaluation	SDEdit (SD 3)	InstructPix2Pix	Wan-Video V2V
Human Eval. (%)	$84.28\%\pm 10.92\%$	$90.10\%\pm 14.54\%$	$90.11\%\pm 14.13\%$
VLM Eval. (%)	79.18%	75.41%	88.57%

Table 4: User Study. Both human participants and VLM evaluators were asked to compare our results against each individual baseline and select the better one. We report the percentage of samples where our method is preferred over the baselines. A preference rate above 50% indicates that our method is preferred.

Figure 5: Ablation on Loss Design. Removing perceptual supervision leads to oversmoothed outputs, while using a conventional LPIPS loss produces high-frequency artifacts. Our multi-scale formulation mitigates these artifacts and yields perceptually better results.

Model Variant	Temporal Consistency Score
Model Variant	In-domain Test	Out-of-domain Test
Our Model	0.9827	0.9670
Our Model w/o Temporal Loss	0.9806	0.9618
Our Model w/o Temporal Modules	0.9714	0.9502

Table 5: Ablation on Temporal Components. Adding temporal loss and temporal modules effectively improves temporal consistency.

Metric	Full	w/o ISP	w/o	w/o Shadow	w/o Asset	w/o Artifacts
	Model	Modif.	Relight.	Simul.	Re-ins.	Correc.
FID $\downarrow$	101.27	104.63	102.15	104.28	103.42	105.29
FVD $\downarrow$	453.17	465.7	457.92	462.31	459.34	476.82

Table 6: Quantitative ablation on curated data sources. Model trained on all data sources provides the best performance.

Figure 6: Ablation on Curated Data Sources. Removing any curated data source degrades performance: without artifact-correction data the model fails to fix reconstruction errors; without shadow data it cannot synthesize plausible shadows; and without appearance data it produces color-tone inconsistencies. Each data source provides complementary and essential supervision.

Figure 7: Visualization of Curated Training Dataset. We show representative paired samples from our curated training data: Relighting, ISP Modification, Asset Re-insertion, Artifacts Correction, and PBR Simulation.

Figure 8: User Study Interface. We show our study instructions and interface. Evaluators are shown the input image and two predictions (ours and a baseline) and asked to select the more realistic result, with prediction order randomized to avoid bias.

Figure 9: Additional Comparison with Image and Video Editing Baselines on Out-of-Domain Testing Data (Part 1 of 2). Our method harmonizes color tone and synthesizes realistic lighting and shadows, while editing baselines often fail to produce physically plausible shadowing. Although both can reduce reconstruction artifacts, baselines tend to hallucinate inconsistent content and over-edit well-reconstructed regions, whereas our method preserves scene geometry and input structure. Moreover, image-editing baselines introduce frame-to-frame jitter, whereas our model maintains strong temporal coherence.

Figure 10: Additional Comparison with Image and Video Editing Baselines on Out-of-Domain Testing Data (Part 2 of 2). Our method harmonizes color tone and synthesizes realistic lighting and shadows, while editing baselines often fail to produce physically plausible shadowing. Although both can reduce reconstruction artifacts, baselines tend to hallucinate inconsistent content and over-edit well-reconstructed regions, whereas our method preserves scene geometry and input structure. Moreover, image-editing baselines introduce frame-to-frame jitter, whereas our model maintains strong temporal coherence.

Figure 11: Additional Comparison with Video Harmonization Baselines on ISP modification held out test set.

Figure 12: Additional Comparison with Video Harmonization Baselines on ISP modification held out test set.

Figure 13: Comparison with Ground Truth on Holdout Datasets. Our model’s predictions closely match the ground-truth real-world captures, producing faithful, physically plausible results suitable for online simulation systems.

Explore similar work

Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction
May 21, 2026Henry Che, Jingkang Wang, Yun Chen +33D Gaussian Splatting Diffusion Model Guidance
1Waabi · University of Illinois Urbana-Champaign · University of Toronto
HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles
Feb 24, 2026Yifan Wang, Francesco Pittaluga, Zaid Tasneem +3Autonomous Driving Scenario Generation 3D Scene Editing
NEC Labs America · Stony Brook University · UC San Diego
OneFixer: High-Quality and Consistent One-Step Autoregressive 3DGS Refinement for Driving Scenes
Sep 26, 2026Boseong Jeon, Junhyeop Lee, Juhan Cha +1One-Step Generative Modeling Temporal Consistency in Video Generation
142dot · Hyundai Motor Company

May 21, 2026J/K move · Enter open · S save

Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction

Henry Che, Jingkang Wang, Yun Chen, Ze Yang+2

1Waabi · University of Illinois Urbana-Champaign · University of Toronto

3D Gaussian Splatting Diffusion Model Guidance 3D Reconstruction Feed-Forward 3D Reconstruction Autonomous Driving Simulation

Open paper AskPDF arXiv

Urban scene reconstruction from real-world observations has emerged as a powerful tool for self-driving development and testing. While current neural rendering approaches achieve high-fidelity rendering along the recorded trajectories, their quality degrades significantly under large viewpoint shifts, limiting the applicability for closed-loop simulation. Recent works have shown promising results in using diffusion models to enhance quality at these challenging viewpoints and distill improvements back into 3D representations. However, they often require costly per-scene optimization, and the distilled representations remain fragile and fail to generalize beyond limited synthesized views. To address these limitations, we propose GenRe, a novel diffusion-guided generalizable enhancer for urban scene reconstruction. GenRe takes as input any pretrained 3D Gaussian representation and fixes the deficiencies within a few minutes. By learning to distill generative priors across diverse scenes, GenRe produces robust and high-fidelity representation efficiently that generalizes reliably to challenging unseen viewpoints (e.g., lane change). Experiments show that GenRe outperforms existing methods in both quality and efficiency and benefits various downstream tasks, enabling robust and scalable sensor simulation for autonomous driving.

Abstract

Explore similar work

Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

OneFixer: High-Quality and Consistent One-Step Autoregressive 3DGS Refinement for Driving Scenes