SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Rendering

Gen Li; Shu Han; Yun Xi Qiao; Hua Chen; Xuyang Dai; Bohan Li; Hao Zhao; Chaojian Li

cs.CVAug 18, 2026

SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Rendering

Authors: Gen Li, Shu Han, Yun Xi Qiao, Hua Chen, Xuyang Dai, Bohan Li, Hao Zhao, Chaojian Li

Organizations: Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · Zhejiang University, Hangzhou, China. · University of Wisconsin–Madison, Madison, WI, USA. · Tsinghua University, Beijing, China. · Great Wall Motor Company Limited, Baoding, China. · Shanghai Jiao Tong University, Shanghai, China. · The Hong Kong University of Science and Technology, Hong Kong SAR, China.

Abstract

Driving scene reconstruction and rendering, especially with 3D Gaussian Splatting, has become an important component of autonomous driving simulation. However, rendered views often degrade under extrapolated ego trajectories and scene edits, producing blurry structures, temporal flicker, and foreground-background misalignment. Existing refinement methods are commonly designed for a specific setting, such as image-level novel-view repair or object-editing correction. In this paper, we introduce SPVC, a structured and panoptic video fixing framework for cross-dataset driving scene rendering. The name summarizes four design principles. (1) Structured fixing denotes the use of explicit spatial conditions, including camera pose, 3D bounding boxes, and HD maps, to guide the repair process and reduce uncontrolled hallucination. (2) Panoptic fixing refers to correcting both background rendering artifacts, such as distorted roads, buildings, and lanes, and foreground vehicle artifacts introduced by scene editing, such as inconsistent object appearance. (3) Video fixing means that the model operates on driving sequences rather than isolated frames, allowing temporal cues to be used during artifact correction. (4) Cross-dataset fixing means that a single shared network is trained and applied across multiple driving datasets, reducing the need for dataset-specific or scene-specific fixers. Concretely, we construct paired degraded-clean training data by simulating under-constrained 3DGS rendering and foreground vehicle insertion artifacts, and train a two-stage controllable video diffusion model that first addresses video-level appearance and then refines scene layout with structured controls.

Figures & tables

Fig. 1: Our Method improves neural-rendered driving videos by removing 3DGS artifacts, correcting novel-trajectory distortions

Fig. 2: Comparison with different methods.

Fig. 3: Overview of SPVC , a unified framework for structured, panoptic, and temporally consistent driving scene fixing across datasets. SPVC constructs paired degraded–clean videos for novel-trajectory and 3D asset insertion artifacts, and progressively integrates video priors with camera poses, HD maps, and 3D bounding boxes through a two-stage diffusion model.

Fig. 4: Qualitative results under 2 m and 4 m lane shifts on Waymo [ 107 ] .

Method	Lane Shift @ 3m					Lane Shift @ 4m
	Visual Quality			View Consistency		Visual Quality			View Consistency
	FID $\downarrow$	IQ $\uparrow$	CLIP-F $\uparrow$	FVD $\downarrow$	CLIP-V $\uparrow$	FID $\downarrow$	IQ $\uparrow$	CLIP-F $\uparrow$	FVD $\downarrow$	CLIP-V $\uparrow$
PVG [ 108 ]	140.6	41.08	0.7563	1402.0	0.9783	156.5	39.72	0.7429	1570.7	0.9774
StreetGaussian [ 83 ]	86.9	52.81	0.8163	915.5	0.9745	103.5	51.32	0.7968	1208.9	0.9733
OmniRe [ 82 ]	80.6	52.89	0.8154	881.4	0.9759	96.9	51.12	0.8007	1122.5	0.9742
FreeVS [ 60 ]	74.7	53.88	0.8050	1004.8	0.9609	79.9	53.34	0.7982	1075.9	0.9576

TABLE I: Quantitative results on the Waymo dataset under 3m and 4m lane shifts.

Fig. 5: Qualitative results under 2 m and 4 m lane shifts on nuScenes.

Method	Visual Quality			View Consistency
Method	FID $\downarrow$	IQ $\uparrow$	CLIP-F $\uparrow$	FVD $\downarrow$	CLIP-V $\uparrow$
PVG [ 108 ]	154.7	35.78	0.7645	1368.2	0.9736
StreetGaussian [ 83 ]	132.5	42.08	0.7766	1580.7	0.9686
OmniRe [ 82 ]	126.4	42.36	0.7935	1567.1	0.9697
Difix3D+ [ 58 ]	69.0	58.53	0.8375	1272.0	0.9578
SPVC	45.8	65.70	0.8419	719.3	0.9795

TABLE II: Quantitative results on nuScenes at 4m lane shift.

Method	Lane 2m	Lane 3m	Vert. 1m
UniSim [ 76 ]	74.7	97.5	–
NeuRAD [ 112 ]	72.3	93.9	76.3
StreetGaussian [ 83 ]	66.3	80.7	78.4
ReconDreamer [ 62 ]	65.4	74.9	67.7
ReconDreamer++ [ 63 ]	61.9	71.7	62.4
SPVC	35.1	45.0	42.1

TABLE III: Quantitative results on PandaSet.

Fig. 6: Zero-shot evaluation results on the EUVS dataset.

Method	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
3DGS	0.6436	16.9629	0.4301
Ours	0.6870	19.6655	0.3027

TABLE IV: Zero-shot evaluation results on the EUVS dataset.

Fig. 7: SPVC shows superior temporal consistency over single-frame refinement.

Fig. 8: Qualitative results under 3D assets insertion.

Method	Visual Quality				View Consistency
Method	FID-A $\downarrow$	FID $\downarrow$	CLIP-F $\uparrow$	IQ $\uparrow$	FVD $\downarrow$
Naive Insertion [ 82 ]	106.4	154.3	0.7766	0.53	1366.1
Wan2.2 [ 105 ]	115.6	164.0	0.7626	0.59	2134.7
LTX-Video [ 113 ]	141.9	145.5	0.7782	0.50	1690.9
Difix3D+ [ 58 ]	94.80	133.1	0.8171	0.61	1343.7
SPVC	87.9	124.9	0.8322	0.65	1297.2

TABLE V: Quantitative results under 3D asset insertion settings.

Method	UniAD		VAD
Method	Collision Rate $\downarrow$	NNS $\uparrow$ [ 114 ]	Collision Rate $\downarrow$	NNS $\uparrow$ [ 114 ]
Original	60%	2.193	50%	2.659
Finetuned	50%	2.757	30%	3.707

TABLE VI: Closed-loop downstream evaluation results.

Fig. 9: Qualitative results of the ablation study on the two-stage conditioning strategy.

Method	IQ $\uparrow$	CLIP-F $\uparrow$	FID $\downarrow$	FVD $\downarrow$
Conditioning Strategy
w/o Stage I & II	59.87	0.8071	60.1	845.0
w/o Stage I	59.74	0.8123	60.7	820.4
w/o Stage II	63.10	0.8462	46.7	571.4
w/o Camera Pose	63.65	0.8302	52.5	604.3
Training Data

TABLE VII: Ablation study of the two-stage conditioning strategy and artifact-aware data generation.

Fig. 10: Ablation on camera pose conditioning. Camera pose guidance improves geometric correction under novel viewpoints.

Explore similar work

RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos
Jun 15, 2026Zhenhua Wu, Yun Pang, Mingkun Chang +43D Gaussian Splatting Sim-to-Real Transfer
1Sun Yat-sen University · 2Shanghai Innovation Institute · 3Shenzhen Loop Area Institute
OneFixer: High-Quality and Consistent One-Step Autoregressive 3DGS Refinement for Driving Scenes
Sep 26, 2026Boseong Jeon, Junhyeop Lee, Juhan Cha +1One-Step Generative Modeling Temporal Consistency in Video Generation
142dot · Hyundai Motor Company
ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenes
May 10, 2026Rui Song, Tianhui Cai, Markus Gross +53D Gaussian Splatting Diffusion Guidance
University of California, Los Angeles · Technical University of Munich · University of Cambridge

Jun 15, 2026J/K move · Enter open · S save

RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos

Zhenhua Wu, Yun Pang, Mingkun Chang, Yuwei Ning+3

1Sun Yat-sen University · 2Shanghai Innovation Institute · 3Shenzhen Loop Area Institute

3D Gaussian Splatting Sim-to-Real Transfer Autonomous Driving Simulation Video Restoration Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Long-tail hazardous scenarios are essential for safety-oriented autonomous driving, yet they are difficult to collect and reproduce at scale. Editable 3D Gaussian Splatting (3DGS) simulation offers a promising alternative by reconstructing real driving scenes and supporting controllable scene editing. However, edited 3DGS-rendered videos still suffer from a significant Sim-to-Real gap, including rendering artifacts, degraded foreground assets, inconsistent illumination, and temporal flickering. Existing restoration and video generation methods are insufficient for this task, as they often fail to jointly repair 3DGS-specific artifacts, improve visual realism, and ensure temporal consistency. To fill this gap, we propose RealityBridge, a structure-preserving and asset-aware Sim-to-Real framework for edited 3DGS driving videos. RealityBridge uses multimodal controls, including rendered videos, foreground masks, edge maps, and semantic masks, together with a lightweight GateNet for adaptive condition allocation across backbone layers. We further construct targeted training data and introduce autoregressive long-video training with reward-guided post-training to improve restoration quality, temporal stability, and hallucination suppression. Extensive experiments on internal and public driving datasets show that RealityBridge outperforms existing methods in artifact removal, illumination harmonization, and long-sequence temporal consistency.

Abstract

Explore similar work

RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos

OneFixer: High-Quality and Consistent One-Step Autoregressive 3DGS Refinement for Driving Scenes

ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenes