DUET4D: Self-Corrective Coupled Guidance for Sparse-View 4D Reconstruction

Beizhen Zhao; Nanjie Yao; Boyi Fu; Qingsong Yan; Haofan Ren; Kaiyong Zhao; Hao Wang

cs.CVOct 26, 2026

DUET4D: Self-Corrective Coupled Guidance for Sparse-View 4D Reconstruction

Authors: Beizhen Zhao, Nanjie Yao, Boyi Fu, Qingsong Yan, Haofan Ren, Kaiyong Zhao, Hao Wang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · XGRIDS

Abstract

Sparse-view 4D reconstruction is highly under-constrained when only a few cameras are available. Generative priors can recover missing appearances, but video models are computationally impractical for iterative reconstruction, whereas single-view models produce temporally inconsistent pseudo-observations. Moreover, directly absorbing uncertain generative outputs into Gaussian optimization often creates redundant primitives and floating artifacts. To address these challenges, we propose DUET4D, a self-corrective coupled guidance framework for sparse-view 4D Gaussian splatting. The core of our approach is a self-corrective framework where the 4D representation guides generative prior for spatio-temporal consistent supervisions. We first construct sequential alignments from current Gaussian evidence to steer the latent restoration. Then we use Sequential Fidelity Resampling to lift generative supervisions into Gaussian space to selectively distinguish repeatedly supported primitives. The corrected representation then provides more reliable Gaussian evidence for the next restoration cycle. Extensive experiments demonstrate that DUET4D significantly improves rendering quality and rendering efficiency.

Figures & tables

Figure 2: Overview of DUET4D. First, we construct sequential anchor alignment through Gaussian evidence to generate temporal anchor frames and confidence map. Then we design a training-free anchor-driven latent steering mechanism within the latent space of SDM for temporally consistent generative outputs. The frames and confidence map will be used to locate Gaussians with weak spatial and temporal consistency for optimization. The reserved Gaussians are controlled by a spatial grid and Fourier spline priors for motion smoothness and regularization.

Method	Venue	Nvidia Dynamic Scenes					Neural 3D Video					Google Immersive
Method	Venue	FPS $\uparrow$	$N_{\text{GS}}\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$	$N_{\text{GS}}\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$	$N_{\text{GS}}\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
4DGS	CVPR 2024	54	0.26 M	16.28	0.325	0.525	49	0.21 M	16.87	0.665	0.380	99	0.84M	14.27	0.517	0.555
4D-Rotor	SIGGRAPH 2024	7	0.35M	16.43	0.352	0.482	5	0.38M	19.48	0.738	0.315	10	0.71M	14.46	0.533	0.490
RealTime4DGS	ICLR 2024	11	0.38M	16.02	0.359	0.545	12	0.46M	17.53	0.652	0.434	21	1.09M	14.33	0.521	0.553
Grid4D	NeurIPS 2024	16	1.15M	16.35	0.334	0.538	80	0.21 M	18.30	0.709	0.391	16	0.46 M	12.47	0.492	0.566
Ex4DGS	NeurIPS 2024	61	2.98M	16.74	0.373	0.523	32	0.57M	19.21	0.730	0.304	49	1.36M	15.36	0.565	0.531

Table 1: Quantitative comparisons on the Nvidia Dynamic Scenes ( Yoon et al. 2020 ) , Neural 3D Video ( Li et al. 2022 ) , and Google Immersive ( Broxton et al. 2020 ) datasets. The best and second-best results are highlighted in bold and underlined, respectively. Our method outperforms all baseline methods across all evaluated metrics. “

\uparrow

” indicates higher is better, while “

\downarrow

” indicates lower is better. Specifically, DUET4D ∗ uses exactly the same preprocessed inputs as SparseCam4D, while all model components and optimization settings remain unchanged.

Figure 3: Qualitative comparisons of state-of-the-art methods. Visual differences are highlighted with red insets for better clarity. Our approach outperforms other models on different scenes, demonstrating advantages in challenging scenarios. Best viewed in color.

Configuration	FPS	$\textbf{N}_{\text{GS}}$	tPSNR $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Baseline	80	0.21M	36.91	18.30	0.709	0.391
+Fidelity Resampling	268	0.04M	36.89	18.54	0.713	0.358
+Control Regularization	271	0.03M	36.88	18.80	0.729	0.329
+Vanilla Diffusion	230	0.06M	36.33	20.04	0.742	0.297
+Anchor Alignment	233	0.06M	36.89	20.08	0.743	0.305
+Latent Steering (Full)	246	0.05M	37.99	20.28	0.748	0.296

Table 2: Ablation study of proposed components on the Neural 3D Video dataset . We progressively evaluate Sequential Fidelity Resampling, Smoothness Control Regularization, Sequential Anchor Alignment, and Anchor-Driven Latent Steering. Vanilla single-view diffusion is included as an intermediate baseline to distinguish the effect of the generative prior from that of the proposed guidance mechanisms.

Figure 4: Visual ablation. Fidelity Resampling removes floating artifacts, while vanilla diffusion restores details but may introduce appearance drift. The full model uses Gaussian-derived anchors and confidence-aware latent steering to recover sharper and more consistent results.

Figure 5: Temporal consistency on static regions. We compare vanilla single-view diffusion with the full method on fixed-view sequences. The left column shows the difference in long-term drift from a reference frame, while the right column shows the difference in short-term flicker between adjacent frames. Red indicates lower error for DUET4D, and blue indicates the opposite.

	Setting	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	tPSNR $\uparrow$
$\rho$	$0.25$	20.21	0.744	0.299	37.82
	$0.50$	20.28	0.748	0.296	37.99
	$1.00$	20.06	0.739	0.325	38.32
$\tau_{m}$	$0.001$	20.18	0.742	0.300	37.79
	$0.002$	20.28	0.748	0.296	37.99
	$0.005$	20.22	0.744	0.298	37.87

Table 3: Hyperparameter sensitivity on the Neural 3D Video dataset. For each group, only the indicated parameter is changed while all other settings remain fixed.

4D Reconstruction

4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.

See all 140 papers

FillGS: Filling Observation Gaps in 4D Gaussian Splatting via Viewpoint-Time Selection and Generative Refinement
Jul 31, 2026Takashi Otonari, Toshihiko Yamasaki3D Gaussian Splatting Active Perception
The University of Tokyo, Japan
4C4D: 4 Camera 4D Gaussian Splatting
Apr 5, 2026Junsheng Zhou, Zhifan Yang, Liang Han +4Sparse-View 4D Reconstruction 4D Reconstruction
School of Software, Tsinghua University · Kuaishou Technology
4DGS-Fixer: Generative Sparse-View 4D Gaussian Splatting with Iterative Refinement Guided by Video Diffusion Priors
Sep 18, 2026Haitao Huang, Shenghao Zhao, Boyuan Tian +73D Gaussian Splatting Sparse-View 4D Reconstruction
Goertek Alpha Labs Shanghai, China · Singapore Institute of Technology Singapore, Singapore · Goertek Alpha Labs Santa Clara, USA +1

Jul 31, 2026J/K move · Enter open · S save

FillGS: Filling Observation Gaps in 4D Gaussian Splatting via Viewpoint-Time Selection and Generative Refinement

Takashi Otonari, Toshihiko Yamasaki

The University of Tokyo, Japan

3D Gaussian Splatting Active Perception 4D Reconstruction Sparse-View 4D Reconstruction Dynamic Scene Reconstruction Active View Selection

Open paper AskPDF arXiv

4D Gaussian Splatting (4DGS) can render dynamic scenes photorealistically. However, with limited viewpoint coverage, some spatiotemporal regions remain sparsely observed, leading to artifacts, particularly in scenes with large motion. Existing approaches leveraging generative models rely on heuristic virtual-viewpoint selection before refining rendered views. As a result, they cannot actively explore such sparsely observed regions. To address this issue, we propose a pipeline that actively selects spatiotemporal virtual viewpoints to improve 4DGS reconstruction. Our method selects virtual viewpoints for generative enhancement based on the rendering sensitivity and motion-aware observation density of 4D Gaussians, prioritizing views that alleviate observation sparsity. In the refined images, we filter out regions that conflict with captured observations or are likely to contain generative artifacts and then fine-tune 4DGS using only the reliable regions. We evaluate our method on multi-view video benchmarks using new train/test splits designed to induce observation gaps. Results show consistent improvements over prior viewpoint selection strategies and fine-tuning methods in both qualitative and quantitative evaluations, while reducing artifacts.

Abstract

4D Reconstruction

Explore similar work

FillGS: Filling Observation Gaps in 4D Gaussian Splatting via Viewpoint-Time Selection and Generative Refinement

4C4D: 4 Camera 4D Gaussian Splatting

4DGS-Fixer: Generative Sparse-View 4D Gaussian Splatting with Iterative Refinement Guided by Video Diffusion Priors