Egocentric video is crucial for next-generation 4D scene reconstruction, with applications in AR/VR and embodied AI. However, reconstructing dynamic first-person scenes is challenging due to complex ego-motion, occlusions, and hand-object interactions. Existing decomposition methods are ill-suited, assuming fixed viewpoints or merging dynamics into a single foreground. To address these limitations, we introduce DP-DeGauss, a dynamic probabilistic Gaussian decomposition framework for egocentric 4D reconstruction. Our method initializes a unified 3D Gaussian set from COLMAP priors, augments each with a learnable category probability, and dynamically routes them into specialized deformation branches for background, hands, or object modeling. We employ category-specific masks for better disentanglement and introduce brightness and motion-flow control to improve static rendering and dynamic reconstruction. Extensive experiments show that DP-DeGauss outperforms baselines by +1.70dB in PSNR on average with SSIM and LPIPS gains. More importantly, our framework achieves the first and state-of-the-art disentanglement of background, hand, and object components, enabling explicit, fine-grained separation, paving the way for more intuitive ego scene understanding and editing.
Figures & tables
Figure 1: Fine-grained decomposition of egocentric scenes with DP-DeGauss. We achieve accurate and clean separation of background, hands, and objects, overcoming prior methods’ limitations of low-detail reconstruction, misclassification, and coarse foreground–background separation without hand–object distinction.
Figure 2: Overview of our proposed DP-DeGauss . A unified Gaussian set initialized from COLMAP is augmented with a learnable brightness attribute and dynamic category probabilities, which route points to category-specific deformation branches via a two-stage soft-to-hard gating process. Category-level controls jointly drive accurate reconstruction and fine-grained decomposition.
Figure 3: Qualitative comparison of full-scene reconstruction. Our approach yields sharper geometry, reduced motion blur, and fewer holes in both static background and dynamic object/hand regions.
Figure 4: Visual comparison of scene decomposition into background, object, and hand components. Our method achieves clean, fine-grained decomposition with accurate boundaries and fewer artifacts.
Dataset
Metric
4DGaussians [ 14 ]
MotionGS [ 22 ]
NeuralDiff [ 12 ]
DeGauss [ 13 ]
DP-DeGauss (Ours)
HOI4D
PSNR ↑
33.42
26.09
30.45
31.52
33.69
SSIM ↑
0.950
0.901
0.904
0.951
0.952
LPIPS ↓
0.047
0.156
0.114
0.042
0.043
EPIC- FIELD
PSNR ↑
33.69
27.83
33.87
31.86
34.60
SSIM ↑
0.936
0.871
0.934
0.929
0.941
LPIPS ↓
0.045
0.152
0.043
0.066
0.055
Table 1: Quantitative results of full reconstruction.
Figure 5: Visual comparison with EgoGaussian [ 20 ] on full-scene and object–background-separated reconstruction.
Figure 6: Ablation studies on brightness control, motion flow control and zero gradients on mask.
Dynamic 3D hand reconstruction from egocentric videos is essential for next-generation computing platforms such as AR/VR and AI glasses. Despite its importance, most prior works focus either on multi-view 3D hand reconstruction or on 4D human body reconstruction. Egocentric 4D hand reconstruction remains difficult due to rapid hand and camera motion, hand-object and inter-hand interactions, and inherent ambiguity from single-view observations. To address these challenges, we introduce Hand-4DGS, a feed-forward framework for dynamic 4D hand reconstruction from egocentric videos. Our approach incorporates a mesh-guided representation for structural priors and temporal convolutions to model dynamic motion. We evaluate our framework on H2O and ARCTIC, two egocentric hand-object interaction datasets, and show improvements over baselines. Our model can efficiently adapt to unseen videos from datasets that are not included in training. In addition, image supervision through differentiable Gaussian rasterization provides an additional signal for appearance optimization and pose adjustment during training and test-time optimization, without ground-truth 3D hand pose annotations.
Jeongmin Bae, Seoha Kim, Marc Pollefeys +3
Yonsei University · Electronics and Telecommunications Research Institute · ETH Zurich +2
Reconstructing dynamic 4D scenes from monocular videos is a fundamental yet challenging task. While recent 3D foundation models provide strong geometric priors, their performance significantly degrades in dynamic environments. This degradation stems from a fundamental tension: the inherent coupling of camera ego-motion and object motion within global attention mechanisms. In this paper, we propose a novel, training-free progressive decoupling framework that disentangles dynamics from statics in a principled, coarse-to-fine manner. Our core insight is to resolve the tension by first stabilizing the camera pose, followed by geometric refinement. Specifically, our approach consists of three synergistic components: (1) a Dynamic-Mask-Guided Pose Decoupling module that isolates pose estimation from dynamic interference, yielding a stable motion-free reference frame; (2) a Topological Subspace Surgery mechanism that orthogonally decomposes the depth manifold, safely preserving dynamic objects while injecting refined, mask-aware geometry into static regions; and (3) an Information-Theoretic Confidence-Aware Fusion strategy that formulates depth integration as a heteroscedastic Bayesian inference problem, adaptively blending multi-pass predictions via inverse-variance weighting. Extensive experiments on standard 4D reconstruction benchmarks demonstrate that our method achieves consistent and substantial improvements across principal point-cloud metrics. Notably, our approach shows competitive performance in robust 4D scene reconstruction without requiring fine-tuning, suggesting the potential of mathematically grounded dynamic-static disentanglement.
Ying Zang, Xuanyi Liu, Yidong Han +9
4Huzhou University · 2Peking University · 1KOKONI 3D, Moxin Technology +4
Dynamic scene reconstruction from monocular video remains a fundamental challenge in computer vision. Existing feed-forward methods predict 3D Gaussians pixel-wise for each frame, suffering from duplicated Gaussians and view-dependent biases that hinder effective learning of scene motion. We present C4G, a feed-forward 4D reconstruction framework built upon a compact set of timestamp-conditioned learnable Gaussian query tokens. Each token aggregates corresponding features across the full temporal context and decodes a 3D Gaussian whose position is modulated by the target timestamp, enabling globally coherent motion modeling without per-scene optimization. To capture fine-grained details, we further introduce a video diffusion model-based rendering enhancement module. Since our framework effectively aggregates features into Gaussians, we extend this capability to feature lifting, producing a 4D feature field that supports point tracking and dynamic scene understanding. C4G achieves strong novel-view synthesis performance using significantly fewer Gaussians and without requiring camera poses, while exhibiting stronger motion modeling and robustness to large temporal gaps.