Abstract
Reconstructing dynamic human--object interaction scenes from monocular video is difficult because the human, manipulated object, and background obey different motion models while sharing the same pixels. Existing dynamic radiance-field and Gaussian-splatting methods often entangle these components, causing object motion to leak into the human or static scene, and monocular human reconstruction remains underconstrained in regions that are rarely observed. We present CoGS, a compositional Gaussian-splatting framework for monocular human--object scene reconstruction. CoGS decomposes the video into three coordinated branches: an articulated human initialized from a complete canonical prior, a rigid object field driven by an estimated object trajectory, and a static scene field regularized by weak scene-only planar primitives when available. A six-stage optimization schedule first stabilizes the human and object independently, then fuses them with the scene under full-image supervision, visibility-aware human anchoring, object silhouette and motion constraints, and delayed scene regularization. This design keeps each component responsible for its own geometry and motion while allowing photometric evidence to correct the final composite. Experiments on HOSNeRF and NeuMan show that CoGS improves both human--object interaction reconstruction and in-the-wild human--scene rendering, achieving stronger fidelity and perceptual quality across full-frame and human-focused evaluations. Code will be released upon publication.
Explore similar work
Jul 6, 2026cs.CV
Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering. Recent feed-forward methods can efficiently predict geometry, but they are often limited to non-photorealistic representations such as point clouds and meshes, or they fail to handle non-rigid objects, particularly dynamic humans. To fill this gap, we present GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction), a feed-forward framework for online dynamic human-scene reconstruction. From a monocular human-scene video, our method reconstructs dynamic humans (everyone) and static scenes (everywhere) in a single forward pass (all at once) as 3D Gaussian Splatting (3DGS) primitives (as gaussians), which are geometrically consistent and capable of novel view synthesis. Experiments on monocular human-scene datasets demonstrate that our approach achieves competitive novel view synthesis quality while significantly improving inference efficiency compared to optimization-based methods.
Keito Abe, Kaede Shiohara, Takashi Otonari +1
May 25, 2026cs.CV
Reconstructing dynamic 3D scenes from multi-view videos is a foundational task for robotics, AR/VR, and digital twins. While 3D Gaussian Splatting (3DGS) provides state-of-the art scene representation, its dynamic adaptations (4DGS) face significant challenges. Early MLP-based deformation models excel at interpolating within training frames but fail at temporal extrapolation --- important feature for forecasting in predictive systems. To address this, physically grounded approaches have been proposed to enable robust future forecasting, while, concurrently, semantic 4DGS methods have been developed for scene analysis and language-driven prompting. However, there is currently no framework that unites both physical consistency and semantic understanding. In this paper, we propose the first, to the best of our knowledge, unified 4DGS approach that is both physically grounded and semantically aware. Furthermore, to optimize computational efficiency, we introduce a novel rigid-body constraint mechanism. Instead of predicting the physical dynamics for every Gaussian particle, our model predicts the motion of a single anchor Gaussian per semantic object, updating the remaining Gaussians via relative transformations. This architectural design significantly accelerates novel view synthesis compared to an unconstrained baseline, while enabling object-accurate extrapolation and intuitive semantic interaction. Our project page: https://be2rlab.github.io/r5dgs/.
Denis Gridusov, Maxim Popov, Sergey Kolyubin
Jun 9, 2026cs.CV
Reconstructing dynamic and interactive 3D scenes from real-world observations remains a fundamental challenge in computer vision and robotics. While recent advances in 3D Gaussian Splatting have enabled high-fidelity static reconstruction, extending it to interactive environments with articulated robots and manipulable objects remains difficult due to complex contact interactions and abrupt pose changes. To address these challenges, we introduce ManiSplat, a unified framework that reconstructs controllable and decoupled Gaussian digital twins directly from monocular ego-view robotic videos. Our method introduces a Graph-Structured Disentangled Representation that separates the robot, objects, and background into independently optimizable Gaussian subfields organized within a scene graph. To ensure stability, we propose a Task-Oriented Spatio-Temporal Alignment module that leverages the inherent logic of manipulation tasks-alternating between Motion and Skill phases-to construct accurate pseudo-ground-truth trajectories. Finally, a joint photometric-geometric optimization ensures the reconstructed scenes are temporally coherent, physically consistent, and simulation-ready. Extensive experiments demonstrate that our approach reconstructs interaction-driven dynamic scenes with high fidelity and controllability, effectively supporting downstream robotic tasks and policy learning.
Wenhao Hu, Haonan Zhou, Liu Liu +5