Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting

Weiquan Wang; Jun Xiao; Feifei Shao; Yi Yang; Yueting Zhuang; Long Chen

cs.CVApr 3, 2026

Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting

Authors: Weiquan Wang, Jun Xiao, Feifei Shao, Yi Yang, Yueting Zhuang, Long Chen

Organizations: State Key Lab of CAD&CG, College of Computer Science, Zhejiang University, Hangzhou 310027, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong

Abstract

Reconstructing dynamic scenes with multiple interacting humans and objects from sparse-view inputs is a critical yet challenging task, essential for creating high-fidelity digital twins for robotics and VR/AR. This problem, which we term Multi-Human Multi-Object (MHMO) rendering, presents two significant obstacles: achieving view-consistent representations for individual instances under severe mutual occlusion, and explicitly modeling the complex and combinatorial dependencies that arise from their interactions. To overcome these challenges, we propose MM-GS, a novel hierarchical framework built upon 3D Gaussian Splatting. Our method first employs a Per-Instance Multi-View Fusion module to establish a robust and consistent representation for each instance by aggregating visual information across all available views. Subsequently, a Scene-Level Instance Interaction module operates on a global scene graph to reason about relationships between all participants, refining their attributes to capture subtle interaction effects. Extensive experiments on challenging datasets demonstrate that our method significantly outperforms strong baselines, producing state-of-the-art results with high-fidelity details and plausible inter-instance contacts.

Figures & tables

Fig. 1: Core challenges in Multi-Human Multi-Object (MHMO) rendering. From sparse views, rendering complex interactions involves overcoming two key challenges: ensuring cross-view consistency under severe occlusion (top) and modeling the mutual influence between instances at contact regions (bottom). Our MM-GS is designed to address both.

Fig. 2: Overview of MM-GS pipeline. Our method refines initial 3D Gaussian representations through three main stages. (a) Human-Object Deformation: We initialize the scene by deforming canonical human and object models to their target poses and representing them as collections of 3D Gaussians. (b) Per-Instance Multi-View Fusion: A Cross-View Fusion network refines each instance’s appearance and local geometry by aggregating visual features from all its visible viewpoints, ensuring a view-consistent representation. (c) Scene-Level Instance Interaction: Finally, an Instance Interaction network operates on a global scene graph to model the dependencies between all participants, enabling a final refinement to capture interaction-driven effects.

Method	Livingroom			Fitnessroom			Office
Method	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
NeuralHOIFVV-MM [ 55 ]	21.33	0.8704	0.1884	22.66	0.9148	0.1575	21.65	0.8876	0.1702
GTU-MM [ 21 ]	20.82	0.8527	0.2045	21.91	0.9095	0.1683	21.71	0.8841	0.1734
Ours	22.47	0.8894	0.1722	23.24	0.9224	0.1504	22.89	0.9029	0.1633

TABLE I: Quantitative comparison for novel view synthesis on the HOI-M 3 dataset. We report PSNR(

\uparrow

), SSIM(

\uparrow

), and LPIPS(

\downarrow

) on three representative scenes. We use red and yellow to denote the best and second-best results.

Fig. 3: Qualitative comparison on the HOI-M 3 dataset. We highlight specific regions with colored dashed circles to illustrate the differences. Note that our MM-GS generates significantly sharper details and more plausible contact regions. In contrast, the NeRF-based NeuralHOIFVV-MM tends to produce overly smooth or blurry results, while the 3DGS-based GTU-MM suffers from floating artifacts and geometric inconsistencies.

Method	Bucket			Box
Method	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
Neu.-MM	19.43	0.9347	0.0602	18.78	0.9218	0.0629
GTU-MM	19.07	0.9281	0.0644	18.56	0.9231	0.0637
Ours	20.08	0.9387	0.0527	19.22	0.9302	0.0598

TABLE II: Generalization performance on the CORE4D-Real dataset. Results are averaged over two interaction scenarios. Neu.-MM: NeuralHOIFVV-MM.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$	Training
MM-GS (Full)	22.47	0.8894	0.1722	160+	$\sim$ 40min
w/o View Fusion	20.98	0.8566	0.1927	160+	$\sim$ 32min
w/o Interaction	21.51	0.8621	0.1893	160+	$\sim$ 36min
w/o Both	19.42	0.8140	0.2031	160+	$\sim$ 30min

TABLE III: Ablation study of core components on the HOI-M 3 dataset (Livingroom).

Fig. 4: Qualitative results of our ablation study. Removing both modules (w/o Both) leads to blurry results. Adding the View Fusion module (+ View Fusion) significantly improves sharpness. Further incorporating the Interaction network (+ View Fusion + Interaction) resolves ambiguities at contact regions, resulting in cleaner boundaries.

Dynamic Scene Reconstruction

Dynamic scene reconstruction builds time-varying 3D models of moving objects and environments from video or multi-view images.

See all 288 papers

HOIGS: Human-Object Interaction Gaussian Splatting
Apr 5, 2026Taewoo Kim, Suwoong Yeom, Jaehyun Pyun +6Human-Scene Interaction Reconstruction 3D Reconstruction
Department of Electronics Engineering, Sogang University, South Korea · NAVER Cloud Corp, South Korea · Department of Electrical & Electronics Engineering, Pusan National University, South Korea
CoGS: Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video
Jun 27, 2026Jerrin Bright, John Zelek3D Gaussian Splatting 3D Reconstruction
Vision and Image Processing Lab, University of Waterloo, Canada
Human Interaction-Aware 3D Reconstruction from a Single Image
Apr 7, 2026Gwanghyun Kim, Junghun James Kim, Suh Yoon Jeon +23D Human Reconstruction Single-View 3D Reconstruction
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

Apr 5, 2026J/K move · Enter open · S save

HOIGS: Human-Object Interaction Gaussian Splatting

Taewoo Kim, Suwoong Yeom, Jaehyun Pyun, Geonho Cha+5

Department of Electronics Engineering, Sogang University, South Korea · NAVER Cloud Corp, South Korea · Department of Electrical & Electronics Engineering, Pusan National University, South Korea

Human-Scene Interaction Reconstruction 3D Reconstruction Dynamic Scene Reconstruction 3D Gaussian Splatting Human-Object Interaction 3D Human Reconstruction

Open paper AskPDF arXiv

Reconstructing dynamic scenes with complex human-object interactions is a fundamental challenge in computer vision and graphics. Existing Gaussian Splatting methods either rely on human pose priors while neglecting dynamic objects, or approximate all motions within a single field, limiting their ability to capture interaction-rich dynamics. To address this gap, we propose Human-Object Interaction Gaussian Splatting (HOIGS), which explicitly models interaction-induced deformation between humans and objects through a cross-attention-based HOI module. Distinct deformation baselines are employed to extract features: HexPlane for humans and Cubic Hermite Spline (CHS) for objects. By integrating these heterogeneous features, HOIGS effectively captures interdependent motions and improves deformation estimation in scenarios involving occlusion, contact, and object manipulation. Comprehensive experiments on multiple datasets demonstrate that our method consistently outperforms state-of-the-art human-centric and 4D Gaussian approaches, highlighting the importance of explicitly modeling human-object interactions for high-fidelity reconstruction.

Abstract

Dynamic Scene Reconstruction

Explore similar work

HOIGS: Human-Object Interaction Gaussian Splatting

CoGS: Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video

Human Interaction-Aware 3D Reconstruction from a Single Image