HOIGS: Human-Object Interaction Gaussian Splatting

Taewoo Kim; Suwoong Yeom; Jaehyun Pyun; Geonho Cha; Dongyoon Wee; Joonsik Nam; Yun-Seong Jeong; Kyeongbo Kong

cs.CVApr 5, 2026

HOIGS: Human-Object Interaction Gaussian Splatting

Authors: Taewoo Kim, Suwoong Yeom, Jaehyun Pyun, Geonho Cha, Dongyoon Wee, Joonsik Nam, Yun-Seong Jeong, Kyeongbo Kong,

Organizations: Department of Electronics Engineering, Sogang University, South Korea · NAVER Cloud Corp, South Korea · Department of Electrical & Electronics Engineering, Pusan National University, South Korea

Abstract

Reconstructing dynamic scenes with complex human-object interactions is a fundamental challenge in computer vision and graphics. Existing Gaussian Splatting methods either rely on human pose priors while neglecting dynamic objects, or approximate all motions within a single field, limiting their ability to capture interaction-rich dynamics. To address this gap, we propose Human-Object Interaction Gaussian Splatting (HOIGS), which explicitly models interaction-induced deformation between humans and objects through a cross-attention-based HOI module. Distinct deformation baselines are employed to extract features: HexPlane for humans and Cubic Hermite Spline (CHS) for objects. By integrating these heterogeneous features, HOIGS effectively captures interdependent motions and improves deformation estimation in scenarios involving occlusion, contact, and object manipulation. Comprehensive experiments on multiple datasets demonstrate that our method consistently outperforms state-of-the-art human-centric and 4D Gaussian approaches, highlighting the importance of explicitly modeling human-object interactions for high-fidelity reconstruction.

Figures & tables

Panel of Figure 1 — Figure 1: 4DGS-based and human-centric models fail to render HOI scenes.

Panel of Figure 5 — Figure 5: Qualitative comparison of reconstructed rendered views. We compare our method (HOIGS) against state-of-the-art baselines on both the HOSNeRF [ 26 ] and BEHAVE [ 3 ] datasets. We display the full-frame (top) rendering and a zoom-in (bottom) of the red Region of Interest (ROI).

Methods	Backpack		Tennis		Suitcase		Playground		Dance		Lounge
	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
K-Planes [ 10 ]	19.05	0.557	19.31	0.536	18.64	0.602	17.92	0.635	18.17	0.623	24.21	0.453
D 2 NeRF [ 45 ]	20.52	0.608	23.97	0.540	20.99	0.645	21.23	0.616	19.92	0.647	27.13	0.509
Nerfies [ 32 ]	19.56	0.559	22.12	0.443	19.01	0.555	21.14	0.533	19.37	0.524	25.90	0.342
HyperNeRF [ 33 ]	19.62	0.587	21.26	0.510	19.41	0.607	21.67	0.578	19.30	0.601	27.25	0.332
NeuMan [ 17 ]	21.21	0.478	23.17	0.442	20.84	0.551	21.46	0.551	21.19	0.490	28.40	0.341

Table 1: Per-scene quantitative evaluation on the HOSNeRF [ 26 ] dataset against baselines of our method. We color code each cell as best and second best .

Scene	4DGS [ 44 ]		E - D3DGS [ 2 ]		ExAvatar [ 31 ]		HOIGS (Ours)
	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Backpack_1	21.81	0.076	19.99	0.086	27.86	0.041	31.79	0.031
Plasticcontainer_1	22.92	0.072	20.15	0.086	29.96	0.042	33.10	0.032
Plasticcontainer_2	26.37	0.081	24.75	0.078	30.11	0.038	32.39	0.034
Suitcase_2	26.66	0.071	25.85	0.058	30.86	0.032	34.58	0.028
Backpack_3	24.59	0.085	23.72	0.074	26.47	0.054	30.17	0.044

Table 2: Per-scene quantitative evaluation on the BEHAVE [ 3 ] dataset against baselines.

Panel of Figure 6 — Figure 6: Qualitative comparison of reconstructed rendered view results on the ARCTIC [ 9 ] dataset.

Methods	Capsulemachine		Mixer		Espressomachine		Box
	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
4DGS [ 44 ]	26.15	0.124	23.21	0.166	21.80	0.196	22.22	0.182
E - D3DGS [ 2 ]	25.10	0.089	22.14	0.139	19.50	0.227	20.60	0.153
HOLD [ 8 ]	25.52	0.522	23.35	0.540	23.52	0.547	24.72	0.494
HOIGS (Ours)	27.05	0.069	24.59	0.095	25.29	0.079	23.50	0.124

Table 3: Per-scene quantitative evaluation on the ARCTIC [ 9 ] dataset against baselines of our method.

Panel of Figure 7 — Figure 7: More qualitative comparison between ground truth and our HOIGS on the HOSNeRF [ 26 ] dataset.

Panel of Figure 8 — Figure 8: 4DGS-based and human-centric models fail to render HOI scenes on the BEHAVE [ 3 ] dataset.

Table 9

Metric	HOLD [ 8 ]	HOIGS (Ours)
$CD^{\mathrm{best}}$ [ $m^{2}$ ] $\downarrow$	1.5035	0.1337

Table A: Geometric evaluation on the ARCTIC [ 9 ] dataset. Our method (HOIGS) demonstrates significantly lower distance metrics compared to the baseline, indicating closer, more physically plausible hand-object contact. Results are averaged over all sequences.

Method Combinations	Backpack		Tennis		Suitcase		Playground		Dance		Lounge		Average
Method Combinations	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Samurai [ 48 ] + MetricV2 [ 13 ]	25.78	0.082	27.12	0.108	22.09	0.246	25.23	0.103	24.17	0.098	30.97	0.048	25.89	0.114
Samurai [ 48 ] + Video Depth Anything [ 5 ]	25.85	0.080	27.18	0.106	22.15	0.241	25.28	0.102	24.22	0.096	31.05	0.046	25.96	0.112
Samurai [ 48 ] + DepthCrafter [ 15 ]	25.72	0.088	27.08	0.108	22.06	0.249	25.20	0.109	24.08	0.099	30.93	0.048	25.85	0.117
TrackAnything [ 49 ] + MetricV2 [ 13 ]	25.72	0.086	27.05	0.109	22.03	0.246	25.18	0.106	24.15	0.100	30.93	0.052	25.84	0.116
SAMv2 [ 37 ] + Video Depth Anything [ 5 ]	26.01	0.076	27.38	0.103	22.33	0.241	25.47	0.100	24.42	0.095	31.20	0.041	26.14	0.109

Table B: Sensitivity analysis of HOIGS on the HOSNeRF [ 26 ] dataset using different combinations of segmentation and depth estimation priors. The results demonstrate the robustness of our method, with consistent performance across various modern priors and strong performance even with older baselines (MaskRCNN [ 29 ] ).

Methods	Training Time	Inference Speed (FPS)
4DGS [ 44 ]	40 min	61.04
Ex4DGS [ 23 ]	2 hr 30 min	46.38
D3DGS [ 51 ]	3 hr	37.79
E-D3DGS [ 2 ]	2 hr	54.71
HOIGS (Ours)	5 hr	44.27

Table C: Runtime performance comparison on the HOSNeRF [ 26 ] dataset. We report the approximate training time per scene and the inference speed in Frames Per Second (FPS). Our method maintains real-time performance (

>

30 FPS) despite the added complexity of interaction modeling.

Methods	Backpack		Tennis		Suitcase		Playground		Dance		Lounge		Average
Methods	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
MASt3R Prior	23.51	0.135	25.25	0.121	22.40	0.197	24.65	0.074	23.63	0.115	28.99	0.057	24.59	0.128
Depth Recon Prior	21.63	0.142	25.65	0.122	22.13	0.230	25.24	0.103	24.08	0.123	28.95	0.095	25.36	0.136
Diffusion Prior (Ours)	23.70	0.082	27.13	0.112	22.96	0.235	25.63	0.123	24.17	0.093	29.97	0.043	25.89	0.114

Table D: Quantitative ablation study on Object Priors using the HOSNeRF [ 26 ] dataset. We evaluate the effectiveness of our Diffusion Prior against MASt3R [ 7 ] and Depth Reconstruction priors.

Figure A : Object feature extraction. Extraction of object motion features using the embedded parameters and velocity vectors of each key frame.

Human-Scene Interaction Reconstruction

See all 13 papers

Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction
Nov 18, 2025Hao Tian, Chenyangguang Zhang, Rui Liu +23D Gaussian Splatting 3D Reconstruction
Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu, China · PetroChina (Beijing) Digital Intelligence Research Institute Co., Ltd., Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2
CoGS: Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video
Jun 27, 2026Jerrin Bright, John Zelek3D Gaussian Splatting 3D Reconstruction
Vision and Image Processing Lab, University of Waterloo, Canada
Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting
Apr 3, 2026Weiquan Wang, Jun Xiao, Feifei Shao +3Dynamic Scene Reconstruction 3D Reconstruction
State Key Lab of CAD&CG, College of Computer Science, Zhejiang University, Hangzhou 310027, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong

Nov 18, 2025J/K move · Enter open · S save

Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction

Hao Tian, Chenyangguang Zhang, Rui Liu, Wen Shen+1

Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu, China · PetroChina (Beijing) Digital Intelligence Research Institute Co., Ltd., Beijing, China · University of Chinese Academy of Sciences, Beijing, China+2

3D Gaussian Splatting 3D Reconstruction Hand-Object Interaction Reconstruction Dynamic Scene Reconstruction

Open paper AskPDF arXiv

This paper focuses on a challenging setting of simultaneously modeling geometry and appearance of hand-object interaction scenes without any object priors. We follow the trend of dynamic 3D Gaussian Splatting based methods, and address several significant challenges. To model complex hand-object interaction with mutual occlusion and edge blur, we present interaction-aware hand-object Gaussians with newly introduced optimizable parameters aiming to adopt piecewise linear hypothesis for clearer structural representation. Moreover, considering the complementarity and tightness of hand shape and object shape during interaction dynamics, we incorporate hand information into object deformation field, constructing interaction-aware dynamic fields to model flexible motions. To further address difficulties in the optimization process, we propose a progressive strategy that handles dynamic regions and static background step by step. Correspondingly, explicit regularizations are designed to stabilize the hand-object representations for smooth motion transition, physical interaction reality, and coherent lighting. Experiments show that our approach surpasses existing dynamic 3D-GS-based methods and achieves state-of-the-art performance in reconstructing dynamic hand-object interaction.

Abstract

Human-Scene Interaction Reconstruction

Explore similar work

Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction

CoGS: Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video

Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting