GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

Ahmed Tawfik Aboukhadra; Marcel Rogge; Nadia Robertini; Abdalla Arafa; Jameel Malik; Ahmed Elhayek; Didier Stricker

cs.CVMar 19, 2026

GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

Authors: Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, Didier Stricker

Organizations: RPTU · DFKI-AV Kaiserslautern · NUST-SEECS Pakistan · UPM Saudi Arabia

Abstract

Understanding realistic hand-object interactions from monocular RGB videos is essential for AR/VR, robotics, and embodied AI. Existing methods rely on category-specific templates or heavy computation, yet still produce physically inconsistent hand-object alignment in 3D. We introduce GHOST (Gaussian Hand-Object Splatting), a fast, category-agnostic framework for reconstructing dynamic hand-object interactions using 2D Gaussian Splatting. GHOST represents both hands and objects as dense, view-consistent Gaussian discs and introduces three key innovations: (1) a geometric-prior retrieval and consistency loss that completes occluded object regions, (2) a grasp-aware alignment that refines hand translations and object scale to ensure realistic contact, and (3) a hand-aware background loss that prevents penalizing hand-occluded object regions. GHOST achieves complete, physically consistent, and animatable reconstructions from a single RGB video while running an order of magnitude faster than prior category-agnostic methods. Extensive experiments on ARCTIC, HO3D, and in-the-wild datasets demonstrate state-of-the-art accuracy in 3D reconstruction and 2D rendering quality, establishing GHOST as an efficient and robust solution for realistic hand-object interaction modeling. Code is available at https://github.com/ATAboukhadra/GHOST.

Figures & tables

Figure 1: Our method reconstructs complete 3D hand–object interactions from a single monocular RGB video—recovering full object surfaces and realistic hand contact even under severe occlusions—while enabling fast, accurate, category-agnostic reconstruction and novel-view rendering.

Panel of Figure 2 — Figure 2: (Top) Overview of our pipeline, which consists of three stages. In preprocessing , we extract hand meshes, camera poses, and object information (i.e. mask, point cloud, and geometric prior). During hand–object alignment , object’s scale and hand translations are optimized using grasp-aware and temporal reasoning. In the Gaussian Splatting stage , hands and objects are jointly reconstructed with occlusion-aware losses. (Bottom) Retrieval and alignment pipeline used to obtain the object’s geometric prior.

Panel of Figure 3 — Figure 3: Qualitative comparison showing (a) the effect of our novel background loss $\mathcal{L}_{bkg,h}$ on object reconstruction quality. (b) Top : Gaussian centers on the canonical hand; middle : deformed hand mesh with aligned Gaussian centers after $\mathcal{T}_{aff}$ ; bottom : final animatable Gaussian hand after training.

Figure 4: Qualitative results of GHOST on ARCTIC [ 14 ] , HO3D [ 15 ] , and in-the-wild examples: Left : shows aligned 3D hand meshes with the object’s geometric prior obtained during HO alignment. Right: presents photorealistic Gaussian Splatting renderings from original viewpoint and novel viewpoints. GHOST produces consistent hand–object alignment in 3D and maintains realistic appearance even under view changes, enabling physically plausible interaction reconstruction and high-fidelity rendering across viewpoints.

Figure 5: Qualitative comparison demonstrating the effect of the geometric loss $\mathcal{L}_{geo}$ on the quality of reconstructed object point clouds derived from Gaussian centers.

SfM	$\mathcal{L}_{geo}$	$\mathcal{\tau}_{fill}$	$\mathcal{\tau}_{out}$	CD h $\downarrow$	CD ICP $\downarrow$
HLoc [ 47 , 45 , 46 ]	✗	-	-	-	8.18
HLoc [ 47 , 45 , 46 ]	✓	-	-	-	4.86
VGGSfM [ 50 ]	✗	-	-	18.32	2.78
VGGSfM [ 50 ]	✓	1.0	5.0	18.21	3.66
VGGSfM [ 50 ]	✓	0.1	2.0	18.42	2.33
VGGSfM [ 50 ]	✓	0.01	2.0	18.37	2.83

Table 1: Ablation study on the impact of SfM method and geometric consistency loss on object reconstruction and interaction scores for the ARCTIC dataset.

\mathcal{\tau}_{fill}

is reported in

mm

\mathcal{\tau}_{out}

is in

cm

, and CD metrics are in

cm^{2}

Method	$\text{MPJPE}_{RA,h}$ $\downarrow$	$\text{MPJPE}_{RA,l}$ $\downarrow$	$\text{MPJPE}_{RA,r}$ $\downarrow$	$\text{CD}_{ICP}$ $\downarrow$	$\text{CD}_{r}$ $\downarrow$	$\text{CD}_{l}$ $\downarrow$	$\text{CD}_{h}$ $\downarrow$	$\text{F}_{10mm}$ (%) $\uparrow$	$\text{F}_{5mm}$ (%) $\uparrow$
HOLD [ 13 ]	25.91	27.13	24.70	2.07	123.54	105.92	114.73	63.92	37.13
BIGS [ 35 ]	24.49	24.63	24.35	1.36	31.28	46.11	38.69	81.78	56.41
Ours	24.07	25.42	22.71	2.26	13.40	23.41	18.40	60.88	34.67

Table 2: Comparison of 3D metrics across HOLD, BIGS, and our method. All MPJPE metrics are in

mm

, and all CD metrics are in

cm^{2}

Method	2D Rendering Quality (ARCTIC)			2D Rendering Quality (HO3D)			Runtime $\downarrow$
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Runtime $\downarrow$
HOLD [ 13 ]	12.83	0.66	0.32	16.20	0.74	0.21	16h
BIGS [ 35 ]	24.87	0.96	0.05	24.51	0.92	0.07	13h
Ours	25.93	0.88	0.02	21.37	0.75	0.03	1h

Table 3: Evaluation of 2D rendering quality on ARCTIC and HO3D datasets, comparing our method with prior works. In addition, we report significant runtime improvement.

Figure 6: Qualitative comparison of 2D rendered images: We compare our results against HOLD [ 13 ] using representative examples from the evaluated datasets. As shown, ours produces higher quality and more consistent 2D renderings, whereas HOLD exhibits noise, blur, and degraded visual quality.

Panel of Figure 7 — Figure 7: Rendering quality as a function of surface Gaussian density. Higher density improves reconstruction fidelity.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: (a) SAM2 [ 42 ] is initialized with 3 seed pixels to segment and track the hands ( $\mathcal{M}_{t}^{h}$ ) and the object ( $\mathcal{M}_{t}^{o}$ ) in the scene. (b) During the grasping detection, the object’s motion vector $\hat{\mathcal{T}}^{o}_{xy}$ (blue arrow) is compared with left hand’s motion vector $\hat{\mathcal{T}}^{l}_{xy}$ (orange arrow) and right hand’s motion vector $\hat{\mathcal{T}}^{r}_{xy}$ (red arrow). The example shows a left hand grasp based on the similarity between motion vectors.

Figure 9: OpenShape [ 30 ] retrieves 3D models from Objaverse [ 11 , 10 ] , however, the retrieved 3D models do not always match the geometry of the desired object. Therefore, the final geometric prior $\mathcal{O}$ can be suboptimal.

Panel of Figure 10 — Figure 10: a) Initial hand reconstructions $\mathcal{V}_{t}^{h}$ obtained from HaMeR [ 38 ] suffer from Jitter under occlusion. Detecting jitter based on temporal cues, detection confidence, and interpolation results in improving initial hand meshes $\mathcal{V}_{t}^{h}$ . b) Structure-from-Motion (SfM) has a large impact on subsequent steps. VGGSfM [ 50 ] improved SfM when applied to the arctic data compared to the HLoc+COLMAP [ 47 , 45 ] pipeline. However, VGGSfM is sensitive to hyperparameter selection and does not always show similar improvements as seen in the last row.

Panel of Figure 12 — Figure 12: a) Applying $\mathcal{L}_{bkg,h}$ for object reconstruction fails when the hand never changes its contact point with the object. In that case, unwanted gaussians spawn in the hand region. b) In some cases, the retrieved geometric prior $\mathcal{O}$ does not align perfectly with the initial object’s point cloud $\mathcal{P}_{sfm}$ (Middle column). The results of applying $\mathcal{L}_{geo}$ in this case will result in moving gaussian centers $c_{o}$ towards unwanted regions (see blue point cloud).

Figure 13: We also provide an interactive 3D viewer for Gaussian hand avatars. The interface visualizes and controls MANO-based [ 44 ] hand reconstructions, allowing users to adjust both pose and shape parameters in real time. Imported motion sequences can be played to animate the Gaussian hand avatar.

Hand-Object Interaction Reconstruction

See all 18 papers

Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions
Apr 14, 2026Ayce Idil Aytekin, Xu Chen, Zhengyang Shen +4Hand-Object Interaction Reconstruction 3D Reconstruction
MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA +1
Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction
Nov 18, 2025Hao Tian, Chenyangguang Zhang, Rui Liu +23D Gaussian Splatting 3D Reconstruction
Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu, China · PetroChina (Beijing) Digital Intelligence Research Institute Co., Ltd., Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2
Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
May 20, 2026Hao Xu, Yilin Liu, Yinqiao Wang +2Human-Object Interaction 3D Reconstruction
The Chinese University of Hong Kong, China · University College London, United Kingdom · Adobe Research, United Kingdom

Apr 14, 2026J/K move · Enter open · S save

Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions

Ayce Idil Aytekin, Xu Chen, Zhengyang Shen, Thabo Beeler+3

MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA+1

Hand-Object Interaction Reconstruction 3D Reconstruction Monocular 3D Reconstruction 3D Hand Pose Estimation

Open paper AskPDF arXiv

We present Grasp in Gaussians (GraG), a fast and robust method for reconstructing dynamic 3D hand-object interactions from a single monocular video. Unlike recent approaches that optimize heavy neural representations, our method focuses on tracking the hand and the object efficiently, once initialized from pretrained large models. Our key insight is that, given strong pretrained object and hand priors, accurate and temporally stable hand-object motion can be recovered using a compact Sum-of-Gaussians (SoG) representation, revived from classical tracking literature and integrated with generative Gaussian-based initializations. We initialize object pose and geometry using a video-adapted SAM3D pipeline, then convert the resulting dense Gaussian representation into a lightweight SoG via subsampling. This compact representation enables efficient and fast tracking while preserving geometric fidelity, with the pretrained priors and simple geometric/contact losses providing accurate hand-object placement. For the hand, we adopt a complementary strategy: starting from off-the-shelf monocular hand pose initialization, we refine hand motion using simple yet effective 2D joint, silhouette, depth, and contact losses, avoiding per-frame refinement of a detailed 3D hand appearance model while maintaining stable articulation. Extensive experiments on public benchmarks demonstrate that GraG reconstructs temporally coherent hand-object interactions on long sequences 4.4x-38.9x faster than prior work while preserving temporally coherent hand-object motion.

Abstract

Hand-Object Interaction Reconstruction

Explore similar work

Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions

Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction

Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction