Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves

Xinyu Zhang; Ziyi Kou; Chuan Qin; Mia Huang; Ergys Ristani; Ankit Kumar; Lele Chen; Kun He

cs.CVMar 21, 2026

Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves

Authors: Xinyu Zhang, Ziyi Kou, Chuan Qin, Mia Huang, Ergys Ristani, Ankit Kumar, Lele Chen, Kun He,

Organizations: Meta Reality Labs · ‡Rutgers University

Abstract

Understanding hand-object interaction (HOI) is fundamental to computer vision, robotics, and AR/VR. However, conventional hand videos often lack essential physical information such as contact forces and motion signals, and are prone to frequent occlusions. To address the challenges, we present Glove2Hand, a framework that translates multi-modal sensing glove HOI videos into photorealistic bare hands, while faithfully preserving the underlying physical interaction dynamics. We introduce a novel 3D Gaussian hand model that ensures temporal rendering consistency. The rendered hand is seamlessly integrated into the scene using a diffusion-based hand restorer, which effectively handles complex hand-object interactions and non-rigid deformations. Leveraging Glove2Hand, we create HandSense, the first multi-modal HOI dataset featuring glove-to-hand videos with synchronized tactile and IMU signals. We demonstrate that HandSense significantly enhances downstream bare-hand applications, including video-based contact estimation and hand tracking under severe occlusion.

Figures & tables

Figure 2: Glove2Hand Training. We extract hand poses from input videos and render corresponding hand only frames using our 3D Gaussian hand model. The rendered hands are then overlayed onto the original videos with wrist regions masked. The resulting video is encoded into a latent space and serves as a condition for our diffusion hand restorer, which is trained to restore the original hand video.

Figure 3: Glove2Hand Inference. We render hand only frames using hand poses from glove videos. Prior to pasting, we erase the glove region with an optical-flow based background inpainter [ 81 ] , and ensures object pixels remain intact during pasting. The resulting video is encoded and fed as input to diffusion hand restorer.

Figure 4: Surface-grounded and Relightable Gaussian Hand. We defines Gaussians over the surface of a hand mesh, which benefits from the mesh geometric priors and naturally enables lighting estimation based on the consistent mesh surface normals.

Figure 5: HandSense Dataset. HandSense provides egocentric HOI videos collected by both sensing gloves and human bare hands. $\dagger$ denotes multi-modal sensing modalities from gloves.

Figure 6: Qualitative Comparison for Glove-to-Hand. Our Glove2Hand produces photorealistic hands with clean object contact. In contrast, existing methods generate blurry results, or suffer from strong visual artifacts, unrealistic textures, and inconsistent lighting. The bottom row demonstrates the high generalization of Glove2Hand to unseen subjects, objects, and backgrounds. For a fair comparison, Pix2Pix, BrushNet, and HandRefiner were retrained on our dataset using the same inputs and HOI masks as our method (Fig. 2 and Fig. 3 ).

Method	FID $\downarrow$	FVD $\downarrow$	FVD-long $\downarrow$
Vanilla Glove	74.9	38.3	45.5
InpaintAnything ∗ [ 73 ]	63.6	32.7	39.6
FoundHand ∗ [ 8 ]	82.2	47.9	50.5
VideoPainter ∗ [ 3 ]	72.2	35.7	42.2
CycleGAN [ 83 ]	63.6	30.8	37.4
Pix2Pix [ 31 ]	38.6	24.7	31.4

Table 1: Evaluation for Glove-to-Hand Video Translation. We evaluate Glove2Hand on HandSense dataset with FID, FVD (2-sec. clips), and FVD-long (on 10-30 sec. clips). (

*

) denotes methods evaluated off-the-shelf without fine-tuning on HandSense.

Figure 7: Human Evaluation Results . Our study compares the synthesized videos by Glove2Hand with real hand videos across four metrics ( $N$ =number of answers). The average of score distribution for our synthesized hands are close to real hands, with a max gap of 0.51. This gap (lower is better) is small relative to the 1.0 distance between adjacent categories, showing high realism.

Figure 8: Qualitative Ablations. Our surface-grounded Gaussian reconstructs coherent hand geometry and appearance compared to 2DGS [ 29 ] . Adding the relightable Gaussian improves shading. The diffusion hand restorer further enhances skin texture, connects the wrist, and removes visual artifacts. Finally, glove removal and object masks eliminate glove artifacts and refine the hand-object boundary.

Configuration	FID $\downarrow$	FVD $\downarrow$	FVD-long $\downarrow$
2DGS [ 29 ]	91.1	50	62.9
+Surface Grounding	60.3	35.1	46.6
+Relightable	56.7	30.7	40.2
+Diffusion	32.3	19.8	22.7
+Glove Removal	31.2	20.9	25
+Object Mask	30.1	19.5	24.5

Table 2: Ablation Results. We perform an incremental ablation study to evaluate the contribution of each proposed component. The results demonstrate that our components work cohesively, with each addition progressively improving the final performance. Surface grounding and the diffusion yield the most significant gains. Qualitative ablations are presented in Fig. 8 .

Figure 9: Glove-to-Hand (different subjects). We customize the output hand identity by training a 3D Gaussian hand per-subject, while sharing the diffusion hand restorer. The figure shows glove inputs (left) and the translated hand of different subjects (right).

Method		Contact IoU (%)	Precision (%)	Recall (%)
PressureVision++ [ 14 ]		0.8	62.5	0.9
Ours	Glove only	71.5	82.8	83.9
	G2H only	75.6	90.6	82
	Hand only	85.3	90	94.2
	Hand+G2H	88.2	92.6	94.9

Table 3: Results on Contact Estimation. We train video contact estimator on glove-to-hand (G2H) videos, and evaluate on real, manually-annotated videos using Contact IoU [ 15 ] , Precision, and Recall. Training on G2H videos significantly outperforms the glove ones. Combining real and G2H data yields the best result. G2H contact labels are automatically derived from tactile signals.

Figure 10: Contact Estimation Example. The model trained with only manually-annotated real hand data ( orange ) produces noisy and unstable predictions. In contrast, adding our glove-to-hand (G2H) data ( blue ) results in stable and accurate contact estimation.

Figure 11: Hand Tracking Visualization. UmeTrack struggles with occlusions, our model, finetuned on our synthesized videos with IMU tracking annotation, robustly tracks the occluded hand.

Method	MKPE $\downarrow$		MKPE.T $\downarrow$
Method	Occlusion	Overall	Occlusion	Overall
UmeTrack [ 22 ]	19.2	19.5	10.8	9.8
UmeTrack + Glove	27.2	26.5	11.5	11.2
UmeTrack + G2H (Ours)	16.6	17.8	9.9	9.4

Table 4: Results on Hand Tracking. We directly evaluate UmeTrack on HandSense and finetune it with IMU tracking results as ground-truth. Finetuning with Glove2Hand videos improves tracking accuracy over UmeTrack, especially for occluded cases as IMU sensors are robust to camera occlusion. In contrast, training on glove data degrades performance due to the domain gap.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Exemplar Instruction
1. Bottle Opening	“Open the mustard bottle using only your index finger and thumb.”
2. Object Rotation	“Hold the object with index, thumb, and middle fingers; rotate it in front of you.”
3. Piano Keystroke	“Press different piano keys sequentially using only your ring finger.”
4. Pick-and-Place	“Pick up the object using a whole-hand grasp and place it on the table.”
5. Surface Interaction	“Press and slide your index finger firmly against the table surface.”
6. In-Hand Rotation	“Rotate the object within your hand, maximizing finger contact and occlusion.”

Appendix

Table 5: List of interaction tasks and exemplar instructions used during data collection.

Dataset	Images	Subjects/Objects	Pose	Contact
H2O [ 38 ]	572K	4 / 8	Optim.	Estim.
ARCTIC [ 12 ]	2.1M	10 / 11	MoCap	Estim.
HO-Cap [ 64 ]	699K	9 / 64	Optim.	-
HOI4D [ 42 ]	2.4M	4 / 800	Manual	-
HOT3D [ 2 ]	3.7M	19 / 33	MoCap
HandSense (Ours)	200K	5 / 10	MoCap	Measured.

Appendix

Table 6: Datasets with Contact Labels. HandSense is the first HOI video dataset to provide direct, sensor-measured contact.

Figure 12: Qualitative Examples of User Ratings. Samples rated as “Realistic” or “Very Realistic” are perceptually indistinguishable from real hands. “Neutral” samples generally exhibit valid geometry and interaction plausibility but may lack skin details. “Unrealistic” samples typically display unnatural hand-object boundary or non-negligible visual artifacts.

Metric		Mean Opinion Score $\uparrow$ / Gap to Real $\downarrow$
Metric		In-Domain	Cross-Subject	In-the-Wild
Image	Hand Realism	4.04 / 0.02	3.67 / 0.39	2.68 / 1.37
Image	HOI Realism	4.06 / 0.01	3.51 / 0.56	2.61 / 1.46
Video	Hand Realism	3.83 / 0.51	3.65 / 0.69	2.69 / 1.65
	HOI Realism	3.96 / 0.35	3.78 / 0.53	2.84 / 1.47
	Motion Stability	3.64 / 0.50	3.37 / 0.77	2.54 / 1.60

Appendix

Table 7: Human Evaluation Results. We report the Mean Opinion Score (MOS) and the perceptual gap relative to real data (Gap). In-Domain synthesis achieves performance near ground truth (Gap

<0.05

for images). Cross-Subject synthesis maintains high efficacy with gaps consistently below 1.0. In-the-Wild performance reflects the expected challenge of unseen environments but remains within a reasonable qualitative range.

Figure 13: Human Evaluation Interface. Participants view a randomly sampled image or video and rate it against specific criteria before proceeding.

Panel of Figure 14 — Figure 14: Human Evaluation Questionnaire.

Figure 15: Additional Qualitative Comparison for Glove-to-Hand.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting
Mar 19, 2026Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini +4Hand-Object Interaction Reconstruction Hand-Object Interaction
RPTU · DFKI-AV Kaiserslautern · NUST-SEECS Pakistan +1
DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction
Sep 29, 2026Wenliang Guo, Zhanbo Huang, Yu Kong3D Reconstruction Monocular 3D Reconstruction
Michigan State University
PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph
Aug 3, 2026Zhenhao Zhang, Jiajun Zhang, Wei Min +1HOI Generation Dexterous Manipulation
Tsinghua University · Shadow AI

Mar 19, 2026J/K move · Enter open · S save

GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa+3

RPTU · DFKI-AV Kaiserslautern · NUST-SEECS Pakistan+1

Hand-Object Interaction Reconstruction Hand-Object Interaction 3D Reconstruction Human-Object Interaction 3D Hand Reconstruction Monocular 3D Reconstruction+2

Open paper AskPDF arXiv

Understanding realistic hand-object interactions from monocular RGB videos is essential for AR/VR, robotics, and embodied AI. Existing methods rely on category-specific templates or heavy computation, yet still produce physically inconsistent hand-object alignment in 3D. We introduce GHOST (Gaussian Hand-Object Splatting), a fast, category-agnostic framework for reconstructing dynamic hand-object interactions using 2D Gaussian Splatting. GHOST represents both hands and objects as dense, view-consistent Gaussian discs and introduces three key innovations: (1) a geometric-prior retrieval and consistency loss that completes occluded object regions, (2) a grasp-aware alignment that refines hand translations and object scale to ensure realistic contact, and (3) a hand-aware background loss that prevents penalizing hand-occluded object regions. GHOST achieves complete, physically consistent, and animatable reconstructions from a single RGB video while running an order of magnitude faster than prior category-agnostic methods. Extensive experiments on ARCTIC, HO3D, and in-the-wild datasets demonstrate state-of-the-art accuracy in 3D reconstruction and 2D rendering quality, establishing GHOST as an efficient and robust solution for realistic hand-object interaction modeling. Code is available at https://github.com/ATAboukhadra/GHOST.

Abstract

Human-Object Interaction

Explore similar work

GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph