Understanding hand-object interaction (HOI) is fundamental to computer vision, robotics, and AR/VR. However, conventional hand videos often lack essential physical information such as contact forces and motion signals, and are prone to frequent occlusions. To address the challenges, we present Glove2Hand, a framework that translates multi-modal sensing glove HOI videos into photorealistic bare hands, while faithfully preserving the underlying physical interaction dynamics. We introduce a novel 3D Gaussian hand model that ensures temporal rendering consistency. The rendered hand is seamlessly integrated into the scene using a diffusion-based hand restorer, which effectively handles complex hand-object interactions and non-rigid deformations. Leveraging Glove2Hand, we create HandSense, the first multi-modal HOI dataset featuring glove-to-hand videos with synchronized tactile and IMU signals. We demonstrate that HandSense significantly enhances downstream bare-hand applications, including video-based contact estimation and hand tracking under severe occlusion.
Figures & tables
Figure 2: Glove2Hand Training. We extract hand poses from input videos and render corresponding hand only frames using our 3D Gaussian hand model. The rendered hands are then overlayed onto the original videos with wrist regions masked. The resulting video is encoded into a latent space and serves as a condition for our diffusion hand restorer, which is trained to restore the original hand video.
Figure 3: Glove2Hand Inference. We render hand only frames using hand poses from glove videos. Prior to pasting, we erase the glove region with an optical-flow based background inpainter [ 81 ] , and ensures object pixels remain intact during pasting. The resulting video is encoded and fed as input to diffusion hand restorer.
Figure 4: Surface-grounded and Relightable Gaussian Hand. We defines Gaussians over the surface of a hand mesh, which benefits from the mesh geometric priors and naturally enables lighting estimation based on the consistent mesh surface normals.
Figure 5: HandSense Dataset. HandSense provides egocentric HOI videos collected by both sensing gloves and human bare hands. † denotes multi-modal sensing modalities from gloves.
Figure 6: Qualitative Comparison for Glove-to-Hand. Our Glove2Hand produces photorealistic hands with clean object contact. In contrast, existing methods generate blurry results, or suffer from strong visual artifacts, unrealistic textures, and inconsistent lighting. The bottom row demonstrates the high generalization of Glove2Hand to unseen subjects, objects, and backgrounds. For a fair comparison, Pix2Pix, BrushNet, and HandRefiner were retrained on our dataset using the same inputs and HOI masks as our method (Fig. 2 and Fig. 3 ).
Method
FID ↓
FVD ↓
FVD-long ↓
Vanilla Glove
74.9
38.3
45.5
InpaintAnything ∗ [ 73 ]
63.6
32.7
39.6
FoundHand ∗ [ 8 ]
82.2
47.9
50.5
VideoPainter ∗ [ 3 ]
72.2
35.7
42.2
CycleGAN [ 83 ]
63.6
30.8
37.4
Pix2Pix [ 31 ]
38.6
24.7
31.4
Table 1: Evaluation for Glove-to-Hand Video Translation. We evaluate Glove2Hand on HandSense dataset with FID, FVD (2-sec. clips), and FVD-long (on 10-30 sec. clips). ( ∗ ) denotes methods evaluated off-the-shelf without fine-tuning on HandSense.
Figure 7: Human Evaluation Results . Our study compares the synthesized videos by Glove2Hand with real hand videos across four metrics ( N =number of answers). The average of score distribution for our synthesized hands are close to real hands, with a max gap of 0.51. This gap (lower is better) is small relative to the 1.0 distance between adjacent categories, showing high realism.
Figure 8: Qualitative Ablations. Our surface-grounded Gaussian reconstructs coherent hand geometry and appearance compared to 2DGS [ 29 ] . Adding the relightable Gaussian improves shading. The diffusion hand restorer further enhances skin texture, connects the wrist, and removes visual artifacts. Finally, glove removal and object masks eliminate glove artifacts and refine the hand-object boundary.
Configuration
FID ↓
FVD ↓
FVD-long ↓
2DGS [ 29 ]
91.1
50
62.9
+Surface Grounding
60.3
35.1
46.6
+Relightable
56.7
30.7
40.2
+Diffusion
32.3
19.8
22.7
+Glove Removal
31.2
20.9
25
+Object Mask
30.1
19.5
24.5
Table 2: Ablation Results. We perform an incremental ablation study to evaluate the contribution of each proposed component. The results demonstrate that our components work cohesively, with each addition progressively improving the final performance. Surface grounding and the diffusion yield the most significant gains. Qualitative ablations are presented in Fig. 8 .
Figure 9: Glove-to-Hand (different subjects). We customize the output hand identity by training a 3D Gaussian hand per-subject, while sharing the diffusion hand restorer. The figure shows glove inputs (left) and the translated hand of different subjects (right).
Method
Contact IoU (%)
Precision (%)
Recall (%)
PressureVision++ [ 14 ]
0.8
62.5
0.9
Ours
Glove only
71.5
82.8
83.9
G2H only
75.6
90.6
82
Hand only
85.3
90
94.2
Hand+G2H
88.2
92.6
94.9
Table 3: Results on Contact Estimation. We train video contact estimator on glove-to-hand (G2H) videos, and evaluate on real, manually-annotated videos using Contact IoU [ 15 ] , Precision, and Recall. Training on G2H videos significantly outperforms the glove ones. Combining real and G2H data yields the best result. G2H contact labels are automatically derived from tactile signals.
Figure 10: Contact Estimation Example. The model trained with only manually-annotated real hand data ( orange ) produces noisy and unstable predictions. In contrast, adding our glove-to-hand (G2H) data ( blue ) results in stable and accurate contact estimation.
Figure 11: Hand Tracking Visualization. UmeTrack struggles with occlusions, our model, finetuned on our synthesized videos with IMU tracking annotation, robustly tracks the occluded hand.
Method
MKPE ↓
MKPE.T ↓
Occlusion
Overall
Occlusion
Overall
UmeTrack [ 22 ]
19.2
19.5
10.8
9.8
UmeTrack + Glove
27.2
26.5
11.5
11.2
UmeTrack + G2H (Ours)
16.6
17.8
9.9
9.4
Table 4: Results on Hand Tracking. We directly evaluate UmeTrack on HandSense and finetune it with IMU tracking results as ground-truth. Finetuning with Glove2Hand videos improves tracking accuracy over UmeTrack, especially for occluded cases as IMU sensors are robust to camera occlusion. In contrast, training on glove data degrades performance due to the domain gap.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Exemplar Instruction
1. Bottle Opening
“Open the mustard bottle using only your index finger and thumb.”
2. Object Rotation
“Hold the object with index, thumb, and middle fingers; rotate it in front of you.”
3. Piano Keystroke
“Press different piano keys sequentially using only your ring finger.”
4. Pick-and-Place
“Pick up the object using a whole-hand grasp and place it on the table.”
5. Surface Interaction
“Press and slide your index finger firmly against the table surface.”
6. In-Hand Rotation
“Rotate the object within your hand, maximizing finger contact and occlusion.”
Appendix
Table 5: List of interaction tasks and exemplar instructions used during data collection.
Dataset
Images
Subjects/Objects
Pose
Contact
H2O [ 38 ]
572K
4 / 8
Optim.
Estim.
ARCTIC [ 12 ]
2.1M
10 / 11
MoCap
Estim.
HO-Cap [ 64 ]
699K
9 / 64
Optim.
-
HOI4D [ 42 ]
2.4M
4 / 800
Manual
-
HOT3D [ 2 ]
3.7M
19 / 33
MoCap
HandSense (Ours)
200K
5 / 10
MoCap
Measured.
Appendix
Table 6: Datasets with Contact Labels. HandSense is the first HOI video dataset to provide direct, sensor-measured contact.
Figure 12: Qualitative Examples of User Ratings. Samples rated as “Realistic” or “Very Realistic” are perceptually indistinguishable from real hands. “Neutral” samples generally exhibit valid geometry and interaction plausibility but may lack skin details. “Unrealistic” samples typically display unnatural hand-object boundary or non-negligible visual artifacts.
Metric
Mean Opinion Score ↑ / Gap to Real ↓
In-Domain
Cross-Subject
In-the-Wild
Image
Hand Realism
4.04 / 0.02
3.67 / 0.39
2.68 / 1.37
HOI Realism
4.06 / 0.01
3.51 / 0.56
2.61 / 1.46
Video
Hand Realism
3.83 / 0.51
3.65 / 0.69
2.69 / 1.65
HOI Realism
3.96 / 0.35
3.78 / 0.53
2.84 / 1.47
Motion Stability
3.64 / 0.50
3.37 / 0.77
2.54 / 1.60
Appendix
Table 7: Human Evaluation Results. We report the Mean Opinion Score (MOS) and the perceptual gap relative to real data (Gap). In-Domain synthesis achieves performance near ground truth (Gap <0.05 for images). Cross-Subject synthesis maintains high efficacy with gaps consistently below 1.0. In-the-Wild performance reflects the expected challenge of unseen environments but remains within a reasonable qualitative range.
Figure 13: Human Evaluation Interface. Participants view a randomly sampled image or video and rate it against specific criteria before proceeding.
Figure 14: Human Evaluation Questionnaire.
Figure 15: Additional Qualitative Comparison for Glove-to-Hand.
Understanding realistic hand-object interactions from monocular RGB videos is essential for AR/VR, robotics, and embodied AI. Existing methods rely on category-specific templates or heavy computation, yet still produce physically inconsistent hand-object alignment in 3D. We introduce GHOST (Gaussian Hand-Object Splatting), a fast, category-agnostic framework for reconstructing dynamic hand-object interactions using 2D Gaussian Splatting. GHOST represents both hands and objects as dense, view-consistent Gaussian discs and introduces three key innovations: (1) a geometric-prior retrieval and consistency loss that completes occluded object regions, (2) a grasp-aware alignment that refines hand translations and object scale to ensure realistic contact, and (3) a hand-aware background loss that prevents penalizing hand-occluded object regions. GHOST achieves complete, physically consistent, and animatable reconstructions from a single RGB video while running an order of magnitude faster than prior category-agnostic methods. Extensive experiments on ARCTIC, HO3D, and in-the-wild datasets demonstrate state-of-the-art accuracy in 3D reconstruction and 2D rendering quality, establishing GHOST as an efficient and robust solution for realistic hand-object interaction modeling. Code is available at https://github.com/ATAboukhadra/GHOST.
Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini +4
We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.
Hand-object interaction (HOI) is a fundamental human behavior with broad applications in AR/VR, digital humans, and embodied interaction. Existing methods typically require predefined object geometry, object trajectories, or task-specific conditions, limiting their use with natural real-world inputs. To address this, we study a more practical problem of synthesizing 3D hand-object interaction sequences from a single RGB photograph and an open-vocabulary language instruction, and introduce PhotoHOI. PhotoHOI first uses a vision-language model to parse the input image and instruction into a structured task specification, including the interaction object, target region, and spatial relation. It then recovers a compact task-relevant 3D scene and plans a smooth collision-aware object trajectory based on the recovered object states, support relations, and surrounding scene geometry. To synthesize hand motion that generalizes to real-world photographs and unseen objects, it learns transferable task-conditioned contact and contact-conditioned grasp priors from large-scale affordance and HOI data. The grasp is further refined in a learned latent space, constraining the optimization to a plausible hand-pose manifold. Experiments on GRAB and H2O demonstrate improved contact quality and reduced penetration over representative baselines. Results on real-world photographs further demonstrate higher task success and scene consistency, together with generalization to unseen objects and open-vocabulary instructions.