Organizations: MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA · MPI for Informatics, Germany and Bielefeld University, Germany
We present Grasp in Gaussians (GraG), a fast and robust method for reconstructing dynamic 3D hand-object interactions from a single monocular video. Unlike recent approaches that optimize heavy neural representations, our method focuses on tracking the hand and the object efficiently, once initialized from pretrained large models. Our key insight is that, given strong pretrained object and hand priors, accurate and temporally stable hand-object motion can be recovered using a compact Sum-of-Gaussians (SoG) representation, revived from classical tracking literature and integrated with generative Gaussian-based initializations. We initialize object pose and geometry using a video-adapted SAM3D pipeline, then convert the resulting dense Gaussian representation into a lightweight SoG via subsampling. This compact representation enables efficient and fast tracking while preserving geometric fidelity, with the pretrained priors and simple geometric/contact losses providing accurate hand-object placement. For the hand, we adopt a complementary strategy: starting from off-the-shelf monocular hand pose initialization, we refine hand motion using simple yet effective 2D joint, silhouette, depth, and contact losses, avoiding per-frame refinement of a detailed 3D hand appearance model while maintaining stable articulation. Extensive experiments on public benchmarks demonstrate that GraG reconstructs temporally coherent hand-object interactions on long sequences 4.4x-38.9x faster than prior work while preserving temporally coherent hand-object motion.
Figures & tables
Figure 1. Grasp in Gaussians (GraG): Given a single monocular video of a hand interacting with an object, GraG reconstructs 3D geometry and pose of the hand and the object. Our method is designed to be efficient for long sequences, and can reconstruct in-the-wild captured examples.
Figure 2. Method Overview. Given a monocular video, we recover per-frame hand-object poses and geometry. We first preprocess the video to obtain masks, initial hand poses, per-frame binary contact information, pointmaps and camera intrinsics/extrinsics. Preprocessing is followed by three stages: Stage 1 ( Section 4.1 ) for keyframe selection and object reconstruction, Stage 2 ( Section 4.2 ) for estimating a temporally stable initial object trajectory (and scale) by adapting SAM3D to videos, and Stage 3 ( Section 4.3 ) for hand and object pose and scale refinement via Sums-of-Gaussians tracking.
Figure 3. Stage 1 reconstructs a canonical object with MV-SAM3D by selecting keyframes and decoding shape tokens into dense Gaussians ( Section 4.1 ).
Figure 4. Stage 2: Temporally stable SAM3D pose estimation. Data flow for per-frame pose estimation. Top (Shape): Canonical shape tokens ZS⋆ are projected to latents XS⋆ and processed by the transformer. The predicted shape velocity is zeroed to keep the geometry unchanged. Bottom (Pose): Previous frame’s layout tokens Zt−1P are projected to latents XtP . The predicted velocity vθP is modified by a temporal guidance term (Eq. 1 ) derived from the difference between current and previous frame latents ( Xt−1P ). The guided velocity v~θP is mapped back to token space and added to the previous frame’s tokens to produce ZtP for decoding.
Figure 5. Stage 3 refines hand and object motion using an explicit tracking objective: we sparsify the dense Gaussians into a compact SoG and perform hand-aware SoG alignment with lightweight priors including silhouette loss Lsil to represent the distance to the silhouette edges, depth loss Ldepth between estimated pointmap medians and hand-object medians, and contact loss Lcontact between hand fingertips and object surface ( Section 4.3 ). For the silhouette loss, we visualize the log(DThand+1) and log(DTobject+1) .
Figure 6. Approximating the image as Gaussians and projected object SoG. From left to right: Reference frame, quad-tree build over the object area, image SoG used for tracking where each square represents a Gaussian with the same color, projected object SoG before optimization, projected object SoG at the end of optimization.
Figure 7. Comparing mesh/Gaussians-based optimization to SoG. We show the differences between objectives in optimization using mesh/Gaussian rendering vs using SoG. Since MV-SAM3D/SAM3D has difficulty generating high-quality texture, specifically texture with text, trying to optimize the object pose with the rendered mesh/Gaussians does not work well, as shown in Table 2 . In contrast, the SoG-based tracking formulation matches image regions more approximately and thus, even with inaccurate texture, we can still optimize the object pose.
Figure 8. Qualitative Comparison. We compare the output of GraG with previous SoTA works HOLD, BIGS, and MagicHOI on HO3D (first 2 rows) and HOT3D (last 2 rows). In the 2nd row 4th column, MagicHOI fails to produce a valid reconstruction; we therefore report it as N/A . Overall, GraG preserves sharper object geometry and yields more plausible hand poses (with fewer interpenetrations), while being substantially more efficient.
Figure 9. Ablation experiments. We visualize how key design choices affect reconstruction quality (two representative views per setting: camera view and back view). Using all frames during reconstruction yields an inaccurate canonical object shape and scale, leading to implausible grasps. Running Stage 2 SAM3D pose estimation without freezing the shape tokens destabilizes per-frame pose estimates (note that Stage 3 still utilizes the Stage 1 canonical object in this experiment). Replacing our compact SoG refinement with dense Gaussian Splatting (GS) tracking is slower and often fails to converge under the same iteration budget. Similarly, replacing SoG with optical flow causes tracking to collapse during hand-object occlusion, degrading the final reconstruction. Without the visibility gate, SoG alignment is biased by hand-occluded regions, producing incorrect object poses. Our full model achieves the highest fidelity, closely matching the ground truth.
Dataset
Method
CD [cm]↓
F10 [%]↑
MPJPE [mm]↓
CDh[cm]↓
Success Rate [%]↑
Runtime [h]↓
HO3Dv3
HOLD
0.78
92.0
23.4
4.27
100
10.5
BIGS *
0.67
94.1
23.9
7.61
100
3.60
MagicHOI
1.58
73.7
4.35
129
67.0
1.20
Ours
0.51
97.2
9.38
5.02
100
0.27
HOT3D
HOLD
3.14
53.3
21.9
122
50.0
10.5
BIGS
2.92
51.7
36.9
84.2
50.0
3.60
Table 1. Quantitative comparison. We report object reconstruction, hand accuracy, success rate, and runtime on HO3Dv3 and HOT3D. We highlight the best and second-best results. All methods are averaged over the successful results only. (*) BIGS results on HO3Dv3 differ from those reported by ( On et al., 2025 ) ; we obtain them by running the authors’ open-source implementation with the reported hyperparameters.
Ablation
CD [cm]↓
F10 [%]↑
MPJPE [mm]↓
CDh[cm]↓
Runtime [h]↓
random K=4 keyframe selection
0.69
93.9
11.5
9.36
0.27
single view from first frame
0.73
92.1
18.2
22.73
0.27
using all frames
0.68
93.6
11.5
9.95
0.30
w/o freezing shape
0.45
98.8
11.4
10.81
0.28
w/o temporal guidance ( λtemp=0 )
0.45
98.8
11.5
10.60
0.27
only foundation models (w/o Stage 3)
0.45
98.8
12.1
20.62
0.10
Table 2. Ablation study. We ablate key components of our method on 4 sequences, 2 sequences from HO3Dv3 and 2 sequences from HOT3D. The first block ablates Stage 1 object reconstruction, the second block ablates Stage 2 SAM3D pose adaptation, and the third block ablates Stage 3 tracking objectives. For Stage 2 and Stage 3 variants, the same canonical object reconstructed with balanced keyframes is used, so CD and F10 remain unchanged. The “w/o freezing shape” row does not freeze the shape latent during SAM3D pose adaptation, but still uses the Stage 1 object during Stage 3 tracking. Runtime is normalized per 100 frames.
Figure 12
Figure 12. Human-to-robot tool manipulation from GraG object trajectories. Top: monocular RGB frames from a human short-screwdriver demonstration. Bottom: corresponding visualizations of the SimToolReal policy observations. The green screwdriver indicates the current goal pose obtained from GraG’s reconstructed 6D object trajectory (Zoom in for best view). The policy receives the robot state, estimated object pose, and goal pose; once the current object pose is sufficiently close to the goal, the goal advances to the next pose in the sequence. This visualization renders the policy inputs and goal-tracking behavior, rather than a physics-based real-robot execution.
Setting
Number of Gaussians
Runtime (h)
Storage (MB)
Peak CUDA (GB)
CD ↓
F10 ↑
MPJPE ↓
CD h ↓
Sparse SoG
∼2,000
0.27
0.11
2.4
0.45
98.8
11.3
7.65
Reduced
∼60,000
1.20
3.20
5.5
0.44
99.1
11.3
7.50
Full
∼300,000
34.1
16.0
6.2
0.44
99.1
11.3
7.50
Table 3. Trade-off between Gaussian count, compute, memory, and reconstruction quality on the ablation set. Runtime is normalized to hours per 100 frames as in the main paper setting. “Sparse SoG” is our default setting where we use farthest point sampling to subsample from the reduced SAM3D Gaussian asset and convert to isotropic Gaussians. “Reduced” denotes SAM3D’s lower-count Gaussian decoder output, trained for downstream needs such as decoding fewer Gaussians for faster inference (see SAM3D’s Supplementary Material for detailed information); “Full” denotes the dense SAM3D Gaussian decoder output. Storage reports the memory occupied by the object Gaussian parameter tensors, while Peak CUDA reports the maximum CUDA memory allocated during optimization. Thus, Storage measures representation compactness, whereas Peak CUDA captures practical optimization-time GPU memory.
Dataset
Method
CDev [mm] ↓
MDev [mm] ↓
ACC h [m/s 2 ] ↓
ACC o [m/s 2 ] ↓
MRRPE h→o [mm] ↓
HO3Dv3
HOLD ( Fan et al., 2024 )
15.8
10.8
11.3
30.2
38.6
BIGS ( On et al., 2025 )
54.2
23.1
14.4
37.8
79.4
MagicHOI ( Wang et al., 2025 )
533
21.5
15.4
10.3
747
Ours
17.0
5.37
6.17
12.5
45.4
HOT3D
HOLD ( Fan et al., 2024 )
451
65.3
15.8
117
606
BIGS ( On et al., 2025 )
570
85.6
24.1
136
762
Table 4. Interaction-consistency metrics. We report interaction metrics from ARCTIC ( Fan et al., 2023 ) on HO3Dv3 and HOT3D, including Contact Deviation (CDev), Motion Deviation (MDev), hand/object acceleration error (ACC h , ACC o ), and Mean Relative Root Position Error between the hand and object (MRRPE h→o ). Lower is better for all metrics.
HO3Dv3
HOT3D
Contact signal
F1 ↑
Bal. Acc ↑
F1 ↑
Bal. Acc ↑
Gemini 3 (VLM, ours)
0.99
0.80
0.97
0.66
Mask-proximity (fallback)
0.96
0.70
0.98
0.59
Table 5. Contact-signal validation against mesh-proximity ground truth. Balanced accuracy (Bal. Acc) averages the in-contact and no-contact class accuracies; the VLM flag is the stronger signal on both datasets, most clearly on the rare no-contact frames.
Projection
Overlap
HO3Dv3 CDh[cm]↓
HOT3D CDh[cm]↓
Runtime [h]↓
Isotropic (ours)
Scalar closed-form
5.02±1.47
9.37±4.68
0.27
Anisotropic
Full matrix ( 2×2 )
5.29±2.04
10.43±6.39
0.30
Table 6. Gaussian-projection ablation. Comparing our scalar closed-form overlap against full 2D anisotropic projection. Modeling Jacobian-based anisotropic projection with matrix overlap does not improve tracking accuracy ( CDh ) and adds ∼11% runtime.
Object
Sequence name
bleach
ABF12
bleach
ABF14
potted meat
GPMF12
potted meat
GPMF14
cracker box
MC1
cracker box
MC4
Table 7. HO3Dv3 sequences for hand-object reconstruction. We directly use the same sequences from HOLD ( Fan et al., 2024 ) . All of these sequences are only right-handed.
Sequence name
Hand side
Start TS (ns)
End TS (ns)
Object ID
P0001_23fa0ee8
right
55003474858015
55015408224119
238686662724712
P0001_8d136980
right
54039408208193
54044341521984
238686662724712
P0001_b2bcbe28
left
54811408205972
54814874891157
258906041248094
P0001_f71fc9b1
right
66482274870136
66485208197200
249541253457812
P0010_8ff3e5c4
left
58695141597662
58697074869546
258906041248094
P0011_0c2d00ed
right
55563708197241
55572041548328
70709727230291
Table 8. HOT3D clips used in our evaluations, reported with sequence identifiers, hand side, temporal boundaries, and object IDs. Temporal boundaries are reported as HOT3D timecode timestamps (TS) in nanoseconds.
Figure 13. More qualitative results. We overlay rendered hand-object meshes on the input images for qualitative inspection and also show side views of the interactions.
This paper focuses on a challenging setting of simultaneously modeling geometry and appearance of hand-object interaction scenes without any object priors. We follow the trend of dynamic 3D Gaussian Splatting based methods, and address several significant challenges. To model complex hand-object interaction with mutual occlusion and edge blur, we present interaction-aware hand-object Gaussians with newly introduced optimizable parameters aiming to adopt piecewise linear hypothesis for clearer structural representation. Moreover, considering the complementarity and tightness of hand shape and object shape during interaction dynamics, we incorporate hand information into object deformation field, constructing interaction-aware dynamic fields to model flexible motions. To further address difficulties in the optimization process, we propose a progressive strategy that handles dynamic regions and static background step by step. Correspondingly, explicit regularizations are designed to stabilize the hand-object representations for smooth motion transition, physical interaction reality, and coherent lighting. Experiments show that our approach surpasses existing dynamic 3D-GS-based methods and achieves state-of-the-art performance in reconstructing dynamic hand-object interaction.
Hao Tian, Chenyangguang Zhang, Rui Liu +2
Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu, China · PetroChina (Beijing) Digital Intelligence Research Institute Co., Ltd., Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2
Understanding realistic hand-object interactions from monocular RGB videos is essential for AR/VR, robotics, and embodied AI. Existing methods rely on category-specific templates or heavy computation, yet still produce physically inconsistent hand-object alignment in 3D. We introduce GHOST (Gaussian Hand-Object Splatting), a fast, category-agnostic framework for reconstructing dynamic hand-object interactions using 2D Gaussian Splatting. GHOST represents both hands and objects as dense, view-consistent Gaussian discs and introduces three key innovations: (1) a geometric-prior retrieval and consistency loss that completes occluded object regions, (2) a grasp-aware alignment that refines hand translations and object scale to ensure realistic contact, and (3) a hand-aware background loss that prevents penalizing hand-occluded object regions. GHOST achieves complete, physically consistent, and animatable reconstructions from a single RGB video while running an order of magnitude faster than prior category-agnostic methods. Extensive experiments on ARCTIC, HO3D, and in-the-wild datasets demonstrate state-of-the-art accuracy in 3D reconstruction and 2D rendering quality, establishing GHOST as an efficient and robust solution for realistic hand-object interaction modeling. Code is available at https://github.com/ATAboukhadra/GHOST.
Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini +4
Synthesizing believable 4D hand-object grasping sequences is important for character animation, extended reality, and embodied agents. Grasping requires a hand to approach, contact, and lift or manipulate an object coherently. Existing synthesis methods often model only the final static grasp or rely on additional conditions, such as language, target grasps, or object trajectories, to specify the interaction. We instead study the underconstrained problem of generating a complete grasping motion from a single initial hand-object state. Our framework decouples hand-motion generation from object-motion recovery. Given an object point cloud in the initial wrist frame and the initial parameterized hand configuration, conditional flow matching predicts a sequence-level contact map and a hand pose at each normalized sequence time. This time-indexed formulation avoids the error accumulation of autoregressive rollout while representing the complete motion. We then infer an attachment frame and recover rigid-object motion through training-free, contact-guided weighted registration. We evaluate on held-out DexYCB objects and unseen object geometries reconstructed from in-the-wild videos by Open-CHOIR. Compared with LatentHOI, our method produces smoother hand and object motion in every setting, matches or improves on its contact and penetration metrics in the wild under a shared initialization, and is about 30x faster per generated frame. In a 15-participant study, our motions were preferred in roughly two thirds of (forced-choice pairwise) comparisons. Precise contact closure on novel geometry, however, remains a key open challenge.
Nga Kiu Kelly Ho, Yilin Liu, Hao Xu +1
University College London · The Chinese University of Hong Kong · Adobe Research