cs.CVApr 14, 2026
SaveGrasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions
Organizations: MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA · MPI for Informatics, Germany and Bielefeld University, Germany
Abstract
We present Grasp in Gaussians (GraG), a fast and robust method for reconstructing dynamic 3D hand-object interactions from a single monocular video. Unlike recent approaches that optimize heavy neural representations, our method focuses on tracking the hand and the object efficiently, once initialized from pretrained large models. Our key insight is that, given strong pretrained object and hand priors, accurate and temporally stable hand-object motion can be recovered using a compact Sum-of-Gaussians (SoG) representation, revived from classical tracking literature and integrated with generative Gaussian-based initializations. We initialize object pose and geometry using a video-adapted SAM3D pipeline, then convert the resulting dense Gaussian representation into a lightweight SoG via subsampling. This compact representation enables efficient and fast tracking while preserving geometric fidelity, with the pretrained priors and simple geometric/contact losses providing accurate hand-object placement. For the hand, we adopt a complementary strategy: starting from off-the-shelf monocular hand pose initialization, we refine hand motion using simple yet effective 2D joint, silhouette, depth, and contact losses, avoiding per-frame refinement of a detailed 3D hand appearance model while maintaining stable articulation. Extensive experiments on public benchmarks demonstrate that GraG reconstructs temporally coherent hand-object interactions on long sequences 4.4x-38.9x faster than prior work while preserving temporally coherent hand-object motion.
Figures & tables
Figure 1. Grasp in Gaussians (GraG): Given a single monocular video of a hand interacting with an object, GraG reconstructs 3D geometry and pose of the hand and the object. Our method is designed to be efficient for long sequences, and can reconstruct in-the-wild captured examples.
Figure 2. Method Overview. Given a monocular video, we recover per-frame hand-object poses and geometry. We first preprocess the video to obtain masks, initial hand poses, per-frame binary contact information, pointmaps and camera intrinsics/extrinsics. Preprocessing is followed by three stages: Stage 1 ( Section 4.1 ) for keyframe selection and object reconstruction, Stage 2 ( Section 4.2 ) for estimating a temporally stable initial object trajectory (and scale) by adapting SAM3D to videos, and Stage 3 ( Section 4.3 ) for hand and object pose and scale refinement via Sums-of-Gaussians tracking.
Figure 3. Stage 1 reconstructs a canonical object with MV-SAM3D by selecting keyframes and decoding shape tokens into dense Gaussians ( Section 4.1 ).
Figure 4. Stage 2: Temporally stable SAM3D pose estimation. Data flow for per-frame pose estimation. Top (Shape): Canonical shape tokens are projected to latents and processed by the transformer. The predicted shape velocity is zeroed to keep the geometry unchanged. Bottom (Pose): Previous frame’s layout tokens are projected to latents . The predicted velocity is modified by a temporal guidance term (Eq. 1 ) derived from the difference between current and previous frame latents ( ). The guided velocity is mapped back to token space and added to the previous frame’s tokens to produce for decoding.
Figure 5. Stage 3 refines hand and object motion using an explicit tracking objective: we sparsify the dense Gaussians into a compact SoG and perform hand-aware SoG alignment with lightweight priors including silhouette loss to represent the distance to the silhouette edges, depth loss between estimated pointmap medians and hand-object medians, and contact loss between hand fingertips and object surface ( Section 4.3 ). For the silhouette loss, we visualize the and .
Figure 6. Approximating the image as Gaussians and projected object SoG. From left to right: Reference frame, quad-tree build over the object area, image SoG used for tracking where each square represents a Gaussian with the same color, projected object SoG before optimization, projected object SoG at the end of optimization.
Figure 7. Comparing mesh/Gaussians-based optimization to SoG. We show the differences between objectives in optimization using mesh/Gaussian rendering vs using SoG. Since MV-SAM3D/SAM3D has difficulty generating high-quality texture, specifically texture with text, trying to optimize the object pose with the rendered mesh/Gaussians does not work well, as shown in Table 2 . In contrast, the SoG-based tracking formulation matches image regions more approximately and thus, even with inaccurate texture, we can still optimize the object pose.
Figure 8. Qualitative Comparison. We compare the output of GraG with previous SoTA works HOLD, BIGS, and MagicHOI on HO3D (first 2 rows) and HOT3D (last 2 rows). In the 2nd row 4th column, MagicHOI fails to produce a valid reconstruction; we therefore report it as N/A . Overall, GraG preserves sharper object geometry and yields more plausible hand poses (with fewer interpenetrations), while being substantially more efficient.
Figure 9. Ablation experiments. We visualize how key design choices affect reconstruction quality (two representative views per setting: camera view and back view). Using all frames during reconstruction yields an inaccurate canonical object shape and scale, leading to implausible grasps. Running Stage 2 SAM3D pose estimation without freezing the shape tokens destabilizes per-frame pose estimates (note that Stage 3 still utilizes the Stage 1 canonical object in this experiment). Replacing our compact SoG refinement with dense Gaussian Splatting (GS) tracking is slower and often fails to converge under the same iteration budget. Similarly, replacing SoG with optical flow causes tracking to collapse during hand-object occlusion, degrading the final reconstruction. Without the visibility gate, SoG alignment is biased by hand-occluded regions, producing incorrect object poses. Our full model achieves the highest fidelity, closely matching the ground truth.
| Dataset | Method | CD | F10 | MPJPE | Success Rate | Runtime | |
|---|---|---|---|---|---|---|---|
| HO3Dv3 | HOLD | 0.78 | 92.0 | 23.4 | 4.27 | 100 | 10.5 |
| BIGS * | 0.67 | 94.1 | 23.9 | 7.61 | 100 | 3.60 | |
| MagicHOI | 1.58 | 73.7 | 4.35 | 129 | 67.0 | 1.20 | |
| Ours | 0.51 | 97.2 | 9.38 | 5.02 | 100 | 0.27 | |
| HOT3D | HOLD | 3.14 | 53.3 | 21.9 | 122 | 50.0 | 10.5 |
| BIGS | 2.92 | 51.7 | 36.9 | 84.2 | 50.0 | 3.60 |
Table 1. Quantitative comparison. We report object reconstruction, hand accuracy, success rate, and runtime on HO3Dv3 and HOT3D. We highlight the best and second-best results. All methods are averaged over the successful results only. (*) BIGS results on HO3Dv3 differ from those reported by ( On et al., 2025 ) ; we obtain them by running the authors’ open-source implementation with the reported hyperparameters.
| Ablation | CD | F10 | MPJPE | Runtime | |
| random keyframe selection | 0.69 | 93.9 | 11.5 | 9.36 | 0.27 |
| single view from first frame | 0.73 | 92.1 | 18.2 | 22.73 | 0.27 |
| using all frames | 0.68 | 93.6 | 11.5 | 9.95 | 0.30 |
| w/o freezing shape | 0.45 | 98.8 | 11.4 | 10.81 | 0.28 |
| w/o temporal guidance ( ) | 0.45 | 98.8 | 11.5 | 10.60 | 0.27 |
| only foundation models (w/o Stage 3) | 0.45 | 98.8 | 12.1 | 20.62 | 0.10 |
Table 2. Ablation study. We ablate key components of our method on 4 sequences, 2 sequences from HO3Dv3 and 2 sequences from HOT3D. The first block ablates Stage 1 object reconstruction, the second block ablates Stage 2 SAM3D pose adaptation, and the third block ablates Stage 3 tracking objectives. For Stage 2 and Stage 3 variants, the same canonical object reconstructed with balanced keyframes is used, so CD and F10 remain unchanged. The “w/o freezing shape” row does not freeze the shape latent during SAM3D pose adaptation, but still uses the Stage 1 object during Stage 3 tracking. Runtime is normalized per 100 frames.
Figure 12. Human-to-robot tool manipulation from GraG object trajectories. Top: monocular RGB frames from a human short-screwdriver demonstration. Bottom: corresponding visualizations of the SimToolReal policy observations. The green screwdriver indicates the current goal pose obtained from GraG’s reconstructed 6D object trajectory (Zoom in for best view). The policy receives the robot state, estimated object pose, and goal pose; once the current object pose is sufficiently close to the goal, the goal advances to the next pose in the sequence. This visualization renders the policy inputs and goal-tracking behavior, rather than a physics-based real-robot execution.
| Setting | Number of Gaussians | Runtime (h) | Storage (MB) | Peak CUDA (GB) | CD | F10 | MPJPE | CD h |
|---|---|---|---|---|---|---|---|---|
| Sparse SoG | 0.27 | 0.11 | 2.4 | 0.45 | 98.8 | 11.3 | 7.65 | |
| Reduced | 1.20 | 3.20 | 5.5 | 0.44 | 99.1 | 11.3 | 7.50 | |
| Full | 34.1 | 16.0 | 6.2 | 0.44 | 99.1 | 11.3 | 7.50 |
Table 3. Trade-off between Gaussian count, compute, memory, and reconstruction quality on the ablation set. Runtime is normalized to hours per 100 frames as in the main paper setting. “Sparse SoG” is our default setting where we use farthest point sampling to subsample from the reduced SAM3D Gaussian asset and convert to isotropic Gaussians. “Reduced” denotes SAM3D’s lower-count Gaussian decoder output, trained for downstream needs such as decoding fewer Gaussians for faster inference (see SAM3D’s Supplementary Material for detailed information); “Full” denotes the dense SAM3D Gaussian decoder output. Storage reports the memory occupied by the object Gaussian parameter tensors, while Peak CUDA reports the maximum CUDA memory allocated during optimization. Thus, Storage measures representation compactness, whereas Peak CUDA captures practical optimization-time GPU memory.
| Dataset | Method | CDev [mm] | MDev [mm] | ACC h [m/s 2 ] | ACC o [m/s 2 ] | MRRPE h→o [mm] |
|---|---|---|---|---|---|---|
| HO3Dv3 | HOLD ( Fan et al., 2024 ) | 15.8 | 10.8 | 11.3 | 30.2 | 38.6 |
| BIGS ( On et al., 2025 ) | 54.2 | 23.1 | 14.4 | 37.8 | 79.4 | |
| MagicHOI ( Wang et al., 2025 ) | 533 | 21.5 | 15.4 | 10.3 | 747 | |
| Ours | 17.0 | 5.37 | 6.17 | 12.5 | 45.4 | |
| HOT3D | HOLD ( Fan et al., 2024 ) | 451 | 65.3 | 15.8 | 117 | 606 |
| BIGS ( On et al., 2025 ) | 570 | 85.6 | 24.1 | 136 | 762 |
Table 4. Interaction-consistency metrics. We report interaction metrics from ARCTIC ( Fan et al., 2023 ) on HO3Dv3 and HOT3D, including Contact Deviation (CDev), Motion Deviation (MDev), hand/object acceleration error (ACC h , ACC o ), and Mean Relative Root Position Error between the hand and object (MRRPE h→o ). Lower is better for all metrics.
| HO3Dv3 | HOT3D | |||
|---|---|---|---|---|
| Contact signal | F1 | Bal. Acc | F1 | Bal. Acc |
| Gemini 3 (VLM, ours) | 0.99 | 0.80 | 0.97 | 0.66 |
| Mask-proximity (fallback) | 0.96 | 0.70 | 0.98 | 0.59 |
Table 5. Contact-signal validation against mesh-proximity ground truth. Balanced accuracy (Bal. Acc) averages the in-contact and no-contact class accuracies; the VLM flag is the stronger signal on both datasets, most clearly on the rare no-contact frames.
| Projection | Overlap | HO3Dv3 | HOT3D | Runtime |
|---|---|---|---|---|
| Isotropic (ours) | Scalar closed-form | 0.27 | ||
| Anisotropic | Full matrix ( ) |
Table 6. Gaussian-projection ablation. Comparing our scalar closed-form overlap against full 2D anisotropic projection. Modeling Jacobian-based anisotropic projection with matrix overlap does not improve tracking accuracy ( ) and adds runtime.
| Object | Sequence name |
|---|---|
| bleach | ABF12 |
| bleach | ABF14 |
| potted meat | GPMF12 |
| potted meat | GPMF14 |
| cracker box | MC1 |
| cracker box | MC4 |
Table 7. HO3Dv3 sequences for hand-object reconstruction. We directly use the same sequences from HOLD ( Fan et al., 2024 ) . All of these sequences are only right-handed.
| Sequence name | Hand side | Start TS (ns) | End TS (ns) | Object ID |
|---|---|---|---|---|
| P0001_23fa0ee8 | right | 55003474858015 | 55015408224119 | 238686662724712 |
| P0001_8d136980 | right | 54039408208193 | 54044341521984 | 238686662724712 |
| P0001_b2bcbe28 | left | 54811408205972 | 54814874891157 | 258906041248094 |
| P0001_f71fc9b1 | right | 66482274870136 | 66485208197200 | 249541253457812 |
| P0010_8ff3e5c4 | left | 58695141597662 | 58697074869546 | 258906041248094 |
| P0011_0c2d00ed | right | 55563708197241 | 55572041548328 | 70709727230291 |
Table 8. HOT3D clips used in our evaluations, reported with sequence identifiers, hand side, temporal boundaries, and object IDs. Temporal boundaries are reported as HOT3D timecode timestamps (TS) in nanoseconds.
Figure 13. More qualitative results. We overlay rendered hand-object meshes on the input images for qualitative inspection and also show side views of the interactions.