Synthesizing believable 4D hand-object grasping sequences is important for character animation, extended reality, and embodied agents. Grasping requires a hand to approach, contact, and lift or manipulate an object coherently. Existing synthesis methods often model only the final static grasp or rely on additional conditions, such as language, target grasps, or object trajectories, to specify the interaction. We instead study the underconstrained problem of generating a complete grasping motion from a single initial hand-object state. Our framework decouples hand-motion generation from object-motion recovery. Given an object point cloud in the initial wrist frame and the initial parameterized hand configuration, conditional flow matching predicts a sequence-level contact map and a hand pose at each normalized sequence time. This time-indexed formulation avoids the error accumulation of autoregressive rollout while representing the complete motion. We then infer an attachment frame and recover rigid-object motion through training-free, contact-guided weighted registration. We evaluate on held-out DexYCB objects and unseen object geometries reconstructed from in-the-wild videos by Open-CHOIR. Compared with LatentHOI, our method produces smoother hand and object motion in every setting, matches or improves on its contact and penetration metrics in the wild under a shared initialization, and is about 30x faster per generated frame. In a 15-participant study, our motions were preferred in roughly two thirds of (forced-choice pairwise) comparisons. Precise contact closure on novel geometry, however, remains a key open challenge.
Figures & tables
Figure 2: Method overview. Given an object point cloud P0 and an initial hand state h0 , both expressed in the initial wrist frame, a contact flow samples a sequence-level contact-distance map c (Sec. 4.1 ). Conditioned on c , a pose flow generates the hand state h(t) independently at each queried motion time t , sharing one noise sample across the sequence (Sec. 4.2 ). A geometric stage then detects the attachment frame tatt and propagates the object rigidly with the hand, yielding transforms {Tk} , through contact-guided weighted registration (Sec. 4.3 ).
Contact
Penetration
Smoothness
Generation time
Dataset
Method
HO-Dist ↓
HO-Dist 778↓
IV ↓
PR ↓
ID ↓
Acc h↓
Acc o↓
Seq. ↓
Frame ↓
(cm)
(cm)
(cm 3 )
(%)
(mm)
(cm/frame 2 )
(cm/frame 2 )
(s)
(ms)
DexYCB
LatentHOI [ 16 ]
0.090
0.056
1.72
3.80
10.3
0.587
0.330
8.24
85.8
Ours (GT init.)
0.105
0.065
1.59
3.30
8.7
0.451
0.291
0.18
2.8
Ours (LatentHOI init.)
0.092
0.059
2.40
4.66
10.5
0.363
0.297
0.18
2.8
Open-CHOIR
LatentHOI [ 16 ]
0.145
0.085
1.70
3.22
8.3
0.615
0.364
8.24
85.8
Table 1: Quantitative comparison. Sequence-mean results on DexYCB unseen objects (200 sequences) and Open-CHOIR (30). LatentHOI and ours select one of ten samples using the attachment and equal-weight HO-Dist/IV ranking protocol. Single-sample timings use the same GPU: LatentHOI generates 96 frames with 1,000 DDPM steps; ours generates 64 with 32 Euler steps.
Figure 3: Results gallery. Generated motions for 20 scenes: input mesh, initial pose, approach, grasp, and post-grasp stages. Increasing opacity indicates temporal order; numbers identify frames. Insets show predicted contact maps on the rest-pose hand.
Figure 4: Comparison. Evaluation on eight DexYCB and Open-CHOIR scenes. DexGraspNet interpolates to a static grasp and holds the final poses; LatentHOI and ours generate motion. Opacity indicates temporal order; numbers identify frames. Insets show our contact maps.
Figure 5: Sampling multiple variations. Three generated samples for the same relative initial configuration for different objects. Solid poses show the grasp frame; translucent poses show preceding and subsequent motion. Insets show predicted contact maps.
Method
Naturalness / temporal coherence
Grasp plausibility / apparent stability
DexGraspNet
11.67%
10.42%
LatentHOI
22.08%
25.00%
Ours
66.25%
64.58%
Table 2: User study. Preferences of 15 participants over 16 scenes. Higher percentages indicate greater preference.
Metric ↓
Full model
GT map
w/o map
HO-Dist (cm)
0.105
0.102
0.108
HO-Dist 778 (cm)
0.065
0.062
0.071
IV (cm 3 )
1.59
1.85
1.79
PR (%)
3.30
3.66
3.60
ID (mm)
8.7
10.0
10.4
Acc h (cm/frame 2 )
0.451
0.450
0.397
Table 3: Ablations. Contact-conditioning ablation on DexYCB with GT initialization for all variants (sequence means; lower is better): sampled map (full model), ground-truth map (oracle), and no contact branch or conditioning (w/o map).
Figure 6: Limitations. Failures from incorrect trigger-frame selection (top), initial-pose sensitivity (middle), and thin or small objects (bottom). Rows show input, approach, trigger frame, and subsequent motion; insets show contact maps. See text for details.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Additional qualitative comparisons on eight scenes. Translucent hands indicate earlier frames, opaque hands later frames, and numbers identify frames. DexGraspNet interpolates to a static grasp and holds its final pose for display; LatentHOI and ours generate subsequent motion. Insets show our predicted contact maps.
Synthesizing realistic articulated hand-object interactions is a fundamental problem in virtual reality, embodied intelligence, and digital human applications. Existing methods for dexterous grasp synthesis typically regress or denoise poses in a joint space that couples global rigid motion with local articulation, which often yields unstable samples and physically implausible contacts. We introduce DEAL-Grasp, built upon the Decoupled Alignment (DEAL) representation, which reformulates grasp synthesis as alignment-space generation: the interaction state comprises task-space geometric anchors and articulation parameters, from which the rigid transform is recovered via closed-form Procrustes alignment while preserving local articulation. On this mixed state, we model grasp generation using heterogeneous-state flow matching with component-wise vector fields, incorporating time-adaptive physical regularization during training. At inference, grasps are synthesized solely by integrating the learned vector field, without test-time optimization or auxiliary physical guidance. Across MultiDex and zero-shot RealDex benchmarks, DEAL-Grasp attains high force-perturbation success rates alongside minimal penetration and high diversity of generated grasps, while substantially reducing native inference latency compared to optimization-heavy baselines. The project page is available at https://wmtlab.github.io/DEAL-Grasp/.
Generating 3D hand-object interactions is essential for applications in robotics, XR, and synthetic data generation, where flexible controllability and strong generalization to diverse object geometries are required. However, existing methods rarely satisfy these requirements, limiting their practical applicability. We present DCGrasp, a distance-aware controllable grasp generation system built on a novel grasp energy term. This term computes Distance Profile, a signed distance from each hand vertex to the nearest object point, coupled with distance-aware weighting, effectively capturing the semantically similar hand-object interaction in near-contact regions while remaining invariant to object and hand identity. Given various controllable signals, DCGrasp first generates a Distance Profile based on a Diffusion Transformer, together with a corresponding candidate hand pose. We then refine the candidate pose through optimization, enforcing consistency between the optimized hand pose and the generated Distance Profile in near-contact regions. Our experiments show that DCGrasp produces high-quality, physically plausible grasps with flexible user control, generalizing to diverse object and hand shapes and scales. Our work establishes a robust and versatile pipeline for the synthesis of controllable 3D hand-object interactions.
Hiroyasu Akada, Jesús Pérez, Emre Aksan +6
Google · Max Planck Institute for Informatics, SIC
Generating realistic 3D hand-object interactions (HOI) is a fundamental challenge in computer vision and robotics, requiring both temporal coherence and high-fidelity physical plausibility. Existing methods remain limited in their ability to learn expressive motion representations for generation and perform temporal reasoning. In this paper, we present HO-Flow, a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects. HO-Flow first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold by incorporating hand and object kinematics, enabling the representation to capture rich interaction dynamics. It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation, improving temporal coherence. To further enhance generalization, HO-Flow predicts object motions relative to the initial frame, enabling effective pre-training on large-scale synthetic data. Experiments on the GRAB, OakInk, and DexYCB benchmarks demonstrate that HO-Flow achieves state-of-the-art performance in both physical plausibility and motion diversity for interaction motion synthesis.