Learning Dynamic Grasping

Nga Kiu Kelly Ho; Yilin Liu; Hao Xu; Niloy J. Mitra

cs.ROOct 10, 2026

Learning Dynamic Grasping

Authors: Nga Kiu Kelly Ho, Yilin Liu, Hao Xu, Niloy J. Mitra

Organizations: University College London · The Chinese University of Hong Kong · Adobe Research

Abstract

Synthesizing believable 4D hand-object grasping sequences is important for character animation, extended reality, and embodied agents. Grasping requires a hand to approach, contact, and lift or manipulate an object coherently. Existing synthesis methods often model only the final static grasp or rely on additional conditions, such as language, target grasps, or object trajectories, to specify the interaction. We instead study the underconstrained problem of generating a complete grasping motion from a single initial hand-object state. Our framework decouples hand-motion generation from object-motion recovery. Given an object point cloud in the initial wrist frame and the initial parameterized hand configuration, conditional flow matching predicts a sequence-level contact map and a hand pose at each normalized sequence time. This time-indexed formulation avoids the error accumulation of autoregressive rollout while representing the complete motion. We then infer an attachment frame and recover rigid-object motion through training-free, contact-guided weighted registration. We evaluate on held-out DexYCB objects and unseen object geometries reconstructed from in-the-wild videos by Open-CHOIR. Compared with LatentHOI, our method produces smoother hand and object motion in every setting, matches or improves on its contact and penetration metrics in the wild under a shared initialization, and is about 30x faster per generated frame. In a 15-participant study, our motions were preferred in roughly two thirds of (forced-choice pairwise) comparisons. Precise contact closure on novel geometry, however, remains a key open challenge.

Figures & tables

Figure 2: Method overview. Given an object point cloud $\mathcal{P}_{0}$ and an initial hand state $\mathbf{h}_{0}$ , both expressed in the initial wrist frame, a contact flow samples a sequence-level contact-distance map $\widehat{\mathbf{c}}$ (Sec. 4.1 ). Conditioned on $\widehat{\mathbf{c}}$ , a pose flow generates the hand state $\widehat{\mathbf{h}}(t)$ independently at each queried motion time $t$ , sharing one noise sample across the sequence (Sec. 4.2 ). A geometric stage then detects the attachment frame $t_{\mathrm{att}}$ and propagates the object rigidly with the hand, yielding transforms $\{T_{k}\}$ , through contact-guided weighted registration (Sec. 4.3 ).

		Contact		Penetration			Smoothness		Generation time
Dataset	Method	HO-Dist $\downarrow$	HO-Dist ${}_{778}\downarrow$	IV $\downarrow$	PR $\downarrow$	ID $\downarrow$	Acc ${}_{h}\downarrow$	Acc ${}_{o}\downarrow$	Seq. $\downarrow$	Frame $\downarrow$
		(cm)	(cm)	(cm 3 )	(%)	(mm)	(cm/frame 2 )	(cm/frame 2 )	(s)	(ms)
DexYCB	LatentHOI [ 16 ]	0.090	0.056	1.72	3.80	10.3	0.587	0.330	8.24	85.8
	Ours (GT init.)	0.105	0.065	1.59	3.30	8.7	0.451	0.291	0.18	2.8
	Ours (LatentHOI init.)	0.092	0.059	2.40	4.66	10.5	0.363	0.297	0.18	2.8
Open-CHOIR	LatentHOI [ 16 ]	0.145	0.085	1.70	3.22	8.3	0.615	0.364	8.24	85.8

Table 1: Quantitative comparison. Sequence-mean results on DexYCB unseen objects (200 sequences) and Open-CHOIR (30). LatentHOI and ours select one of ten samples using the attachment and equal-weight HO-Dist/IV ranking protocol. Single-sample timings use the same GPU: LatentHOI generates 96 frames with 1,000 DDPM steps; ours generates 64 with 32 Euler steps.

Figure 3: Results gallery. Generated motions for 20 scenes: input mesh, initial pose, approach, grasp, and post-grasp stages. Increasing opacity indicates temporal order; numbers identify frames. Insets show predicted contact maps on the rest-pose hand.

Figure 4: Comparison. Evaluation on eight DexYCB and Open-CHOIR scenes. DexGraspNet interpolates to a static grasp and holds the final poses; LatentHOI and ours generate motion. Opacity indicates temporal order; numbers identify frames. Insets show our contact maps.

Figure 5: Sampling multiple variations. Three generated samples for the same relative initial configuration for different objects. Solid poses show the grasp frame; translucent poses show preceding and subsequent motion. Insets show predicted contact maps.

Method	Naturalness / temporal coherence	Grasp plausibility / apparent stability
DexGraspNet	11.67%	10.42%
LatentHOI	22.08%	25.00%
Ours	66.25%	64.58%

Table 2: User study. Preferences of 15 participants over 16 scenes. Higher percentages indicate greater preference.

Metric $\downarrow$	Full model	GT map	w/o map
HO-Dist (cm)	0.105	0.102	0.108
HO-Dist 778 (cm)	0.065	0.062	0.071
IV (cm 3 )	1.59	1.85	1.79
PR (%)	3.30	3.66	3.60
ID (mm)	8.7	10.0	10.4
Acc h (cm/frame 2 )	0.451	0.450	0.397

Table 3: Ablations. Contact-conditioning ablation on DexYCB with GT initialization for all variants (sequence means; lower is better): sampled map (full model), ground-truth map (oracle), and no contact branch or conditioning (w/o map).

Figure 6: Limitations. Failures from incorrect trigger-frame selection (top), initial-pose sensitivity (middle), and thin or small objects (bottom). Rows show input, approach, trigger frame, and subsequent motion; insets show contact maps. See text for details.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Additional qualitative comparisons on eight scenes. Translucent hands indicate earlier frames, opaque hands later frames, and numbers identify frames. DexGraspNet interpolates to a static grasp and holds its final pose for display; LatentHOI and ours generate subsequent motion. Insets show our predicted contact maps.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 80 papers

DEAL-Grasp: Decoupled Alignment Representation for Geometry-Aware Dexterous Grasp Generation
Sep 23, 2026Fuqiang Zhao, Qian LiuRobotic Grasping HOI Generation
Dalian University of Technology, China
DCGrasp: Distance-aware Controllable Grasp Generation
Jun 29, 2026Hiroyasu Akada, Jesús Pérez, Emre Aksan +6HOI Generation Robotic Grasping
Google · Max Planck Institute for Informatics, SIC
HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching
Apr 12, 2026Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen +3Style-Conditioned Motion Generation HOI Generation
Imperial College London · Inria, CNRS, ENS-PSL

Sep 23, 2026J/K move · Enter open · S save

DEAL-Grasp: Decoupled Alignment Representation for Geometry-Aware Dexterous Grasp Generation

Fuqiang Zhao, Qian Liu

Dalian University of Technology, China

Robotic Grasping HOI Generation

Open paper AskPDF arXiv

Synthesizing realistic articulated hand-object interactions is a fundamental problem in virtual reality, embodied intelligence, and digital human applications. Existing methods for dexterous grasp synthesis typically regress or denoise poses in a joint space that couples global rigid motion with local articulation, which often yields unstable samples and physically implausible contacts. We introduce DEAL-Grasp, built upon the Decoupled Alignment (DEAL) representation, which reformulates grasp synthesis as alignment-space generation: the interaction state comprises task-space geometric anchors and articulation parameters, from which the rigid transform is recovered via closed-form Procrustes alignment while preserving local articulation. On this mixed state, we model grasp generation using heterogeneous-state flow matching with component-wise vector fields, incorporating time-adaptive physical regularization during training. At inference, grasps are synthesized solely by integrating the learned vector field, without test-time optimization or auxiliary physical guidance. Across MultiDex and zero-shot RealDex benchmarks, DEAL-Grasp attains high force-perturbation success rates alongside minimal penetration and high diversity of generated grasps, while substantially reducing native inference latency compared to optimization-heavy baselines. The project page is available at https://wmtlab.github.io/DEAL-Grasp/.

Abstract

HOI Generation

Explore similar work

DEAL-Grasp: Decoupled Alignment Representation for Geometry-Aware Dexterous Grasp Generation

DCGrasp: Distance-aware Controllable Grasp Generation

HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching