Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE. · Technical University of Munich (TUM), Munich, Germany. · National University of Singapore (NUS), Singapore. · Westlake University, Hangzhou, China.
Recent vision-language-action (VLA) models can generate plausible end-effector motions, yet they often fail in long-horizon, contact-rich tasks because the underlying hand-object interaction (HOI) structure is not explicitly represented. An embodiment-agnostic interaction representation that captures this structure would make manipulation behaviors easier to validate and transfer across robots. We propose FlowHOI, a two-stage flow-matching framework that generates semantically grounded, temporally coherent HOI sequences, comprising hand poses, object poses, and hand-object contact states, conditioned on an egocentric observation, a language instruction, and a 3D Gaussian splatting (3DGS) scene reconstruction. We decouple geometry-centric grasping from semantics-centric manipulation, conditioning the latter on compact 3D scene tokens and employing a motion-text alignment loss to semantically ground the generated interactions in both the physical scene layout and the language instruction. To address the scarcity of high-fidelity HOI supervision, we introduce a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale egocentric videos, yielding an HOI prior for robust generation. Across the GRAB and HOT3D benchmarks, FlowHOI achieves the highest action recognition accuracy and a 1.7× higher physics simulation success rate than the strongest diffusion-based baseline, while delivering a 40× inference speedup. We further demonstrate real-robot execution on four dexterous manipulation tasks, illustrating the feasibility of retargeting generated HOI representations to real-robot execution pipelines.
Figures & tables
Figure 1: We present a method for generating hand-object interaction (HOI) motions conditioned on egocentric observation, text command, and 3D scene context. We first learn a grasping prior with HOI data extracted from large-scale egocentric videos, and then generate semantically grounded manipulation motions that respect language instructions as well as the surrounding 3D scene context and geometric constraints. The generated motions can be retargeted to robot hands for real-world execution.
Figure 2: Overview of FlowHOI. Two-stage pipeline conditioned on an egocentric observation, text command, and 3D scene: (1) approach-and-grasp from a prior fine-tuned on reconstructed egocentric HOI data, and (2) manipulation conditioned on scene and language.
Figure 3: Hand-object data reconstruction. Detect the grasp-to-manipulation transition, reconstruct the object mesh via segmentation and metric depth, and align MANO under contact and non-penetration constraints. Detailed pipeline in supp.
Figure 4: Qualitative comparison vs. DiffH2O [ 18 ] and LatentHOI [ 19 ] (top: GRAB, bottom: HOT3D with 3D scene). Ours yields more natural grasps and physically plausible manipulations that better follow the instruction. Best viewed in the supplementary video.
Physical Interaction Quality
Motion Quality
Realizable Physical Feasibility
Dataset
Method
IV [ cm3 ] ( ↓ )
ID [ cm ] ( ↓ )
CR [%] ( ↑ )
IVU ( ↓ )
AR ( ↑ )
SD [ m ] ( ↑ )
OD [ m ] ( ↑ )
Phy [%] ( ↑ )
SR [%] ( ↑ )
HT [s] ( ↑ )
Time [s] ( ↓ )
GRAB
Real Mocap
6.49
0.43
6.62
0.09
1.00
-
0.18
95.56
77.06
2.40
-
DiffH2O
13.11
1.32
8.14
0.16
0.87
0.12
0.13
89.44
33.03
0.35
6.34
LatentHOI
13.76
0.99
9.18
0.14
0.78
0.11
0.14
96.40
28.44
0.29
3.57
Ours
10.93
1.28
6.85
0.13
0.95
0.13
0.16
90.90
55.96
1.50
0.16
HOT3D
Real Mocap
2.88
0.63
1.69
0.15
1.00
-
0.23
16.80
6.67
0.08
-
Table 1: Quantitative evaluation on GRAB and HOT3D. Best among generation methods in bold ; Real Mocap is the upper bound. IV/ID are bolded only when paired with non-trivial CR (cf. LatentHOI on HOT3D: low IV/ID with 0% physics SR; Supp. Sec. D ). Binomial SE on SR ≤1.1% on GRAB and ≤0.4% on HOT3D.
Method
Pouring
Drinking
Tilting
Squeezing
Avg.
DiffH2O [ 18 ]
2/10
0/10
2/10
0/10
10.0%
FlowHOI (Ours)
5/10
4/10
7/10
7/10
57.5%
Table 2: Real-world execution success rate (SR, out of 10 trials) on four contact-rich tasks. Aggregated over 40 trials, FlowHOI achieves 57.5% [ 42.2%,71.5% ] vs. DiffH2O 10.0% [ 4.0%,23.0% ] ( 95% Wilson CI); the intervals do not overlap.
Figure 5: Real-world execution on four contact-rich tasks (pouring, drinking, tilting, squeezing): generated HOI retargeted to a Franka Panda + Allegro Hand and run open-loop.
Table 3: Ablation studies. (a) Scene encoder choices on HOT3D; FDE: final displacement error of object pose. (b) Effect of pretraining with large-scale egocentric HOI data on GRAB; GE: grasp error at the end of the grasping stage. (c) Text encoder choices.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Prediction Target
Jerk (Pos.) [ m/s3 ] ( ↓ )
Jerk (Ang.) [ rad/s3 ] ( ↓ )
Acc g (Pos.) [ m/s2 ] ( ↓ )
Acc l (Pos.) [ m/s2 ] ( ↓ )
Acc g (Rot.) [ rad/s2 ] ( ↓ )
Acc l (Rot.) [ rad/s2 ] ( ↓ )
v -prediction
4.9721
137.9271
0.2523
0.1232
0.9567
3.2414
x -prediction (Ours)
0.0438
0.4071
0.0204
0.0055
0.0835
0.0339
Appendix
Table 4: Ablation study on flow matching prediction targets on GRAB [ 26 ] . x -prediction yields lower angular jerk and hand acceleration while maintaining competitive interaction quality. Acc g /Acc l : global/local acceleration (see Sec. H ).
Figure 6: Qualitative comparison of prediction targets. Our x -prediction produces temporally smooth and stable hand trajectories, while v -prediction exhibits noticeable spatial jitter and inconsistent grasping poses across frames.
Figure 7: Detailed hand-object data reconstruction pipeline. Given an egocentric RGB video with tracked hand keypoints and camera parameters, we reconstruct HOI data in three steps. (1) Transition frame detection : we smooth wrist trajectories and identify the grasp-to-manipulation transition via local speed minima and direction change. (2) 3D object reconstruction : we segment the target object with SAM3 [ 79 ] , estimate metric depth with DepthAnything3 [ 80 ] , and reconstruct a mesh with SAM3D [ 81 ] using pre-transition frames where the object is static. (3) Hand-object alignment : we fit MANO hand meshes via inverse kinematics and optimize an object translation offset to satisfy fingerpad contact and non-penetration constraints at the transition frame, then propagate the alignment to all frames.
Metric
Mean
Median
Std
Chamfer distance
9.8 mm
6.5 mm
21.8 mm
F-score @ 5 cm
96.5%
100.0%
10.3%
Position error
5.1 cm
3.6 cm
6.4 cm
Mask IoU
0.70
0.78
0.24
Appendix
Table 5: Geometric and pose accuracy of our EgoDex reconstruction pipeline measured on HOT3D’s ground-truth meshes and 6-DoF poses.
Grasping prior
GE [ m ] ( ↓ )
TV [ m/s ] ( ↓ )
HA [%] ( ↑ )
Phy [%] ( ↑ )
SR [%] ( ↑ )
RG (of 10) ( ↑ )
No EgoDex pretraining
0.10
0.583
96.34
91.06
51.38
6
EgoDex-pretrained (ours)
0.06
0.340
99.02
90.90
55.96
6
Appendix
Table 6: Effect of EgoDex pretraining. Only the grasp prefix differs. RG is measured on the real robot in a separate set of 10 squeezing trials, and all other metrics on GRAB.
Figure 8: Comparison of HOI trajectories: (a) kinematic human hand trajectory, (b) retargeted Allegro Hand trajectory, and (c) physics simulation in Isaac Gym [ 100 ] . Although the kinematic trajectories in (a) and (b) appear plausible, executing them under physics simulation reveals failures: the object detaches and slips from the grasp (bottom row, red circle), indicating that kinematically feasible motions do not guarantee physically stable interactions.
Figure 9: Scene feature analysis. Top-down views of two scenes comparing geometric features (Concerto [ 88 ] , left) and semantic features (SceneSplat [ 89 ] , right), colored by PCA of the respective feature spaces. Black dashed circles highlight where geometric features successfully distinguish objects from the supporting surface (bottom-left) and where semantic features successfully discriminate semantically distinct regions (top-right). This complementarity motivates our bidirectional fusion.
Scene conditioning
IV [ cm3 ] ( ↓ )
CR [%] ( ↑ )
Phy [%] ( ↑ )
GE [ m ] ( ↓ )
HA ( ↑ )
PD [ mm ] ( ↓ )
None
3.26
2.27
16.47
0.20
0.83
2.08
Global only
3.30
2.09
16.63
0.14
0.88
1.97
Local only
3.22
2.33
16.10
0.19
0.85
1.98
Global + local (ours)
3.36
2.14
17.15
0.11
0.92
1.95
Appendix
Table 7: Scene-conditioning granularity on HOT3D. GE: grasp error at the transition frame. HA: handedness accuracy. PD: hand-scene penetration depth.
Metric
Source
Low ( n=120 )
Medium ( n=120 )
High ( n=120 )
Penetration depth [ mm ] ( ↓ )
FlowHOI
1.97
1.94
2.04
Ground truth
1.94
1.94
2.02
GT-tolerance success rate [%] ( ↑ )
FlowHOI
70.0
80.8
85.8
Ground truth
76.7
85.0
86.7
Appendix
Table 8: Hand-scene penetration against the HOT3D MPS static point cloud, stratified by scene clutter.
Setting
Jerkpos ( ↓ )
Jerkang ( ↓ )
Accgpos ( ↓ )
GE [ m ] ( ↓ )
w/o inpainting
0.341
3.218
0.157
0.14
w/ inpainting
0.044
0.407
0.020
0.06
Appendix
Table 9: Effect of stage-transition soft inpainting on GRAB. Inpainting suppresses high-frequency artifacts at the grasp/manipulation boundary by an order of magnitude.
Dataset
Model
IV [ cm3 ] ( ↓ )
CR [%] ( ↑ )
Phy [%] ( ↑ )
AR ( ↑ )
SR [%] ( ↑ )
Time [s] ( ↓ )
GRAB
Single-stage
11.94
6.75
89.83
0.93
54.12
0.12
Two-stage (ours)
10.93
6.85
90.90
0.95
55.96
0.16
HOT3D
Single-stage
3.53
1.70
16.42
0.80
6.67
0.12
Two-stage (ours)
3.36
2.14
17.15
0.78
5.33
0.16
Appendix
Table 10: Single-stage versus two-stage generation. Both variants share the representation, training data, objective, inference steps, parameter budget, and EgoDex pretraining.
Method
RRE [deg] ( ↓ )
CT-F1 ( ↑ )
DiffH2O [ 18 ]
83.0
0.051
FlowHOI (ours)
46.4
0.079
Appendix
Table 11: Post-grasp dexterity on the 174-instance GRAB reorientation subset.
Figure 10: Qualitative comparison on GRAB. Comparison with DiffH2O [ 18 ] and LatentHOI [ 19 ] on diverse actions. Our method produces more physically plausible hand-object interactions with accurate grasping poses and smooth bimanual coordination.
Figure 11: Additional qualitative results on HOT3D with scene context. The generated hand trajectories adapt to the surrounding scene layout, correctly interacting with objects on shelves and tables.
Figure 12: Representative failure cases. Red dashed circles highlight the erroneous regions. (a) Object size mismatch : the small pyramid floats away from the hand due to a mismatch between object size and generated grasp aperture. (b) Incorrect contact establishment : the hand fails to properly wrap around the wooden spoon, resulting in physically implausible finger-object contact. Left: GRAB [ 26 ] , right: HOT3D [ 29 ] with scene context.
Generating realistic 3D hand-object interactions (HOI) is a fundamental challenge in computer vision and robotics, requiring both temporal coherence and high-fidelity physical plausibility. Existing methods remain limited in their ability to learn expressive motion representations for generation and perform temporal reasoning. In this paper, we present HO-Flow, a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects. HO-Flow first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold by incorporating hand and object kinematics, enabling the representation to capture rich interaction dynamics. It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation, improving temporal coherence. To further enhance generalization, HO-Flow predicts object motions relative to the initial frame, enabling effective pre-training on large-scale synthetic data. Experiments on the GRAB, OakInk, and DexYCB benchmarks demonstrate that HO-Flow achieves state-of-the-art performance in both physical plausibility and motion diversity for interaction motion synthesis.
Whole-body human-object interaction (HOI) generation requires coordinating body and finger motion, object trajectories, and time-varying contacts. Existing methods typically rely on iterative diffusion or test-time optimization, resulting in high inference costs. Direct one-step generation is highly desired in downstream applications due to efficient inference. However, it is challenging because these HOI components differ substantially in structure and scale. We present MFlowHOI, an efficient one-step framework for whole-body HOI generation. A Contact-Aware Multipart RVQVAE organizes body, hand, object, and contact signals into a structured latent space, with a shared decoder preserving their global coordination. An ST-DiT then models temporal and cross-part dependencies in this space and generates a complete interaction in a single MeanFlow evaluation. Predicted object-local contact correspondences further enable geometric refinement and interaction-preserving transfer to photorealistic 3D Gaussian avatars with different body proportions. Experiments demonstrate that MFlowHOI outperforms existing base generators in motion quality and text alignment while substantially reducing inference time.
Qiang He, Yonghao Zhang, Yinda Zhang +7
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google
Hand-object interaction (HOI) is a fundamental human behavior with broad applications in AR/VR, digital humans, and embodied interaction. Existing methods typically require predefined object geometry, object trajectories, or task-specific conditions, limiting their use with natural real-world inputs. To address this, we study a more practical problem of synthesizing 3D hand-object interaction sequences from a single RGB photograph and an open-vocabulary language instruction, and introduce PhotoHOI. PhotoHOI first uses a vision-language model to parse the input image and instruction into a structured task specification, including the interaction object, target region, and spatial relation. It then recovers a compact task-relevant 3D scene and plans a smooth collision-aware object trajectory based on the recovered object states, support relations, and surrounding scene geometry. To synthesize hand motion that generalizes to real-world photographs and unseen objects, it learns transferable task-conditioned contact and contact-conditioned grasp priors from large-scale affordance and HOI data. The grasp is further refined in a learned latent space, constraining the optimization to a plausible hand-pose manifold. Experiments on GRAB and H2O demonstrate improved contact quality and reduced penetration over representative baselines. Results on real-world photographs further demonstrate higher task success and scene consistency, together with generalization to unseen objects and open-vocabulary instructions.