FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation

Huajian Zeng; Lingyun Chen; Jiaqi Yang; Yuantai Zhang; Fan Shi; Peidong Liu; Xingxing Zuo

cs.ROFeb 13, 2026

FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation

Authors: Huajian Zeng, Lingyun Chen, Jiaqi Yang, Yuantai Zhang, Fan Shi, Peidong Liu, Xingxing Zuo

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE. · Technical University of Munich (TUM), Munich, Germany. · National University of Singapore (NUS), Singapore. · Westlake University, Hangzhou, China.

Abstract

Recent vision-language-action (VLA) models can generate plausible end-effector motions, yet they often fail in long-horizon, contact-rich tasks because the underlying hand-object interaction (HOI) structure is not explicitly represented. An embodiment-agnostic interaction representation that captures this structure would make manipulation behaviors easier to validate and transfer across robots. We propose FlowHOI, a two-stage flow-matching framework that generates semantically grounded, temporally coherent HOI sequences, comprising hand poses, object poses, and hand-object contact states, conditioned on an egocentric observation, a language instruction, and a 3D Gaussian splatting (3DGS) scene reconstruction. We decouple geometry-centric grasping from semantics-centric manipulation, conditioning the latter on compact 3D scene tokens and employing a motion-text alignment loss to semantically ground the generated interactions in both the physical scene layout and the language instruction. To address the scarcity of high-fidelity HOI supervision, we introduce a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale egocentric videos, yielding an HOI prior for robust generation. Across the GRAB and HOT3D benchmarks, FlowHOI achieves the highest action recognition accuracy and a 1.7 $\times$ higher physics simulation success rate than the strongest diffusion-based baseline, while delivering a 40 $\times$ inference speedup. We further demonstrate real-robot execution on four dexterous manipulation tasks, illustrating the feasibility of retargeting generated HOI representations to real-robot execution pipelines.

Figures & tables

Figure 1: We present a method for generating hand-object interaction (HOI) motions conditioned on egocentric observation, text command, and 3D scene context. We first learn a grasping prior with HOI data extracted from large-scale egocentric videos, and then generate semantically grounded manipulation motions that respect language instructions as well as the surrounding 3D scene context and geometric constraints. The generated motions can be retargeted to robot hands for real-world execution.

Figure 2: Overview of FlowHOI. Two-stage pipeline conditioned on an egocentric observation, text command, and 3D scene: (1) approach-and-grasp from a prior fine-tuned on reconstructed egocentric HOI data, and (2) manipulation conditioned on scene and language.

Figure 3: Hand-object data reconstruction. Detect the grasp-to-manipulation transition, reconstruct the object mesh via segmentation and metric depth, and align MANO under contact and non-penetration constraints. Detailed pipeline in supp.

Figure 4: Qualitative comparison vs. DiffH2O [ 18 ] and LatentHOI [ 19 ] (top: GRAB, bottom: HOT3D with 3D scene). Ours yields more natural grasps and physically plausible manipulations that better follow the instruction. Best viewed in the supplementary video.

		Physical Interaction Quality				Motion Quality			Realizable Physical Feasibility
Dataset	Method	IV [ $cm^{3}$ ] ( $\downarrow$ )	ID [ $cm$ ] ( $\downarrow$ )	CR [%] ( $\uparrow$ )	IVU ( $\downarrow$ )	AR ( $\uparrow$ )	SD [ $m$ ] ( $\uparrow$ )	OD [ $m$ ] ( $\uparrow$ )	Phy [%] ( $\uparrow$ )	SR [%] ( $\uparrow$ )	HT [s] ( $\uparrow$ )	Time [s] ( $\downarrow$ )
GRAB	Real Mocap	6.49	0.43	6.62	0.09	1.00	-	0.18	95.56	77.06	2.40	-
	DiffH2O	13.11	1.32	8.14	0.16	0.87	0.12	0.13	89.44	33.03	0.35	6.34
	LatentHOI	13.76	0.99	9.18	0.14	0.78	0.11	0.14	96.40	28.44	0.29	3.57
	Ours	10.93	1.28	6.85	0.13	0.95	0.13	0.16	90.90	55.96	1.50	0.16
HOT3D	Real Mocap	2.88	0.63	1.69	0.15	1.00	-	0.23	16.80	6.67	0.08	-

Table 1: Quantitative evaluation on GRAB and HOT3D. Best among generation methods in bold ; Real Mocap is the upper bound. IV/ID are bolded only when paired with non-trivial CR (cf. LatentHOI on HOT3D: low IV/ID with

0\%

physics SR; Supp. Sec. D ). Binomial SE on SR

\leq\!1.1\%

on GRAB and

\leq\!0.4\%

on HOT3D.

Method	Pouring	Drinking	Tilting	Squeezing	Avg.
DiffH2O [ 18 ]	2/10	0/10	2/10	0/10	10.0%
FlowHOI (Ours)	5/10	4/10	7/10	7/10	57.5%

Table 2: Real-world execution success rate (SR, out of 10 trials) on four contact-rich tasks. Aggregated over 40 trials, FlowHOI achieves

57.5\%

[

42.2\%,71.5\%

] vs. DiffH2O

10.0\%

[

4.0\%,23.0\%

] (

95\%

Wilson CI); the intervals do not overlap.

Figure 5: Real-world execution on four contact-rich tasks (pouring, drinking, tilting, squeezing): generated HOI retargeted to a Franka Panda + Allegro Hand and run open-loop.

Table 3: Ablation studies. (a) Scene encoder choices on HOT3D; FDE: final displacement error of object pose. (b) Effect of pretraining with large-scale egocentric HOI data on GRAB; GE: grasp error at the end of the grasping stage. (c) Text encoder choices.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Prediction Target	Jerk (Pos.) [ $m/s^{3}$ ] ( $\downarrow$ )	Jerk (Ang.) [ $rad/s^{3}$ ] ( $\downarrow$ )	Acc g (Pos.) [ $m/s^{2}$ ] ( $\downarrow$ )	Acc l (Pos.) [ $m/s^{2}$ ] ( $\downarrow$ )	Acc g (Rot.) [ $rad/s^{2}$ ] ( $\downarrow$ )	Acc l (Rot.) [ $rad/s^{2}$ ] ( $\downarrow$ )
$v$ -prediction	4.9721	137.9271	0.2523	0.1232	0.9567	3.2414
$x$ -prediction (Ours)	0.0438	0.4071	0.0204	0.0055	0.0835	0.0339

Appendix

Table 4: Ablation study on flow matching prediction targets on GRAB [ 26 ] .

x

-prediction yields lower angular jerk and hand acceleration while maintaining competitive interaction quality. Acc g /Acc l : global/local acceleration (see Sec. H ).

Figure 6: Qualitative comparison of prediction targets. Our $x$ -prediction produces temporally smooth and stable hand trajectories, while $v$ -prediction exhibits noticeable spatial jitter and inconsistent grasping poses across frames.

Figure 7: Detailed hand-object data reconstruction pipeline. Given an egocentric RGB video with tracked hand keypoints and camera parameters, we reconstruct HOI data in three steps. (1) Transition frame detection : we smooth wrist trajectories and identify the grasp-to-manipulation transition via local speed minima and direction change. (2) 3D object reconstruction : we segment the target object with SAM3 [ 79 ] , estimate metric depth with DepthAnything3 [ 80 ] , and reconstruct a mesh with SAM3D [ 81 ] using pre-transition frames where the object is static. (3) Hand-object alignment : we fit MANO hand meshes via inverse kinematics and optimize an object translation offset to satisfy fingerpad contact and non-penetration constraints at the transition frame, then propagate the alignment to all frames.

Metric	Mean	Median	Std
Chamfer distance	9.8 mm	6.5 mm	21.8 mm
F-score @ 5 cm	96.5%	100.0%	10.3%
Position error	5.1 cm	3.6 cm	6.4 cm
Mask IoU	0.70	0.78	0.24

Appendix

Table 5: Geometric and pose accuracy of our EgoDex reconstruction pipeline measured on HOT3D’s ground-truth meshes and 6-DoF poses.

Grasping prior	GE [ $m$ ] ( $\downarrow$ )	TV [ $m/s$ ] ( $\downarrow$ )	HA [%] ( $\uparrow$ )	Phy [%] ( $\uparrow$ )	SR [%] ( $\uparrow$ )	RG (of 10) ( $\uparrow$ )
No EgoDex pretraining	0.10	0.583	96.34	91.06	51.38	6
EgoDex-pretrained (ours)	0.06	0.340	99.02	90.90	55.96	6

Appendix

Table 6: Effect of EgoDex pretraining. Only the grasp prefix differs. RG is measured on the real robot in a separate set of 10 squeezing trials, and all other metrics on GRAB.

Figure 8: Comparison of HOI trajectories: (a) kinematic human hand trajectory, (b) retargeted Allegro Hand trajectory, and (c) physics simulation in Isaac Gym [ 100 ] . Although the kinematic trajectories in (a) and (b) appear plausible, executing them under physics simulation reveals failures: the object detaches and slips from the grasp (bottom row, red circle), indicating that kinematically feasible motions do not guarantee physically stable interactions.

Figure 9: Scene feature analysis. Top-down views of two scenes comparing geometric features (Concerto [ 88 ] , left) and semantic features (SceneSplat [ 89 ] , right), colored by PCA of the respective feature spaces. Black dashed circles highlight where geometric features successfully distinguish objects from the supporting surface (bottom-left) and where semantic features successfully discriminate semantically distinct regions (top-right). This complementarity motivates our bidirectional fusion.

Scene conditioning	IV [ $cm^{3}$ ] ( $\downarrow$ )	CR [%] ( $\uparrow$ )	Phy [%] ( $\uparrow$ )	GE [ $m$ ] ( $\downarrow$ )	HA ( $\uparrow$ )	PD [ $mm$ ] ( $\downarrow$ )
None	3.26	2.27	16.47	0.20	0.83	2.08
Global only	3.30	2.09	16.63	0.14	0.88	1.97
Local only	3.22	2.33	16.10	0.19	0.85	1.98
Global + local (ours)	3.36	2.14	17.15	0.11	0.92	1.95

Appendix

Table 7: Scene-conditioning granularity on HOT3D. GE: grasp error at the transition frame. HA: handedness accuracy. PD: hand-scene penetration depth.

Metric	Source	Low ( $n{=}120$ )	Medium ( $n{=}120$ )	High ( $n{=}120$ )
Penetration depth [ $mm$ ] ( $\downarrow$ )	FlowHOI	1.97	1.94	2.04
Penetration depth [ $mm$ ] ( $\downarrow$ )	Ground truth	1.94	1.94	2.02
GT-tolerance success rate [%] ( $\uparrow$ )	FlowHOI	70.0	80.8	85.8
GT-tolerance success rate [%] ( $\uparrow$ )	Ground truth	76.7	85.0	86.7

Appendix

Table 8: Hand-scene penetration against the HOT3D MPS static point cloud, stratified by scene clutter.

Setting	$\mathrm{Jerk}_{\mathrm{pos}}$ ( $\downarrow$ )	$\mathrm{Jerk}_{\mathrm{ang}}$ ( $\downarrow$ )	$\mathrm{Acc}_{g}^{\mathrm{pos}}$ ( $\downarrow$ )	GE [ $m$ ] ( $\downarrow$ )
w/o inpainting	$0.341$	$3.218$	$0.157$	$0.14$
w/ inpainting	$\mathbf{0.044}$	$\mathbf{0.407}$	$\mathbf{0.020}$	$\mathbf{0.06}$

Appendix

Table 9: Effect of stage-transition soft inpainting on GRAB. Inpainting suppresses high-frequency artifacts at the grasp/manipulation boundary by an order of magnitude.

Dataset	Model	IV [ $cm^{3}$ ] ( $\downarrow$ )	CR [%] ( $\uparrow$ )	Phy [%] ( $\uparrow$ )	AR ( $\uparrow$ )	SR [%] ( $\uparrow$ )	Time [s] ( $\downarrow$ )
GRAB	Single-stage	11.94	6.75	89.83	0.93	54.12	0.12
GRAB	Two-stage (ours)	10.93	6.85	90.90	0.95	55.96	0.16
HOT3D	Single-stage	3.53	1.70	16.42	0.80	6.67	0.12
HOT3D	Two-stage (ours)	3.36	2.14	17.15	0.78	5.33	0.16

Appendix

Table 10: Single-stage versus two-stage generation. Both variants share the representation, training data, objective, inference steps, parameter budget, and EgoDex pretraining.

Method	RRE [deg] ( $\downarrow$ )	CT-F1 ( $\uparrow$ )
DiffH2O [ 18 ]	83.0	0.051
FlowHOI (ours)	46.4	0.079

Appendix

Table 11: Post-grasp dexterity on the 174-instance GRAB reorientation subset.

Figure 10: Qualitative comparison on GRAB. Comparison with DiffH2O [ 18 ] and LatentHOI [ 19 ] on diverse actions. Our method produces more physically plausible hand-object interactions with accurate grasping poses and smooth bimanual coordination.

Figure 11: Additional qualitative results on HOT3D with scene context. The generated hand trajectories adapt to the surrounding scene layout, correctly interacting with objects on shelves and tables.

Figure 12: Representative failure cases. Red dashed circles highlight the erroneous regions. (a) Object size mismatch : the small pyramid floats away from the hand due to a mismatch between object size and generated grasp aperture. (b) Incorrect contact establishment : the hand fails to properly wrap around the wooden spoon, resulting in physically implausible finger-object contact. Left: GRAB [ 26 ] , right: HOT3D [ 29 ] with scene context.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 84 papers

HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching
Apr 12, 2026Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen +3Style-Conditioned Motion Generation HOI Generation
Imperial College London · Inria, CNRS, ENS-PSL
MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow
Oct 12, 2026Qiang He, Yonghao Zhang, Yinda Zhang +7HOI Generation One-Step Generative Modeling
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google
PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph
Aug 3, 2026Zhenhao Zhang, Jiajun Zhang, Wei Min +1HOI Generation Dexterous Manipulation
Tsinghua University · Shadow AI

Apr 12, 2026J/K move · Enter open · S save

HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching

Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, Jiankang Deng+2

Imperial College London · Inria, CNRS, ENS-PSL

Style-Conditioned Motion Generation HOI Generation Human Motion Generation Motion Representation Learning

Open paper AskPDF arXiv

Generating realistic 3D hand-object interactions (HOI) is a fundamental challenge in computer vision and robotics, requiring both temporal coherence and high-fidelity physical plausibility. Existing methods remain limited in their ability to learn expressive motion representations for generation and perform temporal reasoning. In this paper, we present HO-Flow, a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects. HO-Flow first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold by incorporating hand and object kinematics, enabling the representation to capture rich interaction dynamics. It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation, improving temporal coherence. To further enhance generalization, HO-Flow predicts object motions relative to the initial frame, enabling effective pre-training on large-scale synthetic data. Experiments on the GRAB, OakInk, and DexYCB benchmarks demonstrate that HO-Flow achieves state-of-the-art performance in both physical plausibility and motion diversity for interaction motion synthesis.

Abstract

HOI Generation

Explore similar work

HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow

PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph