HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching

Zerui Chen; Rolandos Alexandros Potamias; Shizhe Chen; Jiankang Deng; Cordelia Schmid; Stefanos Zafeiriou

cs.CVApr 12, 2026

HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching

Authors: Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, Jiankang Deng, Cordelia Schmid, Stefanos Zafeiriou

Organizations: Imperial College London · Inria, CNRS, ENS-PSL

Abstract

Generating realistic 3D hand-object interactions (HOI) is a fundamental challenge in computer vision and robotics, requiring both temporal coherence and high-fidelity physical plausibility. Existing methods remain limited in their ability to learn expressive motion representations for generation and perform temporal reasoning. In this paper, we present HO-Flow, a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects. HO-Flow first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold by incorporating hand and object kinematics, enabling the representation to capture rich interaction dynamics. It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation, improving temporal coherence. To further enhance generalization, HO-Flow predicts object motions relative to the initial frame, enabling effective pre-training on large-scale synthetic data. Experiments on the GRAB, OakInk, and DexYCB benchmarks demonstrate that HO-Flow achieves state-of-the-art performance in both physical plausibility and motion diversity for interaction motion synthesis.

Figures & tables

Figure 1: Our approach can synthesize realistic hand–object interactions for diverse tasks (left), where darker colors represent later time steps. It outperforms the state-of-the-art LatentHOI [ 21 ] across three benchmarks (right), namely GRAB [ 42 ] , DexYCB [ 5 ] and OakInk [ 51 ] . SD and Phy refer to sample diversity and physical plausibility.

Figure 2: Overview of HO-Flow. Red and gray blocks indicate encoded and noisy motion latents, respectively. Red arrows are active only during training. The framework synthesizes realistic hand-object interactions using two components: an interaction-aware VAE for compact motion latents with fine-grained interaction features, and an auto-regressive flow-matching model that predicts successive latents for faithful, temporally coherent synthesis.

Figure 3: Interaction-aware VAE captures fine-grained interaction features for latent representations ( $\mathbf{z}_{o}$ and $\mathbf{z}_{h}$ ). $\mathbf{T}_{h}$ is the transformation of hand bones.

Figure 4: Illustration of our flow matching model. Gray and pink blocks indicate masked and unmasked tokens, respectively. Conditioned on object point clouds and task descriptions, the model aggregates motion context to autoregressively predict successive motions, thereby reducing uncertainty and enabling high-fidelity interaction synthesis.

	Object kine.	Hand kine.	Sep. latents	Pre- train	$\rm E_{v}^{r}\downarrow$	$\rm E_{j}^{r}\downarrow$	$\rm E_{v}^{l}\downarrow$	$\rm E_{j}^{l}\downarrow$	$\rm E_{o}\downarrow$	$\rm CD_{o}\downarrow$
R1	$\times$	$\times$	$\times$	$\times$	23.93	24.34	19.14	19.55	16.34	1.36
R2	✓	$\times$	$\times$	$\times$	22.15	22.56	20.51	20.58	8.93	0.71
R3	✓	✓	$\times$	$\times$	21.69	21.97	16.59	16.90	5.19	0.37
R4	✓	✓	✓	$\times$	17.29	17.61	13.05	13.33	3.70	0.26
R5	✓	✓	✓	✓	9.13	9.41	8.03	8.31	2.33	0.12

Table 1: Ablation experiments for interaction-aware autoencoder on GRAB dataset. Our proposed model that captures both spatial and temporal interaction features yields the best performance in encoding hand and object motions.

	Latent	Seq.	Auto- reg.	Pre- train	$\rm IV_{r}\downarrow$	$\rm IV_{l}\downarrow$	$\rm ID_{r}\downarrow$	$\rm ID_{l}\downarrow$	$\rm CR_{r}\uparrow$	$\rm CR_{l}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
	Representation		Auto- reg.	Pre- train	$\rm IV_{r}\downarrow$	$\rm IV_{l}\downarrow$	$\rm ID_{r}\downarrow$	$\rm ID_{l}\downarrow$	$\rm CR_{r}\uparrow$	$\rm CR_{l}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
R1	$\times$	✓	$\times$	$\times$	10.35	2.67	1.14	0.48	11.68	3.73	0.13	84.89	0.24
R2	✓	$\times$	$\times$	$\times$	8.13	1.94	0.98	0.43	12.54	0.71	0.13	88.76	0.12
R3	✓	✓	$\times$	$\times$	6.99	1.82	0.82	0.36	10.27	3.07	0.11	92.69	0.26
R4	✓	✓	✓	$\times$	5.48	1.35	0.63	0.30	11.26	2.86	0.08	97.94	0.30
R5	✓	✓	✓	✓	5.31	1.24	0.61	0.26	11.38	3.96	0.07	98.25	0.31

Table 2: Ablation experiments for the auto-regressive flow matching model on GRAB dataset. Our approach achieves the best performance over other variants. Seq. indicates whether the model takes a temporal sequence as input.

Methods	$\rm IV_{r}\downarrow$	$\rm IV_{l}\downarrow$	$\rm ID_{r}\downarrow$	$\rm ID_{l}\downarrow$	$\rm CR_{r}\uparrow$	$\rm CR_{l}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
IMoS [ 11 ]	10.38	-	1.25	-	4.61	-	0.53	84.88	0.00
MDM [ 43 ]	9.12	2.61	1.24	0.51	8.21	1.29	0.19	89.81	0.18
MLD [ 6 ]	9.62	3.14	1.06	0.49	10.23	0.87	0.24	85.68	0.20
LatentHOI [ 21 ]	6.38	1.66	0.77	0.29	11.94	1.11	0.10	96.16	0.13
HO-Flow, wo pt	5.48	1.35	0.63	0.30	11.26	2.86	0.08	97.94	0.30
HO-Flow, w pt	5.31	1.24	0.61	0.26	11.38	3.96	0.07	98.25	0.31

Table 3: Comparison with state-of-the-art methods on GRAB benchmark. HO-Flow demonstrates strong generalization ability to faithfully interact with diverse objects. pt denotes whether this model has been pre-trained on GraspXL data.

Methods	$\rm IV_{r}\downarrow$	$\rm IV_{l}\downarrow$	$\rm ID_{r}\downarrow$	$\rm ID_{l}\downarrow$	$\rm CR_{r}\uparrow$	$\rm CR_{l}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
Text2HOI [ 4 ]	15.19	11.54	2.14	1.39	11.24	6.33	0.26	82.58	0.21
MDM [ 43 ]	8.46	2.47	1.69	0.34	4.97	1.02	0.20	60.89	0.23
MLD [ 6 ]	9.15	4.25	1.79	0.56	5.36	0.77	0.29	46.41	0.32
LatentHOI [ 21 ]	7.22	3.11	1.10	0.37	7.80	1.73	0.14	71.24	0.22
HO-Flow, wo pt	5.82	2.05	0.66	0.30	8.76	2.34	0.11	83.62	0.31
HO-Flow, w pt	4.10	1.72	0.45	0.26	8.85	3.78	0.09	89.76	0.33

Table 4: Comparison with state-of-the-art methods on OakInk benchmark. HO-Flow achieves superior performance in terms of both motion quality and diversity. pt denotes whether this model has been pre-trained on GraspXL data.

Figure 5: Qualitative comparison with the state-of-the-art LatentHOI approach [ 21 ] . Our HO-Flow model can generate more realistic hand-object interactions with more natural contacts and significantly less penetrations.

Methods	$\rm IV\downarrow$	$\rm ID\downarrow$	$\rm CR_{r}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
MDM [ 43 ]	7.78	2.10	8.87	0.12	86.22	0.13
LatentHOI [ 21 ]	7.70	2.01	11.98	0.13	88.52	0.13
HO-Flow, wo pt	6.84	1.82	12.06	0.11	90.77	0.20
HO-Flow, w pt	6.37	1.20	13.83	0.10	95.41	0.20

Table 5: Comparison with state-of-the-art methods on DexYCB benchmark. HO-Flow also shows strong performance for synthesizing single-hand manipulation sequences. pt denotes whether this model has been pre-trained on GraspXL data.

Methods	GRAB	OakInk	DexYCB
LatentHOI [ 21 ]	0.37	0.28	0.38
HO-Flow (Ours)	0.63	0.72	0.62

Table 6: Qualitative user study comparing HO-Flow with LatentHOI [ 21 ] across three benchmarks. Values indicate the preference ratio (higher is better).

Figure 6: Qualitative results of HO-Flow on OakInk and DexYCB benchmarks. Our approach can faithfully synthesize hand and object motions with natural interactions.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Algorithm 1 Inference procedure of proposed HO-Flow model

Approach	$\rm IV_{r}\downarrow$	$\rm IV_{l}\downarrow$	$\rm ID_{r}\downarrow$	$\rm ID_{l}\downarrow$	$\rm CR_{r}\uparrow$	$\rm CR_{l}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
DDPM [ 13 ]	6.56	2.09	0.78	0.31	10.53	3.80	0.09	96.00	0.30
Flow Matching [ 22 ]	5.31	1.24	0.61	0.26	11.38	3.96	0.07	98.25	0.31

Appendix

Table 7: Ablation study of using either diffusion or flow matching approaches to train HO-Flow on GRAB dataset.

Steps	Latency	$\rm IV_{r}\downarrow$	$\rm IV_{l}\downarrow$	$\rm ID_{r}\downarrow$	$\rm ID_{l}\downarrow$	$\rm CR_{r}\uparrow$	$\rm CR_{l}\uparrow$	$\rm IVU\downarrow$	$\rm Phy\uparrow$	$\rm SD\uparrow$
6	2.68	8.51	2.26	0.77	0.34	9.10	3.29	0.13	90.63	0.29
12	4.77	7.48	1.35	0.93	0.35	11.26	2.86	0.10	94.94	0.30
18	6.89	5.31	1.24	0.61	0.26	11.38	3.96	0.07	98.25	0.31
24	9.26	5.48	1.22	0.61	0.31	11.42	3.54	0.08	98.29	0.31

Appendix

Table 8: Ablation study of prediction steps at test time on GRAB dataset. Latency measures the inference latency for generating a single motion sequence.

Figure 7: Qualitative results of proposed HO-Flow approach on GRAB (first four rows) and OakInk (last four rows) benchmarks. Our approach can synthesize realistic hand and object interaction motions across diverse objects and tasks.

Figure 8: Qualitative comparison with the state-of-the-art LatentHOI approach [ 21 ] on GRAB and OakInk benchmarks.

Figure 9: Failure cases of our HO-Flow approach on OakInk objects.

Style-Conditioned Motion Generation

See all 9 papers

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow
Oct 12, 2026Qiang He, Yonghao Zhang, Yinda Zhang +7HOI Generation One-Step Generative Modeling
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google
ULIFLOW: Unified Latent Human-Object Interaction Flow
Oct 16, 2026Wenqi Cai, Chunzhi Gu, Weichen Zhou +5HOI Generation Human Motion Generation
University of Toyama · University of Fukui · Chukyo University +1
JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation
Jul 2, 2026Mingyeong Song, Jungbin Cho, Jisoo Kim +5HOI Generation Diffusion Models
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1

Oct 12, 2026J/K move · Enter open · S save

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow

Qiang He, Yonghao Zhang, Yinda Zhang, Chenyu Meng+6

Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google

HOI Generation One-Step Generative Modeling Flow-Based Generative Modeling Motion Representation Learning

Open paper AskPDF arXiv

Whole-body human-object interaction (HOI) generation requires coordinating body and finger motion, object trajectories, and time-varying contacts. Existing methods typically rely on iterative diffusion or test-time optimization, resulting in high inference costs. Direct one-step generation is highly desired in downstream applications due to efficient inference. However, it is challenging because these HOI components differ substantially in structure and scale. We present MFlowHOI, an efficient one-step framework for whole-body HOI generation. A Contact-Aware Multipart RVQVAE organizes body, hand, object, and contact signals into a structured latent space, with a shared decoder preserving their global coordination. An ST-DiT then models temporal and cross-part dependencies in this space and generates a complete interaction in a single MeanFlow evaluation. Predicted object-local contact correspondences further enable geometric refinement and interaction-preserving transfer to photorealistic 3D Gaussian avatars with different body proportions. Experiments demonstrate that MFlowHOI outperforms existing base generators in motion quality and text alignment while substantially reducing inference time.

Abstract

Style-Conditioned Motion Generation

Explore similar work

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow

ULIFLOW: Unified Latent Human-Object Interaction Flow

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation