DuetHOI: Language-Guided Bimanual Hand--Object Motion Generation with Articulation Planning and Contact Refinement

Zhi Wang; Yuyan Liu; Liu liu; Ruonan Liu; Ruixuan Liu

cs.ROMar 9, 2026

DuetHOI: Language-Guided Bimanual Hand--Object Motion Generation with Articulation Planning and Contact Refinement

Authors: Zhi Wang, Yuyan Liu, Liu liu, Ruonan Liu, Ruixuan Liu

Organizations: Hefei University of Technology · Shanghai Jiao Tong University · Anhui University

Abstract

Bimanual articulated-object interaction generation requires a model to capture the evolution of object articulation, coordination between the two hands, and fine-grained hand--object contact. Existing methods typically encode object and hand motion as a unified high-dimensional sequence, making it difficult to explicitly accommodate the different scales of manipulation progress, relative bimanual motion, and local hand pose. We propose DuetHOI, a structured framework for bimanual articulated-object interaction generation. DuetHOI first uses ContactVAE to predict contact intent on the object surface from language and object geometry. Conditioned on this spatial intent, ArtPlanner provides a trajectory-level articulation reference, and DualFormer generates the global interaction using object states, object-centered hand positions, and compact hand-pose tokens learned by ManiVAE. After global generation, ProxiRefine fixes the object trajectory and residually corrects the two hands using current hand--object proximity, improving local surface alignment. Experiments across articulated and rigid objects and single- and bimanual settings show that DuetHOI outperforms three adapted baselines on most contact and hand--object consistency metrics, with particularly strong results in bimanual articulated interaction generation.

Figures & tables

Figure 1: DuetHOI generation process. Contact intent and an articulation reference guide global interaction generation, after which ProxiRefine keeps the object trajectory fixed while correcting local bimanual geometry with proximity cues.

Figure 2: DuetHOI pipeline. ContactVAE predicts an object-side contact map that conditions both ArtPlanner and DualFormer. ArtPlanner provides a trajectory-level articulation reference for structured DualFormer generation, ManiVAE maps between hand poses and compact tokens, and ProxiRefine residually corrects the generated hands while preserving object motion. The lower-right inset visualizes the object-centered hand-position representation, with $W$ denoting the world origin.

Figure 3: DualFormer. Slot attention models within-frame hand–object coordination across the five semantic streams, while temporal attention captures the evolution of each stream throughout the denoising process.

Figure 4: Object-fixed ProxiRefine. ProxiRefine derives joint-level proximity cues from the generated interaction and residually corrects bimanual position and pose while keeping the object trajectory fixed.

Dataset	Setting	Method	Phy $\uparrow$	C-F1 $\uparrow$	MPJPE A $\downarrow$	OTE $\downarrow$	ORE $\downarrow$	SD $\uparrow$	OD $\uparrow$	C-Rate $\uparrow$	C-Stab $\uparrow$	NN-Rel $\downarrow$
ARCTIC	Bim.-Art.	Text2HOI	0.830	0.698	13.901	0.405	12.670	4.540	8.338	0.619	0.955	0.227
		LatentHOI	0.955	0.767	14.183	0.425	14.439	7.205	11.638	0.799	0.957	0.162
		MDM	0.867	0.743	14.032	0.380	13.909	8.007	11.746	0.713	0.960	0.209
		DuetHOI (Ours)	0.997	0.834	12.043	0.332	12.186	4.489	10.271	0.873	0.970	0.095
	Bim.-Rig.	Text2HOI	0.875	0.631	14.002	0.551	6.027	3.770	6.522	0.608	0.959	0.188
		LatentHOI	0.845	0.694	14.329	0.545	9.438	6.859	10.243	0.642	0.954	0.178

Table 1: Main bimanual results on ARCTIC and GRAB. Arrows indicate the preferred metric direction.

Figure 5: Qualitative results. DuetHOI preserves bimanual coordination during articulation, including for unseen ovens.

Method	Phy $\uparrow$	C-F1 $\uparrow$	MPJPE A $\downarrow$	OTE $\downarrow$	ORE $\downarrow$
ARCTIC, Single-Art.
Text2HOI	0.664	0.518	12.535	0.504	11.587
LatentHOI	0.757	0.569	13.249	0.488	13.550
MDM	0.702	0.571	11.647	0.425	11.663
DuetHOI (Ours)	0.992	0.660	9.573	0.361	11.411
ARCTIC, Single-Rig.

Table 2: Single-hand results of the jointly trained model; full metrics are in the supplement.

Variant	Phy $\uparrow$	C-F1 $\uparrow$	MPJPE A $\downarrow$	OTE $\downarrow$	ORE $\downarrow$
Bim.-Art.
w/o ManiVAE	0.951	0.775	12.458	0.341	12.191
w/o ContactVAE	0.836	0.679	13.690	0.346	12.212
w/o ArtPlanner	0.962	0.779	13.272	0.342	12.202
w/ slot context	0.952	0.776	12.628	0.333	11.872
Flattened atten.	0.968	0.784	12.429	0.335	12.995

Table 3: DualFormer ablation on ARCTIC before ProxiRefine. Rigid objects omit ArtPlanner.

Bim.-Art.
Variant	C-F1 $\uparrow$	MPJPE A $\downarrow$	C-Rate $\uparrow$	C-Stab $\uparrow$	NN-Rel $\downarrow$
w/o $\mathcal{L}_{\mathrm{contact}}$	0.797	12.119	0.829	0.962	0.113
w/o $C_{i}^{h}$	0.827	12.065	0.859	0.968	0.103
Full	0.834	12.043	0.873	0.970	0.095
Bim.-Rig.
w/o $\mathcal{L}_{\mathrm{contact}}$	0.689	9.622	0.632	0.950	0.112

Table 4: Ablation of the contact loss and joint-level proximity cue in object-fixed ProxiRefine on ARCTIC.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 80 papers

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation
Jul 2, 2026Mingyeong Song, Jungbin Cho, Jisoo Kim +5HOI Generation Diffusion Models
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1
Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation
May 28, 2026Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2HOI Generation Human Motion Generation
Imperial College London, UK · University of Alicante, Spain
MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow
Oct 12, 2026Qiang He, Yonghao Zhang, Yinda Zhang +7HOI Generation One-Step Generative Modeling
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google

Jul 2, 2026J/K move · Enter open · S save

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

Mingyeong Song, Jungbin Cho, Jisoo Kim, Ananya Bal+4

Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA+1

HOI Generation Diffusion Models Hand-Object Interaction

Open paper AskPDF arXiv

Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.

Abstract

HOI Generation

Explore similar work

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow