Bimanual articulated-object interaction generation requires a model to capture the evolution of object articulation, coordination between the two hands, and fine-grained hand--object contact. Existing methods typically encode object and hand motion as a unified high-dimensional sequence, making it difficult to explicitly accommodate the different scales of manipulation progress, relative bimanual motion, and local hand pose. We propose DuetHOI, a structured framework for bimanual articulated-object interaction generation. DuetHOI first uses ContactVAE to predict contact intent on the object surface from language and object geometry. Conditioned on this spatial intent, ArtPlanner provides a trajectory-level articulation reference, and DualFormer generates the global interaction using object states, object-centered hand positions, and compact hand-pose tokens learned by ManiVAE. After global generation, ProxiRefine fixes the object trajectory and residually corrects the two hands using current hand--object proximity, improving local surface alignment. Experiments across articulated and rigid objects and single- and bimanual settings show that DuetHOI outperforms three adapted baselines on most contact and hand--object consistency metrics, with particularly strong results in bimanual articulated interaction generation.
Figures & tables
Figure 1: DuetHOI generation process. Contact intent and an articulation reference guide global interaction generation, after which ProxiRefine keeps the object trajectory fixed while correcting local bimanual geometry with proximity cues.
Figure 2: DuetHOI pipeline. ContactVAE predicts an object-side contact map that conditions both ArtPlanner and DualFormer. ArtPlanner provides a trajectory-level articulation reference for structured DualFormer generation, ManiVAE maps between hand poses and compact tokens, and ProxiRefine residually corrects the generated hands while preserving object motion. The lower-right inset visualizes the object-centered hand-position representation, with W denoting the world origin.
Figure 3: DualFormer. Slot attention models within-frame hand–object coordination across the five semantic streams, while temporal attention captures the evolution of each stream throughout the denoising process.
Figure 4: Object-fixed ProxiRefine. ProxiRefine derives joint-level proximity cues from the generated interaction and residually corrects bimanual position and pose while keeping the object trajectory fixed.
Dataset
Setting
Method
Phy ↑
C-F1 ↑
MPJPE A ↓
OTE ↓
ORE ↓
SD ↑
OD ↑
C-Rate ↑
C-Stab ↑
NN-Rel ↓
ARCTIC
Bim.-Art.
Text2HOI
0.830
0.698
13.901
0.405
12.670
4.540
8.338
0.619
0.955
0.227
LatentHOI
0.955
0.767
14.183
0.425
14.439
7.205
11.638
0.799
0.957
0.162
MDM
0.867
0.743
14.032
0.380
13.909
8.007
11.746
0.713
0.960
0.209
DuetHOI (Ours)
0.997
0.834
12.043
0.332
12.186
4.489
10.271
0.873
0.970
0.095
Bim.-Rig.
Text2HOI
0.875
0.631
14.002
0.551
6.027
3.770
6.522
0.608
0.959
0.188
LatentHOI
0.845
0.694
14.329
0.545
9.438
6.859
10.243
0.642
0.954
0.178
Table 1: Main bimanual results on ARCTIC and GRAB. Arrows indicate the preferred metric direction.
Figure 5: Qualitative results. DuetHOI preserves bimanual coordination during articulation, including for unseen ovens.
Method
Phy ↑
C-F1 ↑
MPJPE A ↓
OTE ↓
ORE ↓
ARCTIC, Single-Art.
Text2HOI
0.664
0.518
12.535
0.504
11.587
LatentHOI
0.757
0.569
13.249
0.488
13.550
MDM
0.702
0.571
11.647
0.425
11.663
DuetHOI (Ours)
0.992
0.660
9.573
0.361
11.411
ARCTIC, Single-Rig.
Table 2: Single-hand results of the jointly trained model; full metrics are in the supplement.
Variant
Phy ↑
C-F1 ↑
MPJPE A ↓
OTE ↓
ORE ↓
Bim.-Art.
w/o ManiVAE
0.951
0.775
12.458
0.341
12.191
w/o ContactVAE
0.836
0.679
13.690
0.346
12.212
w/o ArtPlanner
0.962
0.779
13.272
0.342
12.202
w/ slot context
0.952
0.776
12.628
0.333
11.872
Flattened atten.
0.968
0.784
12.429
0.335
12.995
Table 3: DualFormer ablation on ARCTIC before ProxiRefine. Rigid objects omit ArtPlanner.
Variant
C-F1 ↑
MPJPE A ↓
C-Rate ↑
C-Stab ↑
NN-Rel ↓
Bim.-Art.
w/o Lcontact
0.797
12.119
0.829
0.962
0.113
w/o Cih
0.827
12.065
0.859
0.968
0.103
Full
0.834
12.043
0.873
0.970
0.095
Bim.-Rig.
w/o Lcontact
0.689
9.622
0.632
0.950
0.112
Table 4: Ablation of the contact loss and joint-level proximity cue in object-fixed ProxiRefine on ARCTIC.
Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.
Mingyeong Song, Jungbin Cho, Jisoo Kim +5
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1
Recent advances in 4D Human-Object Interaction (HOI) generation have enabled increasingly realistic motion synthesis, particularly for single-object manipulation. Yet current research overlooks an inherent property of human behavior: people naturally coordinate both hands and manipulate multiple objects simultaneously. To address this gap, we present Dex2HOI, a unified diffusion model for single- and two-object HOI synthesis from text. At its core, Dex2HOI employs a Dual-Stream Diffusion approach, where each object is processed in a dedicated interaction stream and coordinated through bidirectional cross-attention. To synthesize the final motion, we introduce a Motion Fusion Network integrated with novel hand-relative object representations and contact-aware conditioning applied across the whole sequence. By sampling the diffusion process autoregressively over prefix-conditioned windows, Dex2HOI generates arbitrarily long sequences at real-time speed omitting redundant test-time optimization, achieving up to x540 inference speed-up over prior state-of-the-art methods. Extensive evaluation on both single- and two-object benchmarks demonstrates state-of-the-art quantitative results, marking a step beyond conventional single-object HOI generation and toward expressive multi-object manipulation. Project: https://cpratikaki.github.io/dex2hoi/
Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2
Imperial College London, UK · University of Alicante, Spain
Whole-body human-object interaction (HOI) generation requires coordinating body and finger motion, object trajectories, and time-varying contacts. Existing methods typically rely on iterative diffusion or test-time optimization, resulting in high inference costs. Direct one-step generation is highly desired in downstream applications due to efficient inference. However, it is challenging because these HOI components differ substantially in structure and scale. We present MFlowHOI, an efficient one-step framework for whole-body HOI generation. A Contact-Aware Multipart RVQVAE organizes body, hand, object, and contact signals into a structured latent space, with a shared decoder preserving their global coordination. An ST-DiT then models temporal and cross-part dependencies in this space and generates a complete interaction in a single MeanFlow evaluation. Predicted object-local contact correspondences further enable geometric refinement and interaction-preserving transfer to photorealistic 3D Gaussian avatars with different body proportions. Experiments demonstrate that MFlowHOI outperforms existing base generators in motion quality and text alignment while substantially reducing inference time.
Qiang He, Yonghao Zhang, Yinda Zhang +7
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google