Hand-object interaction (HOI) reconstruction and synthesis are becoming central to embodied AI and AR/VR. Yet, despite rapid progress, existing HOI generation research remains fragmented across three disjoint tracks: (1) pose-only synthesis that predicts MANO trajectories without producing pixels; (2) single-image HOI generation that hallucinates appearance from masks or 2D cues but lacks dynamics; and (3) video generation methods that require both the entire pose sequence and the ground-truth first frame as inputs, preventing true sim-to-real deployment. Inspired by the philosophy of Joo et al. (2018), we think that HOI generation requires a unified engine that brings together pose, appearance, and motion within one coherent framework. Thus we introduce PAM: a Pose-Appearance-Motion Engine for controllable HOI video generation. The performance of our engine is validated by: (1) On DexYCB, we obtain an FVD of 29.13 (vs. 38.83 for InterDyn), and MPJPE of 19.37 mm (vs. 30.05 mm for CosHand), while generating higher-resolution 480x720 videos compared to 256x256 and 256x384 baselines. (2) On OAKINK2, our full multi-condition model improves FVD from 68.76 to 46.31. (3) An ablation over input conditions on DexYCB shows that combining depth, segmentation, and keypoints consistently yields the best results. (4) For a downstream hand pose estimation task using SimpleHand, augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline.
Figures & tables
Figure 2: Overview of Four Approaches to HOI Synthesis. (a) Pose-Only Synthesis [ 79 ] : This method predicts the MANO trajectories without generating pixel data; (b) Apperance Generation [ 83 ] : This approach generates appearance based on masks or 2D cues but lacks dynamic motion; (c) Motion Generation [ 48 , 3 ] : These methods require both the full pose sequence and the ground-truth first frame as inputs, limiting their application for true sim-to-real transfer. (d) Our Pipeline PAM: In this approach, video generation does not rely on the first frame or the whole HOI pose sequence, allowing for the transfer of HOI pose sequences from the simulator to real-world videos.
Figure 3: Overview of our three-stage generation pipeline. (1) Pose Generation: A pretrained pose generation model generates the intermediate hand-object interaction (HOI) poses based on the initial and target poses, along with the object mesh. (2) Appearance Generation: A controllable image diffusion model synthesizes the first frame of the video, conditioned on multi-modal inputs (depth maps, semantic masks, and keypoint annotations). (3) Motion Generation: The generated HOI sequence and the first frame are rendered into a full video sequence by a video diffusion model, conditioned on the same multi-modal inputs used in the appearance generation stage.
Method
Venue
FVD (↓)
MF (↑)
LPIPS (↓)
SSIM (↑)
PSNR (↑)
MPJPE (↓)
Resolution
CosHand [ 61 ]
ECCV’24
58.51
0.591
0.139
0.767
23.20
30.05
256 x 256
InterDyn [ 3 ]
CVPR’25
38.83
0.680
0.119
0.848
24.86
-
256 x 384
ManiVideo [ 48 ]
CVPR’25
-
-
0.079
0.913
30.10
57.30
-
Ours w/ all
-
29.13
0.712
0.069
0.914
30.17
19.37
480 x 720
Table 1: Quantitative comparison on DexYCB dataset. Our method is evaluated against CosHand, InterDyn, and ManiVideo. Results for InterDyn and ManiVideo are taken from their original papers. For fair comparison, CosHand was fine-tuned on the s0-split training set identical to ours. Our approach achieves state-of-the-art performance across all metrics (FVD, LPIPS, MF, MPJPE) while generating high-resolution 480x720 videos.
Method
FVD (↓)
MF (↑)
LPIPS (↓)
SSIM (↑)
PSNR (↑)
MPJPE (↓)
CosHand [ 61 ]
68.76
0.651
0.156
0.765
23.84
14.49
Ours w/ seg
48.97
0.708
0.084
0.831
25.76
9.61
Ours w/ depth
50.85
0.702
0.086
0.845
26.98
10.07
Ours w/ hand
52.41
0.671
0.113
0.838
25.66
8.01
Ours w/ all
46.31
0.777
0.081
0.851
28.36
7.01
Table 2: Quantitative results on the OAKINK2 dataset. Comparison of our method with CosHand. For a fair evaluation, both models are trained on the same dataset. Our approach achieves state-of-the-art performance, outperforming CosHand across all evaluated metrics.
Figure 4: Qualitative comparison against CosHand. Example results on DexYCB and OAKINK2 highlight the strengths of our method in two key areas: (1) higher visual fidelity in both foreground and background generation, and (2) improved geometric accuracy of the synthesized hand poses.
Conditions
FVD (↓)
MF (↑)
LPIPS (↓)
SSIM (↑)
PSNR (↑)
MPJPE (↓)
Seg
33.23
0.695
0.077
0.900
29.27
21.14
Depth
30.00
0.703
0.070
0.906
29.15
23.16
Hand
33.41
0.713
0.086
0.901
29.07
20.70
Depth, Hand
29.62
0.711
0.071
0.899
29.95
20.46
Seg, Hand
29.53
0.711
0.073
0.902
29.57
19.92
Depth, Seg
29.32
0.712
0.071
0.906
30.60
22.51
Table 3: Ablation study on input conditions on DexYCB dataset.
Figure 5: Ablation study on input conditions on DexYCB dataset.
Figure 6: Sim-to-real transfer results. Our pipeline can generate realistic videos given initial and target states with diversity.
Figure 7: Data augmentation analysis with varying ratios of real data. We augment different portions of the DexYCB training set (25%, 50%, 75%, 100%) with our generated synthetic data. The baseline (dashed line) indicates performance when training solely on 100% of the real DexYCB data without synthetic augmentation.
Figure 8: Zero-shot result on OAKINK2 dataset for i2v task. We use the weight trained on DexYCB dataset.
Resource Usage
Stage-I
Stage-II
Stage-III
Memory (GB)
0.03
41.4
30.3
Time (s)
19.3
36.1
245.7
Table 5: Resource Usage across Different Stages
Settings
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
0 Mask Prob + Clean Cond
28.56
30.99
19.01
0 Mask Prob + Noisy Cond
34.58
27.11
23.67
0.2 Mask Prob + Clean Cond ( Ours )
29.13
30.17
19.37
0.2 Mask Prob + Noisy Cond
30.45
29.67
20.31
Table 6: Ablation Study on Masking Probability and Input Condition Quality
Method (Stage-I)
FVD ( ↓ )
MF ( ↑ )
MPJPE ( ↓ )
D-Grasp
58.17
0.599
36.18
GraspXL ( Ours )
49.98
0.645
30.96
Table 7: Ablation on Stage-I Method
Hand Representation
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
Mesh Projection
29.33
30.17
36.18
Keypoints ( Ours )
29.13
30.05
30.96
Table 8: Ablation on Hand Representation
Encoder Type
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
MLP
31.59
30.07
21.96
VAE ( Ours )
29.13
30.17
19.37
Table 9: Ablation on Hand Encoder
Encoder Type
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
InterDyn (SVD w/ single cond)
38.83
24.86
28.15
SVD w/ multi conds
34.91
25.84
25.11
Ours (CogVideo w/ multi conds)
29.13
30.17
19.37
Table 10: Ablation on Backbone
Figure 9: More qualitative results on DexYCB dataset (a).
Figure 10: More qualitative results on DexYCB dataset (a).
Figure 11: More qualitative results on DexYCB dataset (b).
Figure 12: More qualitative results on OAKINK2 dataset (a).
Figure 13: More qualitative results on OAKINK2 dataset (b).
Hand-Object Interaction (HOI) remains a core challenge in digital human video synthesis, where models must generate physically plausible contact and preserve object identity across frames. Although recent HOI reenactment approaches have achieved progress, they are typically trained and evaluated in-domain and fail to generalize to complex, in-the-wild scenarios. In contrast, all-in-one video editing models exhibit broader robustness but still struggle with HOI-specific issues such as inconsistent object appearance. In this paper, we present GenHOI, a lightweight augmentation to pretrained video generation models that injects reference-object information in a temporally balanced and spatially selective manner. For temporal balancing, we propose Head-Sliding RoPE, which assigns head-specific temporal offsets to reference tokens, distributing their influence evenly across frames and mitigating the temporal decay of 3D RoPE to improve long-range object consistency. For spatial selectivity, we design a two-level spatial attention gate that concentrates object-conditioned attention on HOI regions and adaptively scales its strength, preserving background realism while enhancing interaction fidelity. Extensive qualitative and quantitative evaluations on unseen, in-the-wild scenes demonstrate that GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one video editing methods. Project page: https://xuanhuang0.github.io/GenHOI/
Xuan Huang, Mochu Xiang, Zhelun Shen +9
Department of Computer Vision Technology(VIS), Baidu Inc., China · Shenzhen Campus of Sun Yat-Sen University, China · Northwestern Polytechnical University, China
Generating realistic 3D hand-object interactions (HOI) is a fundamental challenge in computer vision and robotics, requiring both temporal coherence and high-fidelity physical plausibility. Existing methods remain limited in their ability to learn expressive motion representations for generation and perform temporal reasoning. In this paper, we present HO-Flow, a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects. HO-Flow first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold by incorporating hand and object kinematics, enabling the representation to capture rich interaction dynamics. It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation, improving temporal coherence. To further enhance generalization, HO-Flow predicts object motions relative to the initial frame, enabling effective pre-training on large-scale synthetic data. Experiments on the GRAB, OakInk, and DexYCB benchmarks demonstrate that HO-Flow achieves state-of-the-art performance in both physical plausibility and motion diversity for interaction motion synthesis.
Human-Object Interaction (HOI) video reenactment aims to transfer the interaction dynamics of a source video to a novel target object while preserving realistic hand-object coordination. Existing methods typically rely on sparse 2D motion controls and monocular references, which are insufficient for complex out-of-plane motion and large viewpoint changes. We present MVHOI, a two-stage framework combining implicit motion extraction, 3D-aware multi-view reasoning, and video generation. In the first stage, a motion extractor encodes object dynamics into implicit motion descriptors. Conditioned on these descriptors, our Motion-Driven Object Prior (MDOP) module queries a 3D foundation model over multi-view references of the target object and autoregressively predicts coarse object anchors, a sequence of images that track the object's evolving orientation and appearance under the source motion without any explicit pose estimation. In the second stage, a DiT-based video generation model uses these anchors as structural guidance and the multi-view references as appearance guidance. We further reuse cross-view attention from MDOP as a soft attention bias to reduce reference-view confusion. For long videos, a cross-iterative inference strategy refreshes subsequent object priors using refined video outputs. Experiments demonstrate consistent improvements over state-of-the-art methods in object fidelity, motion consistency, visual quality, and interaction realism.
Jinguang Tong, Jinbo Wu, Kaisiyuan Wang +12
The Australian National University · CSIRO · Baidu Inc. +1