Hand-object interaction (HOI) reconstruction and synthesis are becoming central to embodied AI and AR/VR. Yet, despite rapid progress, existing HOI generation research remains fragmented across three disjoint tracks: (1) pose-only synthesis that predicts MANO trajectories without producing pixels; (2) single-image HOI generation that hallucinates appearance from masks or 2D cues but lacks dynamics; and (3) video generation methods that require both the entire pose sequence and the ground-truth first frame as inputs, preventing true sim-to-real deployment. Inspired by the philosophy of Joo et al. (2018), we think that HOI generation requires a unified engine that brings together pose, appearance, and motion within one coherent framework. Thus we introduce PAM: a Pose-Appearance-Motion Engine for controllable HOI video generation. The performance of our engine is validated by: (1) On DexYCB, we obtain an FVD of 29.13 (vs. 38.83 for InterDyn), and MPJPE of 19.37 mm (vs. 30.05 mm for CosHand), while generating higher-resolution 480x720 videos compared to 256x256 and 256x384 baselines. (2) On OAKINK2, our full multi-condition model improves FVD from 68.76 to 46.31. (3) An ablation over input conditions on DexYCB shows that combining depth, segmentation, and keypoints consistently yields the best results. (4) For a downstream hand pose estimation task using SimpleHand, augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline.
Figures & tables
Figure 2: Overview of Four Approaches to HOI Synthesis. (a) Pose-Only Synthesis [ 79 ] : This method predicts the MANO trajectories without generating pixel data; (b) Apperance Generation [ 83 ] : This approach generates appearance based on masks or 2D cues but lacks dynamic motion; (c) Motion Generation [ 48 , 3 ] : These methods require both the full pose sequence and the ground-truth first frame as inputs, limiting their application for true sim-to-real transfer. (d) Our Pipeline PAM: In this approach, video generation does not rely on the first frame or the whole HOI pose sequence, allowing for the transfer of HOI pose sequences from the simulator to real-world videos.
Figure 3: Overview of our three-stage generation pipeline. (1) Pose Generation: A pretrained pose generation model generates the intermediate hand-object interaction (HOI) poses based on the initial and target poses, along with the object mesh. (2) Appearance Generation: A controllable image diffusion model synthesizes the first frame of the video, conditioned on multi-modal inputs (depth maps, semantic masks, and keypoint annotations). (3) Motion Generation: The generated HOI sequence and the first frame are rendered into a full video sequence by a video diffusion model, conditioned on the same multi-modal inputs used in the appearance generation stage.
Method
Venue
FVD (↓)
MF (↑)
LPIPS (↓)
SSIM (↑)
PSNR (↑)
MPJPE (↓)
Resolution
CosHand [ 61 ]
ECCV’24
58.51
0.591
0.139
0.767
23.20
30.05
256 x 256
InterDyn [ 3 ]
CVPR’25
38.83
0.680
0.119
0.848
24.86
-
256 x 384
ManiVideo [ 48 ]
CVPR’25
-
-
0.079
0.913
30.10
57.30
-
Ours w/ all
-
29.13
0.712
0.069
0.914
30.17
19.37
480 x 720
Table 1: Quantitative comparison on DexYCB dataset. Our method is evaluated against CosHand, InterDyn, and ManiVideo. Results for InterDyn and ManiVideo are taken from their original papers. For fair comparison, CosHand was fine-tuned on the s0-split training set identical to ours. Our approach achieves state-of-the-art performance across all metrics (FVD, LPIPS, MF, MPJPE) while generating high-resolution 480x720 videos.
Method
FVD (↓)
MF (↑)
LPIPS (↓)
SSIM (↑)
PSNR (↑)
MPJPE (↓)
CosHand [ 61 ]
68.76
0.651
0.156
0.765
23.84
14.49
Ours w/ seg
48.97
0.708
0.084
0.831
25.76
9.61
Ours w/ depth
50.85
0.702
0.086
0.845
26.98
10.07
Ours w/ hand
52.41
0.671
0.113
0.838
25.66
8.01
Ours w/ all
46.31
0.777
0.081
0.851
28.36
7.01
Table 2: Quantitative results on the OAKINK2 dataset. Comparison of our method with CosHand. For a fair evaluation, both models are trained on the same dataset. Our approach achieves state-of-the-art performance, outperforming CosHand across all evaluated metrics.
Figure 4: Qualitative comparison against CosHand. Example results on DexYCB and OAKINK2 highlight the strengths of our method in two key areas: (1) higher visual fidelity in both foreground and background generation, and (2) improved geometric accuracy of the synthesized hand poses.
Conditions
FVD (↓)
MF (↑)
LPIPS (↓)
SSIM (↑)
PSNR (↑)
MPJPE (↓)
Seg
33.23
0.695
0.077
0.900
29.27
21.14
Depth
30.00
0.703
0.070
0.906
29.15
23.16
Hand
33.41
0.713
0.086
0.901
29.07
20.70
Depth, Hand
29.62
0.711
0.071
0.899
29.95
20.46
Seg, Hand
29.53
0.711
0.073
0.902
29.57
19.92
Depth, Seg
29.32
0.712
0.071
0.906
30.60
22.51
Table 3: Ablation study on input conditions on DexYCB dataset.
Figure 5: Ablation study on input conditions on DexYCB dataset.
Figure 6: Sim-to-real transfer results. Our pipeline can generate realistic videos given initial and target states with diversity.
Figure 7: Data augmentation analysis with varying ratios of real data. We augment different portions of the DexYCB training set (25%, 50%, 75%, 100%) with our generated synthetic data. The baseline (dashed line) indicates performance when training solely on 100% of the real DexYCB data without synthetic augmentation.
Figure 8: Zero-shot result on OAKINK2 dataset for i2v task. We use the weight trained on DexYCB dataset.
Resource Usage
Stage-I
Stage-II
Stage-III
Memory (GB)
0.03
41.4
30.3
Time (s)
19.3
36.1
245.7
Table 5: Resource Usage across Different Stages
Settings
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
0 Mask Prob + Clean Cond
28.56
30.99
19.01
0 Mask Prob + Noisy Cond
34.58
27.11
23.67
0.2 Mask Prob + Clean Cond ( Ours )
29.13
30.17
19.37
0.2 Mask Prob + Noisy Cond
30.45
29.67
20.31
Table 6: Ablation Study on Masking Probability and Input Condition Quality
Method (Stage-I)
FVD ( ↓ )
MF ( ↑ )
MPJPE ( ↓ )
D-Grasp
58.17
0.599
36.18
GraspXL ( Ours )
49.98
0.645
30.96
Table 7: Ablation on Stage-I Method
Hand Representation
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
Mesh Projection
29.33
30.17
36.18
Keypoints ( Ours )
29.13
30.05
30.96
Table 8: Ablation on Hand Representation
Encoder Type
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
MLP
31.59
30.07
21.96
VAE ( Ours )
29.13
30.17
19.37
Table 9: Ablation on Hand Encoder
Encoder Type
FVD ( ↓ )
PSNR ( ↑ )
MPJPE ( ↓ )
InterDyn (SVD w/ single cond)
38.83
24.86
28.15
SVD w/ multi conds
34.91
25.84
25.11
Ours (CogVideo w/ multi conds)
29.13
30.17
19.37
Table 10: Ablation on Backbone
Figure 9: More qualitative results on DexYCB dataset (a).
Figure 10: More qualitative results on DexYCB dataset (a).
Figure 11: More qualitative results on DexYCB dataset (b).
Figure 12: More qualitative results on OAKINK2 dataset (a).
Figure 13: More qualitative results on OAKINK2 dataset (b).
Department of Computer Vision Technology(VIS), Baidu Inc., China · Shenzhen Campus of Sun Yat-Sen University, China · Northwestern Polytechnical University, China