Manipulating diverse objects with multi-fingered dexterous hands is challenging due to the high dimensionality and complex dynamics. Human-Object Interaction (HOI) datasets provide rich knowledge about task information and embodied interactions. Instead of solely imitating the human demonstrations, our method learns to holistically predict future hand-object states by leveraging these datasets. The predicted future states of the object can serve as a general-purpose reward term for reinforcement learning, reducing reliance on task-specific reward engineering and enhancing generalization across tasks. We conduct extensive experiments on three manipulation tasks in simulation and the real world. Our approach outperforms existing SOTA methods in both success rate and generalizability on novel objects. Furthermore, we validate the cross-embodiment compatibility of our methods by successfully deploying the skills on different robot hands.
Figures & tables
Figure 1: Our method begins by extracting observations from the initial robot-object configuration. These observations are input into a motion predictor to generate a reference trajectory. Guided by this trajectory, we employ reinforcement learning to train the target policy. Finally, the learned policy is deployed directly to the physical robot via zero-shot Sim-to-Real transfer. Notably, since the learned priors are general to hand morphologies, our framework is adaptable to various dexterous hands.
Figure 2: Illustration of components in our HOI motion predictor.
Figure 3: Qualitative results from real-world experiments.
Parameter
Value
Maximum steps
106
Bootstrap at done
True
Discount factor ( γ )
0.95
GAE parameter ( λ )
0.9
Learning rate
3×10−4
Parallel environments
1024
Table 1: Training parameters of PPO policies.
Method
Simulation
Real world
DexYCB
Non-DexYCB
Novel poses
Avg.
DexYCB
Non-DexYCB
Novel poses
Avg.
ViViDex ( Chen et al. 2025b )
0.87
0.37
0.75
0.66
0.35
0.40
0.25
0.33
AdaDexGrasp ( Shi et al. 2025 )
0.82
0.21
0.64
0.56
0.80
0.20
0.65
0.55
ManipTrans ( Li et al. 2025 )
0.61
0.28
0.03
0.31
0.60
0.30
0.05
0.32
HOP ( Singh et al. 2025 )
0.63
0.59
0.54
0.59
0.60
0.45
0.40
0.48
PPO w/o following ( Schulman et al. 2017 )
0.37
0.42
0.27
0.35
0.30
0.40
0.20
0.30
Table 2: Success rates for YCB object grasping in simulation and on real robots. The novel-pose condition uses DexYCB objects.
Method
Simulation
Real world
ARCTIC
Daily
Avg.
ARCTIC
Daily
Avg.
ManipTrans ( Li et al. 2025 )
0.61
0.00
0.31
0.55
0.00
0.28
Obj-Dex ( Chen et al. 2025a )
0.74
0.05
0.40
0.70
0.00
0.35
PPO w/o following ( Schulman et al. 2017 )
0.00
0.00
0.00
0.00
0.00
0.00
Ours
0.79
0.53
0.66
0.70
0.35
0.53
Table 3: Success rates for articulated object manipulation in simulation and on real robots. ARCTIC and Daily denote objects from the ARCTIC dataset and daily-life objects, respectively.
Method
Simulation
Real world
DexYCB obstacle
3D-printed obstacle
Avg.
DexYCB obstacle
3D-printed obstacle
Avg.
ViViDex ( Chen et al. 2025b )
0.17
0.38
0.28
0.15
0.40
0.28
AdaDexGrasp ( Shi et al. 2025 )
0.55
0.67
0.61
0.55
0.60
0.58
ManipTrans ( Li et al. 2025 )
0.00
0.00
0.00
0.00
0.00
0.00
HOP ( Singh et al. 2025 )
0.32
0.41
0.37
0.25
0.40
0.33
PPO w/o following ( Schulman et al. 2017 )
0.58
0.63
0.61
0.50
0.55
0.53
Table 4: Success rates for grasping with obstacles in simulation and on real robots. Real-world collisions are detected using xArm’s built-in collision detector.
Figure 4: Cross-embodiment test on 3 robot hands.
Figure 5: Learning curves for reward ablation (Solid lines: the mean over all seeds. Shaded areas: standard deviation).