DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations
Authors: Qingtao Liu, Zhengnan Sun, Yu Cui, Haoming Li, Gaofeng Li, Lin Shao, Jiming Chen, Qi Ye
Organizations: the College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · the College of Control Science and Engineering and the State Key Laboratory of Industrial Control Technology, Zhejiang University, China · the Department of Computer Science, National University of Singapore, 15 Computing Dr, 117418, Singapore
Robotic dexterous manipulation is a challenging problem due to high degrees of freedom (DoFs) and complex contacts of multi-fingered robotic hands. Many existing deep reinforcement learning (DRL) based methods aim at improving sample efficiency in high-dimensional output action spaces. However, existing works often overlook the role of representations in achieving generalization of a manipulation policy in the complex input space during the hand-object interaction. In this paper, we propose DexRep, a novel hand-object interaction representation to capture object surface features and spatial relations between hands and objects for dexterous manipulation skill learning. Based on DexRep, policies are learned for three dexterous manipulation tasks, i.e. grasping, in-hand reorientation, bimanual handover, and extensive experiments are conducted to verify the effectiveness. In simulation, for grasping, the policy learned with 40 objects achieves a success rate of 87.9% on more than 5000 unseen objects of diverse categories, significantly surpassing existing work trained with thousands of objects; for the in-hand reorientation and handover tasks, the policies also boost the success rates and other metrics of existing hand-object representations by 20% to 40%. The grasp policies with DexRep are deployed to the real world under multi-camera and single-camera setups and demonstrate a small sim-to-real gap.
Figures & tables
Figure 2: DexRep and its integration into dexterous manipulation learning. Left: Visualization of the three components of DexRep —Occupancy, Surface, and Local-Geo features—each encoding a different aspect of the hand-object interaction. Right: Policy learning framework with DexRep as input. The dashed box denotes the representation for the second hand in bimanual settings, which is omitted in single-hand scenarios.
Figure 2
Figure 5: Retargeting process. Key vectors—finger-to-finger, finger-to-wrist, and finger-to-object—are computed from both human (MANO) and robotic (Adroit) hands and optimized to align hand postures.
Figure 4
Figure 7: 37 Objects used in In-Hand Reorientation. The first 25 objects (above the dotted line) are used in training (seen), while the rest are for evaluation (unseen).
Figure 8: 17 Objects used in Handover. The objects in the first row are used in training (seen), while the rest are for evaluation (unseen).
Figure 9: 40 Objects used in Grasping for analyzing the effectiveness of DexRep. The first 10 objects are from GRAB [ 50 ] , while the rest are from 3DNet [ 61 ] .
Figure 10: Statistical analysis of success rates across random seeds and datasets. (a) Box plots illustrate the distribution of success rates over multiple training seeds for each method and dataset, where the box boundaries correspond to the 25th and 75th percentiles and the central line denotes the median (50th percentile). (b) Bar plots report the mean success rate and 95% confidence intervals (CIs).
Figure 11: Success rate (%) of our method and its variants during training. The x-axis represents the training iterations. The shaded area represents the standard deviation.
Figure 12: T-SNE projection and grasping success rates of object categories. Global features extracted using PointNet [ 15 ] are visualized via t-SNE over objects sampled from DexGraspNet [ 7 ] . Color indicates object category, while grasp success rate is visualized by shading. Notably, objects with similar global features (e.g., pistol, ipod) exhibit different grasp success rates.
Figure 13: Success rate (%) and mean reward of our method and its variants during training. The y-axis indicates the success rate and mean reward during training, and the shaded area represents the standard deviation.
Figure 14: Test success rate (%) of our method applied to partial point clouds. We visualize both (a) the empirical distributions of success rates across random seeds and (b) the corresponding mean success rate with 95% confidence intervals.
Testing Object Sets
5k
25k
50k
GRAB (seen)
89.3 ± 4.5
90.0 ± 4.8
96.5 ± 0.9
GRAB (unseen)
92.1 ± 5.0
93.9 ± 0.9
96.6 ± 0.4
3DNet (unseen)
92.4 ± 4.0
93.7 ± 4.2
97.6 ± 0.9
DexGraspNet (unseen)
77.9 ± 4.2
81.2 ± 2.9
88.1 ± 2.0
Table III : Test success rate (%) of our method with different numbers of pretraining objects. We test on 50 objects from GRAB [ 50 ] , 30 unseen objects from 3DNet [ 61 ] and 5355 objects from DexGraspNet [ 7 ] .
# Fingers
Hand2obj
pGlo
Ours
2
10.2
37.5
65.4
3
18.7
51.1
78.2
4
15.3
23.8
81.5
Table IV: Grasp success rate (%) on unseen objects using robotic hands with different numbers of fingers.
Figure 15: Examples of grasping unseen objects using robotic hands with different numbers of fingers.
Figure 16: Success rates (%) on test objects with varying numbers of training objects. The x-axis is logarithmically scaled to represent the number of training objects. The y-axis shows the mean success rate, and the error bars indicate the standard deviation.
Object Friction Coefficient
1.0
0.8
0.6
0.4
0.2
[0∼1.0] (random)
96.4
93.1
91.9
91.5
82.9
90.8
Table V : Success rates (%) on test objects under different object friction coefficients.
Grasping
Edge Length of Voxel lv
0.01
0.01
0.01
0.015
0.015
0.015
0.02
0.02
0.02
Maximum Distance σmax
0.1
0.15
0.2
0.1
0.15
0.2
0.1
0.15
0.2
Mean Success Rate
93.4
93.2
92.1
92.8
93.7
94.0
94.3
93.5
96.4
In-Hand Orientation
Edge Length of Voxel lv
0.01
0.01
0.01
0.015
0.015
0.015
0.02
0.02
0.02
Maximum Distance σmax
0.1
0.15
0.2
0.1
0.15
0.2
0.1
0.15
0.2
Table VI : Success rates (%) on unseen objects with different edge lengths of voxel lv ( m ) and the maximum distance σmax ( m ) . The bolded settings are used for the main experiments.
Figure 17: The grasping platform with an Allegro Hand v4, a Unitree Z1 arm and an Azure Kinect DK.
Table 20
Figure 18: Point cloud visualization for ICP alignment and multi-camera fusion . (a) Point clouds sampled from the CAD model of a cup (GT); (b) GT with Gaussian noise; (c) GT and partial depth point cloud after ICP registration; (d) GT and point clouds merged from multiview depth images. In (a, c, d), points in blue represent GT and point in other colors represent partial depth points from different views.
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Jusuk Lee, Sungha Kim, Yeonsoo Park +8
Seoul National University · University of Maryland, College Park · All Purpose AI +1
Dexterous hands enable concurrent prehensile and nonprehensile manipulation, such as holding one object while interacting with another, a capability essential for everyday tasks yet underexplored in robotics. Learning such long-horizon, contact-rich multi-stage behaviors is challenging because demonstrations are expensive to collect and end-to-end policies require substantial data to generalize across varied object geometries and placements. We present DexMulti, a sample-efficient approach for real-world dexterous multi-task manipulation that decomposes demonstrations into object-centric skills with well-defined temporal boundaries. Rather than learning monolithic policies, our method retrieves demonstrated skills based on current object geometry, aligns them to the observed object state using an uncertainty-aware estimator that tracks centroid and yaw, and executes them via a retrieve-align-execute paradigm. We evaluate on three multi-stage tasks requiring concurrent manipulation (Grasp + Pull, Grasp + Open, and Grasp + Grasp) across two dexterous hands (Allegro and LEAP) in over 1,000 real-world trials. Our approach achieves an average success rate of 66% on training objects with only 3-4 demonstrations per object, outperforming diffusion policy baselines by 2-3x while requiring far fewer demonstrations. Results demonstrate robust generalization to held-out objects and spatial variations up to +/-25 cm.
Hao Jiang, Yue Wu, Yue Wang +2
Thomas Lord Department of Computer Science at the University of Southern California, USA.
Dexterous object rotation is a sequential contact problem: each support, release, and re-contact decision must both produce the desired object motion, and prepare the hand configuration for continued rotation. Existing reinforcement learning methods discover such movement patterns through trial and error on specific robotic hand embodiments, without explicitly accounting for how each contact transition affects the hand's ability to sustain object rotation in subsequent steps. We introduce DexMani, a framework that transfers human demonstrations as contact-conditioned manipulability evolution. This prior captures how successful human contact transitions reshape the object-rotation directions available to the hand. DexMani then learns this manipulability evolution and uses it to guide downstream reinforcement learning, enabling rotation skills to be acquired across robot embodiments with distinct kinematics and active-contact configurations. Across the Shadow Hand, Allegro Hand, and XHand, DexMani achieves the highest success rates in every evaluated setting for both seen and unseen objects. DexMani reaches an average success rate of 57.5% on LEAP Hand, outperforming other baselines and producing smoother rotatory motions. Project site: https://dexmani.github.io
Xiaoyang Chen, Shengcheng Luo, Haoran Guo +4
Shanghai Jiao Tong University · ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) +1