DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations
Authors: Qingtao Liu, Zhengnan Sun, Yu Cui, Haoming Li, Gaofeng Li, Lin Shao, Jiming Chen, Qi Ye
Organizations: the College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · the College of Control Science and Engineering and the State Key Laboratory of Industrial Control Technology, Zhejiang University, China · the Department of Computer Science, National University of Singapore, 15 Computing Dr, 117418, Singapore
Robotic dexterous manipulation is a challenging problem due to high degrees of freedom (DoFs) and complex contacts of multi-fingered robotic hands. Many existing deep reinforcement learning (DRL) based methods aim at improving sample efficiency in high-dimensional output action spaces. However, existing works often overlook the role of representations in achieving generalization of a manipulation policy in the complex input space during the hand-object interaction. In this paper, we propose DexRep, a novel hand-object interaction representation to capture object surface features and spatial relations between hands and objects for dexterous manipulation skill learning. Based on DexRep, policies are learned for three dexterous manipulation tasks, i.e. grasping, in-hand reorientation, bimanual handover, and extensive experiments are conducted to verify the effectiveness. In simulation, for grasping, the policy learned with 40 objects achieves a success rate of 87.9% on more than 5000 unseen objects of diverse categories, significantly surpassing existing work trained with thousands of objects; for the in-hand reorientation and handover tasks, the policies also boost the success rates and other metrics of existing hand-object representations by 20% to 40%. The grasp policies with DexRep are deployed to the real world under multi-camera and single-camera setups and demonstrate a small sim-to-real gap.
Figures & tables
Figure 2: DexRep and its integration into dexterous manipulation learning. Left: Visualization of the three components of DexRep —Occupancy, Surface, and Local-Geo features—each encoding a different aspect of the hand-object interaction. Right: Policy learning framework with DexRep as input. The dashed box denotes the representation for the second hand in bimanual settings, which is omitted in single-hand scenarios.
Figure 2
Figure 5: Retargeting process. Key vectors—finger-to-finger, finger-to-wrist, and finger-to-object—are computed from both human (MANO) and robotic (Adroit) hands and optimized to align hand postures.
Figure 4
Figure 7: 37 Objects used in In-Hand Reorientation. The first 25 objects (above the dotted line) are used in training (seen), while the rest are for evaluation (unseen).
Figure 8: 17 Objects used in Handover. The objects in the first row are used in training (seen), while the rest are for evaluation (unseen).
Figure 9: 40 Objects used in Grasping for analyzing the effectiveness of DexRep. The first 10 objects are from GRAB [ 50 ] , while the rest are from 3DNet [ 61 ] .
Figure 10: Statistical analysis of success rates across random seeds and datasets. (a) Box plots illustrate the distribution of success rates over multiple training seeds for each method and dataset, where the box boundaries correspond to the 25th and 75th percentiles and the central line denotes the median (50th percentile). (b) Bar plots report the mean success rate and 95% confidence intervals (CIs).
Figure 11: Success rate (%) of our method and its variants during training. The x-axis represents the training iterations. The shaded area represents the standard deviation.
Figure 12: T-SNE projection and grasping success rates of object categories. Global features extracted using PointNet [ 15 ] are visualized via t-SNE over objects sampled from DexGraspNet [ 7 ] . Color indicates object category, while grasp success rate is visualized by shading. Notably, objects with similar global features (e.g., pistol, ipod) exhibit different grasp success rates.
Figure 13: Success rate (%) and mean reward of our method and its variants during training. The y-axis indicates the success rate and mean reward during training, and the shaded area represents the standard deviation.
Figure 14: Test success rate (%) of our method applied to partial point clouds. We visualize both (a) the empirical distributions of success rates across random seeds and (b) the corresponding mean success rate with 95% confidence intervals.
Testing Object Sets
5k
25k
50k
GRAB (seen)
89.3 ± 4.5
90.0 ± 4.8
96.5 ± 0.9
GRAB (unseen)
92.1 ± 5.0
93.9 ± 0.9
96.6 ± 0.4
3DNet (unseen)
92.4 ± 4.0
93.7 ± 4.2
97.6 ± 0.9
DexGraspNet (unseen)
77.9 ± 4.2
81.2 ± 2.9
88.1 ± 2.0
Table III : Test success rate (%) of our method with different numbers of pretraining objects. We test on 50 objects from GRAB [ 50 ] , 30 unseen objects from 3DNet [ 61 ] and 5355 objects from DexGraspNet [ 7 ] .
# Fingers
Hand2obj
pGlo
Ours
2
10.2
37.5
65.4
3
18.7
51.1
78.2
4
15.3
23.8
81.5
Table IV: Grasp success rate (%) on unseen objects using robotic hands with different numbers of fingers.
Figure 15: Examples of grasping unseen objects using robotic hands with different numbers of fingers.
Figure 16: Success rates (%) on test objects with varying numbers of training objects. The x-axis is logarithmically scaled to represent the number of training objects. The y-axis shows the mean success rate, and the error bars indicate the standard deviation.
Object Friction Coefficient
1.0
0.8
0.6
0.4
0.2
[0∼1.0] (random)
96.4
93.1
91.9
91.5
82.9
90.8
Table V : Success rates (%) on test objects under different object friction coefficients.
Grasping
Edge Length of Voxel lv
0.01
0.01
0.01
0.015
0.015
0.015
0.02
0.02
0.02
Maximum Distance σmax
0.1
0.15
0.2
0.1
0.15
0.2
0.1
0.15
0.2
Mean Success Rate
93.4
93.2
92.1
92.8
93.7
94.0
94.3
93.5
96.4
In-Hand Orientation
Edge Length of Voxel lv
0.01
0.01
0.01
0.015
0.015
0.015
0.02
0.02
0.02
Maximum Distance σmax
0.1
0.15
0.2
0.1
0.15
0.2
0.1
0.15
0.2
Table VI : Success rates (%) on unseen objects with different edge lengths of voxel lv ( m ) and the maximum distance σmax ( m ) . The bolded settings are used for the main experiments.
Figure 17: The grasping platform with an Allegro Hand v4, a Unitree Z1 arm and an Azure Kinect DK.
Table 20
Figure 18: Point cloud visualization for ICP alignment and multi-camera fusion . (a) Point clouds sampled from the CAD model of a cup (GT); (b) GT with Gaussian noise; (c) GT and partial depth point cloud after ICP registration; (d) GT and point clouds merged from multiview depth images. In (a, c, d), points in blue represent GT and point in other colors represent partial depth points from different views.