End-to-End Dexterous Grasp Learning from Single-View Point Clouds via a Multi-Object Scene Dataset

Tao Geng; Dapeng Yang; Ziwei Liu; Le Zhang; Le Qi; WangYang Li; Yi Ren; Shan Luo

cs.ROMar 16, 2026

End-to-End Dexterous Grasp Learning from Single-View Point Clouds via a Multi-Object Scene Dataset

Authors: Tao Geng, Dapeng Yang, Ziwei Liu, Le Zhang, Le Qi, WangYang Li, Yi Ren, Shan Luo, +1 more

Abstract

Dexterous grasping in multi-object scene constitutes a fundamental challenge in robotic manipulation. Current mainstream grasping datasets predominantly focus on single-object scenarios and predefined grasp configurations, often neglecting environmental interference and the modeling of dexterous pre-grasp gesture, thereby limiting their generalizability in real-world applications. To address this, we propose DGS-Net, an end-to-end grasp prediction network capable of learning dense grasp configurations from single-view point clouds in multi-object scene. Furthermore, we propose a two-stage grasp data generation strategy that progresses from dense single-object grasp synthesis to dense scene-level grasp generation. Our dataset comprises 307 objects, 240 multi-object scenes, and over 350k validated grasps. By explicitly modeling grasp offsets and pre-grasp configurations, the dataset provides more robust and accurate supervision for dexterous grasp learning. Experimental results show that DGS-Net achieves grasp success rates of 88.63% in simulation and 78.98% on a real robotic platform, while exhibiting lower penetration with a mean penetration depth of 0.375 mm and penetration volume of 559.45 mm^3, outperforming existing methods and demonstrating strong effectiveness and generalization capability. Our dataset is available at https://github.com/4taotao8/DGS-Net.

Figures & tables

Fig. 1: (a) Joint configuration. (b) Fingertip points and coordinate systems.

Fig. 2: Overview of DGS-Net. PTv3 extracts multi-scale point cloud features. Module I (green) predicts grasp reference point and 6D rotations; Module II (yellow) generates grasp offsets and joint configurations. During training (bottom), ground-truth grasp reference point and pose labels guide Module I to speed convergence and improve stability. During inference (top), Module II uses Module I predictions, forming an end-to-end pipeline.

Fig. 3: (a) Single-object grasp data generation pipeline, generating dense grasps for a single object using fingertip point constraints and admittance control. (b) Circular/rectangular fingertip point constraints. (c) Grasps under different fingertip point constraints. (d) Grasp joint configurations under circular constraints with varying parameters $r$ and $h$ .

Fig. 4: (a) Scene-level grasp data generation pipeline. (b) Grasp labels on point cloud: blue represents object point clouds; purple denotes the dexterous hand’s TCP $T$ ; Red indicates grasp reference points, which are the points in the object point cloud closest to the hand’s TCP; green denotes grasp candidate points in the neighborhood of each grasp reference point; gray point clouds correspond to the tabletop.

	Object Count	Grasps(%)
	Object Count	5finger	3finger	2finger
Cuboid	116	46.652	0	0
Cylinder	69	23.147	0	0
Stick	72	18.031	3.366	0.231
Sphere	26	4.123	2.596	0
Bowl	24	1.178	0.231	0.034

TABLE I: Object and Grasp Data Distribution

Pre-grasp	offset	SR(%)						CR(%)
Pre-grasp	offset	Cuboid	Cylinder	Stick	Sphere	Bowl	Total	CR(%)
✗	✓	71.79	72.24	68.72	32.07	65.21	69.83	80.58
✓	✗	74.67	75.92	65.28	36.17	64.64	71.10	82.07
✓	✓	77.49	79.25	72.42	42.42	74.51	74.81	84.73

TABLE II: Ablation Studies in Simulation

	Objects/Scene	Dataset	SR(%)	CR(%)	Time Cost(s)
HGC-Net [ 28 ]	10	HGC-Net	71.9	78.8	0.25
DGS-Net	10	HGC-Net	73.07	81.13	0.45
DGS-Net	10	Ours (Include YCB)	74.81	84.73	0.45
DGN 2.0 [ 29 ]	10		73.59	83.87	0.5
DGS-Net	4-6		88.63	87.06	0.45
Multi-FinGAN [ 27 ]	1	YCB	60	–	7.4

TABLE III: Grasp Efficiency Experiment Results

	PD( $mm$ ) $\downarrow$	PV( $mm^{3}$ ) $\downarrow$	D( $\%$ ) $\downarrow$
	Mean $\pm$ Variance	Mean $\pm$ Variance	D( $\%$ ) $\downarrow$
HGC-Net [ 28 ]	12.93 $\pm$ 5.28	13970 $\pm$ 13206	51.23
GraspTTA [ 12 ]	8.21 $\pm$ 4.63	2647.70 $\pm$ 1744.08	48.80
DVGG [ 10 ]	4.10 $\pm$ -	3800 $\pm$ -	–
Dexonomy [ 33 ]	8.28 $\pm$ 7.76	7314.02 $\pm$ 10163.44	25.70
DGN 2.0 [ 29 ]	0.594 $\pm$ 7.13	1452.24 $\pm$ 3056.10	27.14

TABLE IV: Grasp Quality Experiment Results

Fig. 5: Real-world execution pipeline: (top) model prediction, (middle) pre-grasp configuration, and (bottom) final grasping execution.

Fig. 6: Real-world experimental setup: (a) UR3 with a DexHand Pro and an overhead depth camera capturing scene point clouds; objects placed in a $0.4\times 0.8$ m workspace. (b) The 29 real-world test objects (unseen during training).

	Ours	HGC-Net [ 28 ]	Kiatos et al. [ 26 ]	Graspit!
SR(%)	78.98	66.7	74.0	58.5
CR(%)	86.86	75.0	75.0	54.3

TABLE V: Real-world Experimental Results

Explore similar work

GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation
Aug 20, 2026Julien Merand, Boris Meden, Mathieu Grossard +1Robotic Grasping Generalization in Robotic Manipulation
Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Ecole Centrale de Lyon, CNRS, LIRIS, UMR5205, Institut Universitaire de France (IUF), F-69130 Ecully, France
OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes
Oct 8, 2026Naiyu Fang, Zhongjin Luo, Yuxin Mo +7Robotic Grasping Robot Manipulation Benchmarks
ACE Robotics · CUHK, MMLab · CPII under InnoHK +4
BiDexGrasp: Coordinated Bimanual Dexterous Grasps across Object Geometries and Sizes
Apr 8, 2026Mu Lin, Yi-Lin Wei, Jiaxuan Chen +7Robotic Grasping Dexterous Manipulation
Sun Yat-sen University · Peking University · Tsinghua University

Aug 20, 2026J/K move · Enter open · S save

GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation

Julien Merand, Boris Meden, Mathieu Grossard, Liming Chen

Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Ecole Centrale de Lyon, CNRS, LIRIS, UMR5205, Institut Universitaire de France (IUF), F-69130 Ecully, France

Robotic Grasping Generalization in Robotic Manipulation

Open paper AskPDF arXiv

Multifingered grasping is a crucial robotic skill, but current deep-learning grasp planners often struggle to generalize to new objects because they are trained on limited, object-specific datasets. We introduce a fundamentally different approach, grounded in the observation that the gripper and the object share identical surface geometry at their mutual contact points. We propose GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation, a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution, enabling the efficient sampling of valid grasp configurations without relying on object-specific training data. We show that by introducing object features only at inference time, our model can effectively retrieve admissible contact areas that are compatible with the gripper's capabilities. We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios, demonstrating its effectiveness with different grippers from the literature. Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%. It offers significantly faster processing when generating numerous grasps, while matching the performance of leading approaches specifically trained on this dataset. Unlike these methods, our approach does not rely on object-specific training data, highlighting the advantages of object-agnostic learning. It effectively addresses the generalization challenges faced by traditional data-driven grasp planners. Code and videos are available on our project website https://cea-list.github.io/goagweb/ .

Abstract

Explore similar work

GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation

OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes

BiDexGrasp: Coordinated Bimanual Dexterous Grasps across Object Geometries and Sizes