Skeleton-Guided Geometry-Aware Auto-Labeling for Robotic Grasping: From 4-DoF Learning to 6-DoF Execution

Ali Sabzejou; Farbod Azimmohseni; Mehdi Tale Masouleh; Ahmad Kalhor

cs.ROOct 28, 2026

Skeleton-Guided Geometry-Aware Auto-Labeling for Robotic Grasping: From 4-DoF Learning to 6-DoF Execution

Authors: Ali Sabzejou, Farbod Azimmohseni, Mehdi Tale Masouleh, Ahmad Kalhor

Organizations: Human and Robot Interaction Laboratory, School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran

Abstract

Robotic grasping relies on large manually annotated datasets, which are costly to create. We introduce a straight-skeleton-based algorithm for generating two-finger grasp keypoints and identifying regions suitable for stable, feasible grasps. Our automated pipeline produces consistent grasp annotations without human intervention. We also propose the Skeleton-based Generative Grasp Convolutional Neural Network (SkelGG-CNN), a 4-DoF architecture that learns geometric grasp features from skeleton-based guidance. To extend auto-labeling to 6-DoF grasping, we develop a generalized quasi-3D skeletonization algorithm that operates directly on point clouds. Using automatically generated 4-DoF labels, SkelGG-CNN achieves accuracies of 95.42%, 96.55%, and 97.1% on Jacquard V1, Jacquard V2, and Cornell, respectively. On the GraspNet novel split, 6-DoF auto-labeling achieves an average precision of 18.33, comparable to state-of-the-art model-based approaches. In simulation, the 6-DoF method achieves success rates of 98.4% on Dex-Net and 89.1% on EGAD!. Real-world experiments with a Delta parallel robot show success rates of 97.4% on single household objects and 95.0% in cluttered scenes, outperforming state-of-the-art methods without domain adaptation. The 4-DoF pipeline adapts across environments and produces annotations approaching human-level precision, enabling efficient training of lightweight models on small datasets. The generalized 6-DoF method provides scalable, geometry-aware annotations for cluttered 3D scenes.

Figures & tables

Figure 1: Experimental setup featuring the Delta parallel robot and an illustration of the SkelGG-CNN model’s output for a real-world scene.

Figure 2: Overview of the proposed method. The pipeline begins with the simulation environment, where RGB-D images are captured from a set of 3D meshes sourced from the Dex-Net dataset. The subsequent stages show the details process of the straight skeleton-based grasp pose estimation algorithm. After this process, the generated grasp poses are used to create the SkelGG-CNN model labels. The skeleton heatmap is generated during the polygon offsetting phase of the algorithm, adding geometric features to the model.

Author	Automatic Dataset Generation	Cornell Results	Jacquard Results	GraspNet Results	EGAD! Results	Cluttered Objects	Code Available
Morrison et al. (2018)	$\times$	✓	$\times$	$\times$	$\times$	✓	✓
Xu et al. (2022)	$\times$	✓	✓	$\times$	$\times$	✓	$\times$
Kumra et al. (2022)	$\times$	✓	✓	$\times$	$\times$	✓	✓
Zhang et al. (2023)	$\times$	✓	✓	$\times$	$\times$	✓	$\times$
Li and Zheng (2024)	$\times$	✓	✓	$\times$	$\times$	✓	$\times$
Duan et al. (2025)	$\times$	✓	✓	$\times$	$\times$	$\times$	$\times$

Table 1: Comparison of various recent grasping approaches across different datasets and approaches.

Figure 3: Comparison of SkelGG-CNN output heatmaps with the pretrained Gr-ConvNetV2 Kumra et al. (2022) model. The first three columns display the quality and angle heatmaps for both models. The fourth column represents the width heatmap for Gr-ConvNet, whereas SkelGG-CNN uses a skeleton heatmap instead, enabling both grasp width estimation and grasp refinement.

Figure 4: Illustration of polygon offsets with the skeleton specified by red lines.

Figure 5: Skeleton heatmap construction steps.

Figure 6: Example of grasp repositioning, the initial grasp center $P_{c_{i}}$ from the quality heatmap is shifted along the orientation $\theta_{i}$ to the point $P_{c}$ with the maximum skeleton heatmap value. This refinement produces more robust grasps compared to the approaches that only use quality maps. In this figure, blue dots represent initial grasp centers and red dots indicate the final grasp centers after the refinement process.

Figure 7: Architecture of the SkelGG-CNN model.

Figure 8: Overview of the generalized 6-DoF skeleton-based auto-labeling pipeline. (Left) The input object point cloud is partitioned into overlapping horizontal cross-sections (80% overlap) orthogonal to the $z$ -axis. (Center — upper path: 3D skeletonization) Each slice is processed with a Laplacian contraction skeletonization; skeleton points within a slice are connected using a Minimum Spanning Tree (MST) and then linked to nearest neighbors in adjacent slices to produce a compact 3D skeleton graph that captures the object’s volumetric structure. (Center — lower path: 2D polygon approximation and initial centers) Each cross-section is projected to 2D and approximated with an alpha-shape polygon; a straight skeleton computed on this polygon yields initial central grasp centers. (Right) Candidate skeleton-node pairs are sampled and refined by traversing the 3D skeleton (depth-first search) from these initial centers; at each candidate pair a local plane is fitted to neighboring skeleton edges and the antipodality of the planes is evaluated as the grasp quality measure. (Far right) Top-scoring pairs are selected and visualized as final 6-DoF gripper poses superimposed on the original point cloud.

Input type	Jacquard accuracy(IoU)
RGB	94.46
Depth	95.32
RGB-D	95.42

Table 2: Performance comparison of algorithms on Accuracy with IoU (%) on Jacquard V1 dataset.

Model parameters	Accuracy (IoU)
951,316	92.10
3,801,252	95.11
15,196,996	95.42

Table 3: Comparison of model performance with different parameter count on the Jacquard V1 dataset.

Authors	Algorithm	Accuracy	Labels
			Train	Test
Zhou et al. (2018)	FCGN, ResNet101	92.80	GT	GT
Zhang et al. (2019)	ROI-GD, ResNet101	93.60	GT	GT
Morrison et al. (2020b)	GG-CNN 2	84.00	GT	GT
Wang et al. (2021b)	DD-Net, Hourglass	97.00	GT	GT
Yu et al. (2022)	SE-ResUNet	95.70	GT	GT

Table 4: Performance comparison of algorithms on Accuracy with IoU (%) on the Jacquard V1 dataset.

Authors	Algorithm	Accuracy	Labels
			Train	Test
He et al. (2016)	ResNet50	91.37	GT	GT
He et al. (2016)	ResNet101	91.35	GT	GT
He et al. (2016)	ResNet152	90.30	GT	GT
Iandola et al. (2016)	Squeeze	94.76	GT	GT
Zhang et al. (2018)	ShuffleNet	94.85	GT	GT

Table 5: Performance comparison of algorithms on Accuracy with IoU (%) on the Jacquard V2 dataset.

Figure 9: Comparison of test accuracies of the trained models on Dex-Net dataset.

Authors	Algorithm	Accuracy	Labels
			Train	Test
Morrison et al. (2020b)	GG-CNN 2	84.0	GT	GT
Zhang et al. (2019)	ROI-GD, ResNet-101	93.6	GT	GT
Wang et al. (2021b)	DD-Net, Hourglass	97.2	GT	GT
Yu et al. (2022)	SE-ResUNet	98.2	GT	GT
Xu et al. (2022)	GKNet	96.9	GT	GT

Table 6: Performance comparison of algorithms on Accuracy with IoU (%) on the Cornell dataset.

Methods	Seen			Unseen			Novel
	AP (%)	AP 0.8 (%)	AP 0.4 (%)	AP (%)	AP 0.8 (%)	AP 0.4 (%)	AP (%)	AP 0.8 (%)	AP 0.4 (%)
GPD Ten Pas et al. (2017)	24.38	30.16	13.46	23.18	28.64	11.32	9.50	10.14	3.16
GG-CNN Morrison et al. (2018)	16.89	22.47	11.23	15.05	19.76	6.19	7.38	8.78	1.32
PointNetGPD Liang et al. (2019)	27.59	34.21	17.83	24.38	30.84	12.83	10.66	11.24	3.21
GraspNet Fang et al. (2020)	29.88	36.19	19.31	27.84	33.19	16.62	11.51	12.92	3.56
RGBMatter Gou et al. (2021)	32.08	39.46	20.85	30.40	37.87	18.72	13.08	13.79	6.01

Table 7: Detailed results on the GraspNet dataset (Kinect), showing average precision (AP) scores.

Figure 10: Results of the quasi-3D auto-labeling method on a real-world scene from the GraspNet dataset, visualized with a blue-to-red gradient indicating grasp quality (red: higher quality, blue: lower quality).

Figure 11: Simulation experiments on the Dex-Net dataset. The first row shows the predicted grasp rectangles generated by the SkelGG-CNN model, along with the corresponding skeleton and quality heatmaps. The second row illustrates the robotic arm’s approach phase. Finally, the third row presents the successfully grasped objects from two different grasps.

Category	Method	EGAD!	Dex-Net Test	Dex-Net Unseen
Learning	GG-CNN	54.0	89.6	88.0
	GR-ConvNetV2	-	94.4	92.2
	SkelGG-CNN ∗	62.4	97.6	95.2
Geometric	2D Auto-Labeling	60.8	92.8	88.0
Geometric	Quasi-3D Auto-Labeling	89.1	98.4	96.0

Table 8: Success rate comparison of algorithms on Dex-Net and EGAD! datasets.

Figure 12: Illustration of the objects used for real robotic experiments.

Figure 13: Grasp predictions produced by the SkelGG-CNN model for single objects in a real-world environment. The figure displays both quality and skeleton maps, highlighting the identified grasp configurations.

Figure 14: Demonstration of a Delta parallel robot grasping objects in a cluttered scene. The first column presents the skeleton and quality output of the SkelGG-CNN model for the initial scene. For each grasp attempt, grasps are generated based on the captured image of the scene in its current state. The second row highlights the bounding box of the selected grasp at each stage.

Algorithm	Training Dataset	Household Isolated	Household Cluttered
SAE, struct. reg. Lenz et al. (2015)	Cornell	89.0 (89/100)	-
AlexNet-based CNN Pinto and Gupta (2016)	Custom	66.0 (99/150)	38.4 (50/130)
Robust Best Grasp Johns et al. (2016)	ModelNet in Sim	80.0 (80/100)	-
Multi-grasp Res-50 Chu et al. (2018)	Cornell	89.0 (89/100)	-
DexNet 2.0 Mahler et al. (2017)	DexNet in Sim	80.0 (40/50)	-
GPD Gualtieri et al. (2016)	CAD models	-	77.3 (116/138)

Table 9: Performance comparison on household isolated and cluttered scenes.

Figure 15: A failure case where the object slipped from the gripper after being grasped.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 316 papers

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes
Oct 26, 2026Rui Cao, Jia-Hui Pan, Yichuan Li +4Robotic Grasping Multimodal Sensor Fusion
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China
GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping
May 31, 2026Beining Han, Yu-Wei Chao, Erwin Coumans +5Robotic Grasping Generalization in Robotic Manipulation
1NVIDIA · 2Princeton University
MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations
Mar 17, 2026Kangxu Wang, Siang Chen, Chenxing Jiang +3Robotic Grasping
The Department of Electronic Engineering, Tsinghua University, Beijing 100084, China. · The Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.

Oct 26, 2026J/K move · Enter open · S save

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

Rui Cao, Jia-Hui Pan, Yichuan Li, Yunfan Yu+3

Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China

Robotic Grasping Multimodal Sensor Fusion

Open paper AskPDF arXiv

Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. Code, models, and data will be released upon publication.

Abstract

Robotic Grasping

Explore similar work

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping

MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations