MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

Rui Cao; Jia-Hui Pan; Yichuan Li; Yunfan Yu; Chi-Wing Fu; Jiangliu Wang; Yun-Hui Liu

cs.ROOct 26, 2026

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

Authors: Rui Cao, Jia-Hui Pan, Yichuan Li, Yunfan Yu, Chi-Wing Fu, Jiangliu Wang, Yun-Hui Liu

Organizations: Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China

Abstract

Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. Code, models, and data will be released upon publication.

Figures & tables

Figure 1: Illustration of MMGNet. Given a single-view RGB-D observation, the image backbone extracts dense image features, which are aligned with the scene points through pixel–point correspondence and injected into successive stages of the 3D backbone to form image-informed 3D features. Rotational graspness prediction then scores joint view–in-plane rotations at candidate grasp centers. Based on the selected rotations, grasp-shaded grouping aggregates local image-informed features within the corresponding gripper frames. Finally, the operation head predicts the remaining depth-wise grasp scores and gripper widths to produce the final 6-DoF grasps.

Figure 2: (a) Grasp representation in MMGNet, showing the grasp coordinate frame with the red point marking the grasp origin. (b) Grasp-shaded grouping, where the black cuboid denotes the grouping region and the red points denote the grouped scene points.

Model	Input	Seen	Similar	Novel	Mean AP
GPD ten Pas et al. [2017]	P	22.87/24.38	21.33/23.18	8.24/9.58	17.48/19.05
PointNetGPD Liang et al. [2019]	P	25.96/27.59	22.68/24.38	9.23/10.66	19.29/20.88
Fang et al. Fang et al. [2023a]	P	27.56/29.88	26.11/27.84	10.55/11.51	21.41/23.08
RGB Matters Gou et al. [2021]	P+I	27.98/32.08	27.23/30.40	12.25/13.08	22.49/25.19
Li et al. Li et al. [2021]	P	36.55/ –	28.36/ –	14.01/ –	26.31/ –
HGGD Chen et al. [2023a]	P+I	59.36/60.26	51.20/48.59	22.17/18.43	44.24/42.43

Table 1: Comparison with state-of-the-art methods on the GraspNet-1Billion dataset Fang et al. [2023a] . Each entry reports RealSense/Kinect results.

Graspness Model	Fusion	AP
Graspness Model	Fusion	Seen	Similar	Novel	Mean
View	None	67.12	54.81	24.31	48.75
View	Hierarchical	71.97	58.37	25.50	51.95
Rotational	None	71.61	62.96	30.30	54.95
Rotational	Direct RGB ‡	71.86	61.84	28.85	54.18
Rotational	Early	76.50	68.44	32.22	59.05

Table 2: Comparison of graspness models and image-to-3D fusion.

Grouping	$K$	Seen	Similar	Novel	Mean	$\Delta$ Mean
Cylinder	8	68.32	59.86	28.22	52.13	-4.74
	16	71.46	62.91	30.19	54.85	-2.02
	32	73.50	65.69	31.41	56.87	0.00
	64	74.02	66.04	31.67	57.24	+0.38
Grasp-shaded	8	76.13	69.90	31.76	59.26	-1.08
	16	77.21	70.36	32.51	60.03	-0.31

Table 3: Sensitivity to the number of grouped local samples

K

under hierarchical image-to-3D injection.

\Delta

denotes the mean AP change relative to

K=32

for the same grouping type.

Target	Metric	View	Rotational	$\Delta$
View graspness	Top-1 (%) $\uparrow$	3.80	4.25	+0.45
Rotation residual	Spearman $\uparrow$	0.311	0.367	+0.056
	Exact Top-1 (%) $\uparrow$	35.76	39.15	+3.39
	Regret $\downarrow$	0.2765	0.2516	$-0.0250$

Table 4: Representation analysis of pre-grouping image-informed 3D features.

Figure 3: Cross-modal representation analysis on the test set. The top row reports CKA, while the middle and bottom rows report bidirectional linear-probe $R^{2}$ for 3D $\rightarrow$ image and image $\rightarrow$ 3D prediction, respectively. Columns correspond to Early, Hierarchical, Late (Concat), Late (Add), and Late (Gate).

Model	Modality	AP
Model	Modality	Seen	Similar	Novel	Mean
GSNet Wang et al. [2021a]	P	67.12	54.81	24.31	48.75
+ Injection	P + I	71.97	58.37	25.50	51.95 ( +3.20 )
Ma et al. Ma and Huang [2023]	P	64.02	59.26	26.18	49.82
+ Injection	P + I	71.07	61.02	27.36	53.15 ( +3.33 )

Table 5: Transfer of hierarchical image-to-3D injection across different grasp methods.

Method	Reference Mean AP	Gaussian Noise	Smoothing	Perlin Region Dropout
GSNet Wang et al. [2021a]	55.92	9.25 (16.54)	49.19 (87.96)	50.13 (89.65)
Ma et al. Ma and Huang [2023]	57.68	25.02 (43.37)	49.43 (85.71)	52.79 ( 91.53 )
AnyGrasp Fang et al. [2023b]	60.42	30.45 (50.40)	50.27 (83.20)	53.04 (87.79)
Point-only	64.27	15.53 (24.16)	56.14 (87.35)	54.16 (84.26)
Ours	67.04	39.28 ( 58.60 )	59.11 ( 88.17 )	61.26 (91.38)

Table 6: Absolute mean AP under the strongest point corruptions. AP retention (%) relative to the uncorrupted synthetic-geometry reference is shown in parentheses.

Variant	Clean AP $\uparrow$	RGB-corrupted AP $\uparrow$
Point-only	54.95 $(+0.00)$	54.95 $(+0.00)$
Early	58.96 $(+4.01)$	56.47 $(+1.51)$
Late (Concat)	56.52 $(+1.56)$	54.34 $(-0.61)$
Late (Add)	58.52 $(+3.56)$	56.55 $(+1.60)$
Late (Gate)	56.59 $(+1.64)$	53.06 $(-1.89)$
Hierarchical (Ours)	60.33 $(\mathbf{+5.38})$	57.28 $(\mathbf{+2.32})$

Table 7: Utility under clean and RGB-corrupted observations. Parentheses show AP differences over Point-only under the same observation condition.

(a) Model latency comparison Model Time (ms) GSNet Wang et al. [2021a] 33.82 AnyGrasp Fang et al. [2023b] 44.55 Ma et al. Ma and Huang [2023] 784.05 Ours 71.74

(b) Resource cost Item Value Parameters 44.14M Profiled GFLOPs 304.47 Infer. peak VRAM 539.86 MiB Train peak VRAM 16.18 GiB Training cost 1.78 GPU-days

Table 8: Inference latency comparison and deployment cost of MMGNet.

Figure 4: Our setup of the real-world evaluation.

Model	Household		Adversarial
Model	Succ./TA	SR (%)	Succ./TA	SR (%)
Point-only	108/140	77.14	102/143	71.33
AnyGrasp Fang et al. [2023b]	110/133	82.71	108/148	72.97
Ours	116/138	84.06	114/140	81.43

Table 9: Results of real-world evaluation

Figure 5: Failure modes in our real-world experiments.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 311 papers

MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations
Mar 17, 2026Kangxu Wang, Siang Chen, Chenxing Jiang +3Robotic Grasping
The Department of Electronic Engineering, Tsinghua University, Beijing 100084, China. · The Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.
DGS-Net: Learning Dexterous Grasps from Single-View Point Clouds via a Multi-Object Scene Dataset
Mar 16, 2026Tao Geng, Dapeng Yang, Ziwei Liu +6Robotic Grasping Synthetic Data Generation for Robotics
Human Universal Grasping
Jun 15, 2026Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu +5HOI Generation Robotic Grasping
New York University · Tsinghua University · University of Michigan

Mar 17, 2026J/K move · Enter open · S save

MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations

Kangxu Wang, Siang Chen, Chenxing Jiang, Shaojie Shen+2

The Department of Electronic Engineering, Tsinghua University, Beijing 100084, China. · The Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.

Robotic Grasping

Open paper AskPDF arXiv

Single-view RGB-D grasp detection remains a common choice in 6-DoF robotic grasping systems, which typically requires a depth sensor. While RGB-only 6-DoF grasp methods has been studied recently, their inaccurate geometric representation is not directly suitable for physically reliable robotic manipulation, thereby hindering reliable grasp generation. To address these limitations, we propose MG-Grasp, a novel depth-free 6-DoF grasping framework that achieves high-quality object grasping. Leveraging two-view 3D foundation model with camera intrinsic/extrinsic, our method reconstructs metric-scale and multi-view consistent dense point clouds from sparse RGB images and generates stable 6-DoF grasp. Experiments on GraspNet-1Billion dataset and real world demonstrate that MG-Grasp achieves state-of-the-art (SOTA) grasp performance among RGB-based 6-DoF grasping methods.

Abstract

Robotic Grasping

Explore similar work

MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations

DGS-Net: Learning Dexterous Grasps from Single-View Point Clouds via a Multi-Object Scene Dataset

Human Universal Grasping