Organizations: The Department of Electronic Engineering, Tsinghua University, Beijing 100084, China. · The Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.
Single-view RGB-D grasp detection remains a common choice in 6-DoF robotic grasping systems, which typically requires a depth sensor. While RGB-only 6-DoF grasp methods has been studied recently, their inaccurate geometric representation is not directly suitable for physically reliable robotic manipulation, thereby hindering reliable grasp generation. To address these limitations, we propose MG-Grasp, a novel depth-free 6-DoF grasping framework that achieves high-quality object grasping. Leveraging two-view 3D foundation model with camera intrinsic/extrinsic, our method reconstructs metric-scale and multi-view consistent dense point clouds from sparse RGB images and generates stable 6-DoF grasp. Experiments on GraspNet-1Billion dataset and real world demonstrate that MG-Grasp achieves state-of-the-art (SOTA) grasp performance among RGB-based 6-DoF grasping methods.
Figures & tables
Fig. 1: Pipeline of MG-Grasp: given sparse posed RGB images, we first perform Depth Aggregation and Dense Correspondence to obtain up-to-scale pointmaps, and then recover metric scale via Triangulation-based Scale Recovery. The metric pointmaps are further refined with multi-view consistency optimization to produce consistent geometry, which is finally used for grasp guidance and 6-DoF grasp generation.
Fig. 2: Left: metrically scaled pointmaps before refinement; right: refined pointmaps. The highlighted regions (red boxes) illustrate misaligned surface layers before refinement, which are largely removed after refinement, resulting in multi-view consistent geometry for grasping.
Method
Data
Seen
Similar
Novel
Average
PointnetGPD [ 29 ]
RGB-D
25.96/27.59
22.68/24.38
9.23/10.66
19.29/20.88
GraspNet [ 2 ]
RGB-D
27.59/29.88
26.11/27.84
10.55/11.51
21.41/23.08
TransGrasp [ 31 ]
RGB-D
39.81/35.97
29.32/29.71
13.83/11.41
27.65/25.70
HGGD [ 4 ]
RGB-D
59.36/60.26
51.20/48.59
22.17/18.43
44.24/42.43
FlexLog [ 37 ]
RGB-D
72.81 / 69.44
65.21 / 59.01
30.04 / 23.67
56.02 / 50.67
GraspNeRF [ 13 ]
RGB
22.49/24.61
14.15/17.67
11.08/12.86
15.91/18.38
TABLE I: Results on GraspNet Dataset, showing AP on the RealSense/Kinect splits respectively
Method
Seen
Similar
Novel
Average
FeedForward-based (VGGT [ 18 ] )
52.29/51.54
47.98/43.18
20.73/ 22.90
40.33/39.21
SLAM-based (MASt3R-SLAM [ 33 ] )
50.39/43.42
38.54/37.50
18.19/14.10
35.71/31.67
MVS-based (MVSAnywhere [ 25 ] )
42.64/39.12
33.92/32.41
14.83/12.78
30.46/28.10
Ours
63.70 / 66.80
56.03 / 57.35
23.22 /20.47
47.65 / 48.21
TABLE II: Results of different geometry pipelines on GraspNet
Fig. 3: Comparison of unprojected point clouds produced by different geometry pipelines. The highlighted regions illustrate misaligned surface.
Fig. 4: Effect of the number of input views on GraspNet grasping performance (AP).
(A)
(B)
Seen
Similar
Novel
Average
✗
✗
29.27/23.99
25.48/19.51
14.32/10.64
23.02/18.05
✗
✓
40.73/35.06
37.10/23.57
20.90/11.85
32.91/23.49
✓
✗
55.29/39.95
43.67/38.66
20.22/16.25
39.73/31.62
✓
✓
63.70 / 66.80
56.03 / 57.35
23.22 / 20.47
47.65 / 48.21
TABLE III: Ablation study of different modules on GraspNet. (A) Triangulation-based scale recovery; (B) multi-view refinement.
Fig. 5: Real world setting.
Scene
Success
Attempt
1
5
6
2
5
5
3
5
5
4
5
6
5
5
6
6
5
6
TABLE IV: REAL ROBOTIC GRASP EXPERIMENT RESULTS
Module
Time (s)
Ratio (%)
MobileSAM masking
0.06
2.1
Depth Aggregation and Correspondence
0.87
30.5
Triangulation-based Scale Recovery
0.11
3.8
Multi-View Refinement
1.34
46.8
Grasp-aware Guidance
0.05
1.6
Grasp Generation
0.30
10.6
TABLE V: Runtime breakdown of MVR-Grasp per inference
Method
Time (s)
VG-Grasp [ 32 ]
11.20
FeedForward-based (VGGT [ 18 ] )
0.61
Ours
2.86
TABLE VI: End-to-end runtime comparison (seconds per inference).
Manav Kulshrestha, S. Apoorva Vashisth, and Aniket Bera are with the IDEAS Lab, Department of Computer Science, Purdue University, West Lafayette, IN, USA. · Damon Conover is with the DEVCOM Army Research Laboratory, Adelphi, MD, USA.