Visual-language grounding aims to establish semantic correspondences between natural language and visual entities, enabling models to accurately identify and localize target objects based on textual instructions. Existing VLG approaches focus on coarse-grained, object-level localization, while traditional robotic grasping methods rely predominantly on geometric cues and lack language guidance, which limits their applicability in language-driven manipulation scenarios. To address these limitations, we propose the RealVLG framework, which integrates the RealVLG-11B dataset and the RealVLG-R1 model to unify real-world visual-language grounding and grasping tasks. RealVLG-11B dataset provides multi-granularity annotations including bounding boxes, segmentation masks, grasp poses, contact points, and human-verified fine-grained language descriptions, covering approximately 165,000 images, over 800 object instances, 1.3 million segmentation, detection, and language annotations, and roughly 11 billion grasping examples. Building on this dataset, RealVLG-R1 employs Reinforcement Fine-tuning on pretrained large-scale vision-language models to predict bounding boxes, segmentation masks, grasp poses, and contact points in a unified manner given natural language instructions. Experimental results demonstrate that RealVLG supports zero-shot perception and manipulation in real-world unseen environments, establishing a unified semantic-visual multimodal benchmark that provides a comprehensive data and evaluation platform for language-driven robotic perception and grasping policy learning. All data and code are publicly available at https://github.com/lif314/RealVLG-R1.
Figures & tables
Type
Dataset
Grasp Label
Objects / Image
Scale
Grounding
Language Description
Images
Objects
Grasps
Seg.
Bbox
Object
Spatial
VLM-Check
Human-Check
Sim.
Dex-net [ 41 ]
Rect.
1
6.7M
1500
6.7M
✓
✗
✗
✗
✗
✗
Jacquard [ 14 ]
Rect.
1
54K
11K
1.1 M
✓
✗
✗
✗
✗
✗
VR-Grasping-101 [ 83 ]
6-DoF
1
10K
101
4.8M
✗
✗
✗
✗
✗
✗
Kappler et al. [ 28 ]
6-DoF
1
700
80
300K
✗
✗
✗
✗
✗
✗
6-DoF GraspNet [ 48 ]
6-DoF
1
206
206
7M
✗
✗
✗
✗
✗
✗
Table 1: Comparison of RealVLG-11B with existing grasping datasets. “–” indicates unknown values.
Figure 2: Annotation pipeline of the RealVLG-11B dataset. The pipeline integrates automatic language generation, model-based verification, and manual review to generate high-quality multi-granularity visual and language annotations.
Set Type
Split
Unique Objects
Images
Instances
Grasps
Training
Train
∼700
∼120K
∼900K
∼10B
Seen
48
15 K
∼150K
∼0.3B
Evaluation
Similar
50
15 K
∼150K
∼0.3B
Novel
37
15 K
∼150K
∼0.3B
Table 2: Dataset split for RealVLG-11B.
Figure 3: Framework of RealVLG-R1. RealVLG-R1 fine-tunes pretrained LVLMs via reward-driven RL using task-specific verifiable rewards, enabling adaptive learning and improved generalization over bounding boxes, segmentation, grasp rectangles, and contact points.
Dataset
MTLD ↑
SCLIP↑
Rs↑
Rg↑
Rc↑
Grasp-Anything [ 75 ]
27.45
0.54
–
0.38
0.69
Grasp-Anything++ [ 76 ]
15.14
0.52
–
0.31
0.62
RealVLG-11B (Ours)
36.49
0.65
0.99
0.69
0.87
Table 3: Linguistic and grounding quality comparison.
Method
Seen
Similar
Novel
Bbox
Seg.
Grasp
Contact
Bbox
Seg.
Grasp
Contact
Bbox
Seg.
Grasp
Contact
gIoU/cIoU/R v
F β /S α /R v
mIoU/gAcc/R v
mIoU/gAcc/R v
gIoU/cIoU/R v
F β /S α /R v
mIoU/gAcc/R v
mIoU/gAcc/R v
gIoU/cIoU/R v
F β /S α /R v
mIoU/gAcc/R v
mIoU/gAcc/R v
Qwen-VL-Max [ 13 ]
92.3 / 93.2 /61.3
87.8 / 44.8 /71.5
16.0 / 16.7 /14.2
34.7 / 25.6 /15.4
89.4 / 90.0 /64.3
86.8 / 43.5 /66.4
14.9 / 7.1 /17.9
32.0 / 7.0 /18.3
88.4 / 88.8 /67.7
47.3 / 38.6 /64.0
8.1 / 5.4 /22.6
18.8 / 20.0 /18.3
Gemini2.5-Flash [ 21 ]
60.1/61.5/ 83.4
3.3/14.1/ 83.4
3.2/1.2/ 100
5.2/3.6/ 100
59.2/60.5/ 80.9
3.8/13.1/ 80.9
2.5/2.2/ 100
6.3/4.7/ 100
62.6/63.7/ 95.1
3.9/14.9/ 95.1
3.4/1.9/ 100
4.0/4.3/ 100
Qwen2.5VL-3B [ 4 ]
50.0/51.9/99.2
75.1/43.5/99.2
4.3/2.9/94.1
25.2/17.7/87.0
51.2/52.4/94.5
75.1/42.4/94.5
4.0/1.4/92.3
27.8/22.9/85.5
52.9/53.7/96.9
45.9/38.7/97.0
3.1/1.3/95.7
14.5/8.6/98.8
Qwen2.5VL-3B + SFT
56.4/57.1/100
76.2/43.4/100
3.4/1.7/100
28.6/21.4/100
54.7/55.9/100
75.4/43.2/100
3.6/2.1/100
29.3/28.2/100
57.2/58.6/100
46.2/38.9/100
4.4/1.5/100
19.7/15.8/100
Table 4: RealVLG benchmark comprehensive results. All metrics are reported in percentage format.
Figure 5: Human-Verification System. This application provides an interactive interface for human-in-the-loop verification, allowing users to review, correct, and confirm automatically generated visual-language annotations. It serves as a crucial component for ensuring the quality and reliability of RealVLG-11B dataset annotations.
Figure 6: Qualitative Comparison of Data Quality. Unlike the diffusion-generated, low-resolution images and weakly aligned textual and grasp annotations in Grasp-Anything datasets, RealVLG-11B provides high-resolution real-world imagery, instance-level language grounding, and standardized, physically executable grasp labels, enabling more accurate and robust visual–language grasping.
Figure 7: Training reward/accuracy curves for GRPO, GSPO, and SFT on Contact tasks. Overall, GRPO and GSPO significantly improve SFT through RLVR. GRPO achieves slightly higher accuracy on 3B, while GSPO performs better on 7B and exhibits more stable outputs across training steps.
Figure 8: Real-world experimental setup. (a) The 7-DoF Franka Research 3 robot equipped with an eye-in-hand Intel RealSense D435i camera, used for real-world evaluation of RealVLG-R1. (b) The set of 10 test objects used to assess the model’s generalization and manipulation performance.
Method
Cup
Orange
Pear
Apple
Banana
Stapler
BlueBottle
Marker
Screwdriver
Razor
Average
GraspNet [ 19 ]
10%
60%
70%
60%
30%
50%
40%
20%
30%
10%
38%
RealVLG-R1 (Ours)
90%
100%
100%
100%
100%
80%
70%
80%
100%
90%
81%
Table 5: Quantitative real-world grasping results in the Single setting. RealVLG-R1 performs language-conditioned grasping, whereas GraspNet serves as a vision-only baseline. Each task is executed 10 times.
Figure 9: Qualitative real-world grasping results in the Single setting. GraspNet often fails or predicts misaligned grasp poses due to noisy or incomplete point cloud data (e.g., Cup ), reflective surfaces, and small or thin objects, such as Marker , Screwdriver , and Razor . In contrast, RealVLG-R1 leverages RGB vision and language instructions to accurately localize the target and generate executable grasp contact points, demonstrating robust and reliable grasping behavior across diverse objects.
Method
Cup
Orange
Pear
Apple
Banana
Stapler
BlueBottle
Marker
Screwdriver
Razor
Average
LGD [ 76 ]
0%
0%
0%
0%
0%
20%
0%
0%
0%
0%
2%
RealVLG-R1 (Ours)
70%
90%
80%
90%
70%
90%
70%
80%
70%
80%
79%
Table 6: Quantitative real-world grasping results in the Clutter setting. The table reports the grasp success rates of RealVLG-R1 and the baseline LGD across 10 target objects. Results highlight RealVLG-R1’s superior zero-shot performance in cluttered multi-object environments, whereas LGD fails to reliably perform language-conditioned grasping.
Figure 10: Qualitative real-world grasping results of LGD [ 76 ] in the Clutter setting. LGD struggles to perform language-conditioned grasps in cluttered environments due to limited perceptual resolution, suboptimal language integration, and reliance on unconditional grasp pose predictions.
Figure 11: Qualitative real-world grasping results of RealVLG-R1 in the Clutter setting. RealVLG-R1 demonstrates accurate language-conditioned grasping, robust zero-shot performance in cluttered environments, and interpretable predictions of grasp poses.