RAGrasp: Geometry-Semantic Template Retrieval and Grasp Transfer

Shenzhe Zhu; Chengxiao He; Jan Harder

cs.AIOct 3, 2026

RAGrasp: Geometry-Semantic Template Retrieval and Grasp Transfer

Authors: Shenzhe Zhu, Chengxiao He, Jan Harder

Organizations: Tongji University

Abstract

We present RAGrasp, a retrieval-augmented pipeline for planar parallel-jaw grasping from a compact set of locally collected, grasp-annotated RGB-D (color and depth) templates. Unlike task-specific predictors trained primarily on large public or synthetic grasp datasets, RAGrasp requires no end-to-end retraining for a new deployment.Its template memory is constructed from observations collected with the deployment camera, robot, and gripper in the target workspace, thereby aligning stored examples with the local sensing and embodiment conditions. The system uses self-supervised DINOv2 visual fea- tures together with appearance and depth cues to prompt the Segment Anything Model 2 (SAM2), which isolates the query object. A two-stage geometry-semantic retrieval cascade then selects a template, and a confidence gate chooses one of two grasp- transfer estimators. The transferred grasp is refined using mask- support and silhouette-contact constraints before calibrated 2D- to-3D conversion. In real-world trials, RAGrasp achieves 20/20 successful grasps on seen objects and 19/20 on unseen objects. Within the evaluated setting, the results demonstrate deployment- specific grasp adaptation from limited local annotation and tolerance to the tested viewpoint and illumination changes.

Figures & tables

Fig. 1: Overview of the RAGrasp pipeline. Stage 1 forms a quality-gated SAM2 query mask from RGB-D evidence. Stage 2 performs contour-ICP shortlisting and DINOv2-based fine ranking. Stage 3 selects a geometric or semantic transfer path using a confidence gate. Stage 4 performs mask-constrained local grasp refinement and optional success-conditioned memory update. Stage 5 converts the refined image-space grasp into a calibrated robot command.

Fig. 2: Qualitative example of geometry-semantic retrieval and confidence-gated dual-path grasp transfer.

Fig. 3: End-to-end grasp prediction and real-robot execution. (a) RGB observation of the target object. (b) Refined image-space grasp: the center marker represents $(u,v)$ , the orientation indicator shows the closing direction $\theta$ , and the separation between the two jaw markers represents the opening $w$ . The gripper closes along the line joining the two jaw markers. (c) Robot approach to the calibrated target pose. (d) Parallel-gripper closure. (e) Successful execution verified by a 20-cm lift and a 10-s hold.

Fig. 4: Conditions used in Table I : (a) normal laboratory illumination; (b) reduced ambient illumination; and (c) camera tilt. The angle $\phi$ is measured between the viewing direction (orange) and the tabletop normal (blue), with $\phi=0^{\circ}$ denoting a top-down view. Arrows are schematic.

Camera Tilt $\phi$	Normal Light	Weak Light
$0^{\circ}$	10/10 (100%)	10/10 (100%)
$15^{\circ}$	9/10 (90%)	8/10 (80%)
$30^{\circ}$	9/10 (90%)	9/10 (90%)
$45^{\circ}$	9/10 (90%)	8/10 (80%)

TABLE I: Physical grasp success under viewpoint and illumination changes. Each condition contains 10 trials.

Method	Seen Objects	Unseen Objects
RAGrasp	20/20 (100%)	19/20 (95%)
GraspSAM	18/20 (90%)	16/20 (80%)
NDF	18/20 (90%)	17/20 (85%)
AnyGrasp	20/20 (100%)	16/20 (80%)

TABLE II: Physical grasp success under normal lighting and a perpendicular camera view. Each split contains 20 objects.

Fig. 5: Transfer-grasp success versus object-template similarity $\rho_{q}$ , defined in ( 7 ) as the maximum contour-aligned mask IoU between the query and any template. Performance decreases as geometric coverage of the query by the bank becomes weaker.

Fig. 6: Success rate versus the number of templates in the retrieval bank. Seen and unseen performance improves rapidly from 10 to 40 templates and then approaches saturation.

Ablated Version	Seen	Unseen
Geometry-only transfer (w/o DINOv2 correspondence and confidence gate)	16/20 (80%)	16/20 (80%)
w/o confidence-gated fallback (always use DINOv2-RANSAC transfer)	17/20 (85%)	14/20 (70%)
w/o DINOv2+SAM2 semantic mask (use RGB-D geometric mask)	15/20 (75%)	13/20 (65%)
w/o mask-constrained local refinement (use transferred grasp directly)	18/20 (90%)	16/20 (80%)
w/o success-conditioned write-back (do not add verified executions to the template bank)	18/20 (90%)	16/20 (80%)
Full RAGrasp	20/20 (100%)	19/20 (95%)

TABLE III: Ablation study on seen and unseen categories. Each entry reports successful grasps over 20 trials.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 316 papers

SAGP: Semantic Affordance-Guided Grasp Planning via Coarse-Zone VLM Reasoning
Jul 31, 2026Muhayy Ud Din, Irfan HussainRobotic Grasping VLMs for Robotics
KU Center for Autonomous Robotic Systems (KUCARS), Khalifa University, United Arab Emirates
SuperGrasp: Single-View Object Grasping via Superquadric Similarity Matching, Evaluation, and Refinement
Mar 31, 2026Lijingze Xiao, Jinhong Du, Supeng Diao +2Superquadric Representation Robotic Grasping
GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation
Aug 20, 2026Julien Merand, Boris Meden, Mathieu Grossard +1Robotic Grasping Generalization in Robotic Manipulation
Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Ecole Centrale de Lyon, CNRS, LIRIS, UMR5205, Institut Universitaire de France (IUF), F-69130 Ecully, France

Jul 31, 2026J/K move · Enter open · S save

SAGP: Semantic Affordance-Guided Grasp Planning via Coarse-Zone VLM Reasoning

Muhayy Ud Din, Irfan Hussain

KU Center for Autonomous Robotic Systems (KUCARS), Khalifa University, United Arab Emirates

Robotic Grasping VLMs for Robotics

Open paper AskPDF arXiv

Geometry-based grasp planners ensure physically valid grasps but ignore functional semantics, often generating grasps that are antipodal and collision-free yet practically inappropriate, for example, gripping a mug by its rim, a knife by the blade, or a bottle near its cap. These inconsistencies cause the downstream task to fail even when traditional grasp metrics are met. Existing vision-language model (VLM) approaches either depend on fine-grained, category-specific part segmentation or attempt to directly infer grasp poses, with the latter prone to spatial hallucinations. As a result, no practical, training-free framework has yet been proposed that robustly links high-level semantic reasoning to geometric grasp planning. We introduce Semantic Affordance-Guided Grasp Planning (SAGP), a training-free pipeline built on a coarse-zone abstraction layer. The method first partitions the object point cloud into spatial regions (top, middle, bottom, lateral sides, and protrusions) by applying PCA-based alignment followed by distance-driven DBSCAN clustering, entirely bypassing learned segmentation. A pre-trained VLM then assesses the grasp quality of each region through a structured zero-shot query, and the resulting zone-wise scores are fused with geometric, reachability, and task-alignment signals to re-rank antipodal grasp candidates. Experiments on YCB objects in PyBullet with a Franka Panda robot show that SAGP preserves the high success rate of geometry-only planning while substantially improving the functional appropriateness of selected grasps, particularly on asymmetric, handle-bearing objects where geometry alone is uninformative. The introduced coarse-zone abstraction offers an effective, training-free bridge between VLM-based reasoning and geometric grasp planning, without the need for fine-grained part segmentation.

Abstract

Robotic Grasping

Explore similar work

SAGP: Semantic Affordance-Guided Grasp Planning via Coarse-Zone VLM Reasoning

SuperGrasp: Single-View Object Grasping via Superquadric Similarity Matching, Evaluation, and Refinement

GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation