Grasping is a fundamental capability for robotic manipulation, with the appropriate grasp depending on the downstream task. For example, a hammer should be held by its handle for hammering but by its head when it is handed over. Selecting such a grasp requires consideration of two criteria: physical feasibility on the object and suitability for the task. Developing such an approach requires dense grasp labels with both feasibility and task scores on real observations, but existing task-oriented grasping datasets are small, synthetic, or costly to annotate. To address this gap, we construct a dataset of 2.68 million scored 5-DoF grasp annotations over 100,911 image-task pairs from 17 object categories. Additionally, we introduce LAGR, a task-oriented grasp detection method that explicitly addresses both these challenges. Given vision-language inputs, LAGR first generates task-independent grasp proposals along with a physical feasibility score, and then scores the generated grasps with a language-conditioned re-ranker that reasons over all proposals jointly. Experiments show that LAGR outperforms the next best method by 9.3 percentage points. We further validate LAGR on real objects by executing task-oriented grasps with a Franka Research 3 robot.
Figures & tables
Figure 1: Examples of LAGR generating distinct task-specific grasps for the same real-world objects. Functional tasks favor grasps that enable effective tool use, whereas handover tasks favor grasps that leave the handles accessible to the recipient.
Figure 2: Comparison of representative grasp annotations from existing datasets and LAGR.
Figure 3: Overview of LAGR. Given multimodal visual observations, a task-independent proposal generator extracts fused visual features and predicts physically plausible grasp candidates with associated confidence scores. A language-conditioned re-ranker combines proposal geometry, multi-scale visual features, and task instruction embeddings, then applies a proposal-context transformer to jointly evaluate candidate suitability. The final grasp is selected by combining task-compatibility predictions with physical confidence, without modifying the proposed grasp geometry.
Configuration
Task-conditioned
Any-task GT
GCR
Time
SR (%)
IoU
Angle ( ∘ )
SR (%)
IoU
Angle ( ∘ )
%
ms/sample
Architectural Components
Physical-confidence selection only
53.03
0.370±0.352
3.23±7.08
88.00
0.619±0.246
2.97±5.98
0.90±6.06
38.95
Without proposal geometry
74.00
0.488±0.295
7.92±12.97
79.09
0.523±0.276
8.10±13.40
1.90±8.56
38.97
Without proposal-context transformer
77.76
0.515±0.275
9.06±13.10
81.27
0.540±0.260
9.09±13.06
1.85±8.64
41.48
Without physical-quality score
80.79
0.538±0.263
8.55±11.93
84.24
0.561±0.246
8.58±11.92
1.90±9.04
40.51
Table 2: Ablation studies on the design components of our approach. Accuracy requires IoU ≥0.25 and angular error ≤30∘ . Angle statistics use positive-overlap matches. Best mean results are bold.
Object
Task
LAGR
GATER
Hammer
Pound
10/10
8/10
Handover
10/10
9/10
Jug
Pour
7/10
7/10
Handover
6/10
0/10
Pan
Cook
10/10
9/10
Handover
10/10
0/10
Table 3: Real-world task-oriented grasping performance using a Franka Research 3 with a parallel-jaw gripper. Each object is evaluated on two tasks, with ten orientations per task. Results indicate successful grasps out of ten trials per object–task pair.
Task-Oriented Grasping (TOG) has emerged as an important skill for purposeful robotic manipulation. Given the task of using a knife to "slice bread," a robot must choose to grasp from the handle, whereas for "hand[ing] the knife over," it is preferable for the handle to remain unobstructed for safe handoff. Current approaches for TOG either focus on 2D planar grasping, remain limited by their reliance on task-agnostic grasp generators, or are constrained by reliance on finite, manually-annotated, task-specific grasp datasets that cannot cover all possible grasp-task pairs. To that end, we present TGrasp6D, a vision-language based approach for generating task-oriented 6DoF grasps in a zero-shot manner. Given a task and a single RGB-D image of an associated object, TGrasp6D reasons about the appropriate semantic affordance for the given task and object geometry to produce an initial task- informed prior. This generated prior is further refined via 6DoF pose optimization for collision avoidance while preserving task consistency. Our approach achieves state-of-the-art success rate of 94% on the TaskGrasp dataset, with the best baseline at 74%. We also deploy our approach on a Franka Research 3 robot to demonstrate zero-shot generalization to real world objects
Manav Kulshrestha, S. Apoorva Vashisth, and Aniket Bera are with the IDEAS Lab, Department of Computer Science, Purdue University, West Lafayette, IN, USA. · Damon Conover is with the DEVCOM Army Research Laboratory, Adelphi, MD, USA.
This paper presents a training-free pipeline for task-oriented grasp generation that combines pre-trained grasp generation models with vision-language models (VLMs). Unlike traditional approaches that focus solely on stable grasps, our method incorporates task-specific requirements by leveraging the semantic reasoning capabilities of VLMs. We evaluate five querying strategies, each utilizing different visual representations of candidate grasps, and demonstrate significant improvements over a baseline method in both grasp success and task compliance rates, with absolute gains of up to 36.9% in overall success rate. Our results underline the potential of VLMs to enhance task-oriented manipulation, providing insights for future research in robotic grasping and human-robot interaction.
Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/
Yang Xu, Gurpreet Singh Mukker, Raymond Wang +3
University of Toronto · Vector Institute · University of British Columbia +1