LAGR: Language-Aware Grasp Re-Ranking for Task Oriented Grasping
Organizations: Purdue University, West Lafayette, IN, USA · IIT Delhi, New Delhi, India · DEVCOM ARL, Adelphi, MD, USA
Abstract
Grasping is a fundamental capability for robotic manipulation, with the appropriate grasp depending on the downstream task. For example, a hammer should be held by its handle for hammering but by its head when it is handed over. Selecting such a grasp requires consideration of two criteria: physical feasibility on the object and suitability for the task. Developing such an approach requires dense grasp labels with both feasibility and task scores on real observations, but existing task-oriented grasping datasets are small, synthetic, or costly to annotate. To address this gap, we construct a dataset of 2.68 million scored 5-DoF grasp annotations over 100,911 image-task pairs from object categories. Additionally, we introduce LAGR, a task-oriented grasp detection method that explicitly addresses both these challenges. Given vision-language inputs, LAGR first generates task-independent grasp proposals along with a physical feasibility score, and then scores the generated grasps with a language-conditioned re-ranker that reasons over all proposals jointly. Experiments show that LAGR outperforms the next best method by 9.3 percentage points. We further validate LAGR on real objects by executing task-oriented grasps with a Franka Research 3 robot.
Figures & tables
| Configuration | Task-conditioned | Any-task GT | GCR | Time | ||||
| SR (%) | IoU | Angle ( ∘ ) | SR (%) | IoU | Angle ( ∘ ) | ms/sample | ||
| Architectural Components | ||||||||
| Physical-confidence selection only | 53.03 | 88.00 | 38.95 | |||||
| Without proposal geometry | 74.00 | 79.09 | 38.97 | |||||
| Without proposal-context transformer | 77.76 | 81.27 | 41.48 | |||||
| Without physical-quality score | 80.79 | 84.24 | 40.51 | |||||
| Object | Task | LAGR | GATER |
|---|---|---|---|
| Hammer | Pound | 10/10 | 8/10 |
| Handover | 10/10 | 9/10 | |
| Jug | Pour | 7/10 | 7/10 |
| Handover | 6/10 | 0/10 | |
| Pan | Cook | 10/10 | 9/10 |
| Handover | 10/10 | 0/10 |
Robotic Grasping
Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.
See all 336 papers