TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping

Manav Kulshrestha; Apoorva Vashisth; Damon Conover; Aniket Bera

cs.ROOct 14, 2026

TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping

Authors: Manav Kulshrestha, Apoorva Vashisth, Damon Conover, Aniket Bera

Organizations: Manav Kulshrestha, S. Apoorva Vashisth, and Aniket Bera are with the IDEAS Lab, Department of Computer Science, Purdue University, West Lafayette, IN, USA. · Damon Conover is with the DEVCOM Army Research Laboratory, Adelphi, MD, USA.

Abstract

Task-Oriented Grasping (TOG) has emerged as an important skill for purposeful robotic manipulation. Given the task of using a knife to "slice bread," a robot must choose to grasp from the handle, whereas for "hand[ing] the knife over," it is preferable for the handle to remain unobstructed for safe handoff. Current approaches for TOG either focus on 2D planar grasping, remain limited by their reliance on task-agnostic grasp generators, or are constrained by reliance on finite, manually-annotated, task-specific grasp datasets that cannot cover all possible grasp-task pairs. To that end, we present TGrasp6D, a vision-language based approach for generating task-oriented 6DoF grasps in a zero-shot manner. Given a task and a single RGB-D image of an associated object, TGrasp6D reasons about the appropriate semantic affordance for the given task and object geometry to produce an initial task- informed prior. This generated prior is further refined via 6DoF pose optimization for collision avoidance while preserving task consistency. Our approach achieves state-of-the-art success rate of 94% on the TaskGrasp dataset, with the best baseline at 74%. We also deploy our approach on a Franka Research 3 robot to demonstrate zero-shot generalization to real world objects

Figures & tables

Figure 1: Example of TGrasp6D producing distinct 6DoF grasps for different tasks on the same real-world object. “handover” task results in grasps that keep the handle clear whereas tasks like “pound” and “cook” that require holding the hammer and pan from the handle are reflected.

Figure 2: Overview of TGrasp6D. Given a single RGB-D observation and a natural-language task instruction, a vision-language model identifies and localizes the task-relevant affordance and predicts an image-space grasp orientation. These cues are lifted into 3D to form a task-informed grasp prior, from which candidate 6DoF grasps are sampled and refined for collision avoidance while preserving task consistency, yielding the final task-oriented grasp.

Method	SR $\uparrow$ (%)	GFR $\downarrow$ (%)	fOE $\downarrow$ (rad)	fPE $\downarrow$ (cm)
GCNGrasp [ 14 ]	72.1	3.21	0.72 $\pm 0.35$	1.36 $\pm 0.82$
GraspGPT [ 20 ]	61.7	20.90	0.86 $\pm 0.38$	1.90 $\pm 1.75$
GraspMolmo [ 4 ]	65.0	19.94	0.77 $\pm 0.39$	1.68 $\pm 1.41$
GraspCoT [ 3 ]	7.5	0.00	0.93 $\pm 0.42$	4.45 $\pm 2.18$
F. Grasp [ 19 ]	73.7	15.43	0.79 $\pm 0.40$	1.46 $\pm 0.83$
TGrasp6D	94.5	0.00	0.77 $\pm 0.44$	1.91 $\pm 1.33$

Table I : Comparison with baseline methods. Note that fOE and fPE are reported only for grasps marked as failures. All baseline results use unmodified author-released code and, where available, author-provided pretrained weights; we do not alter any baseline’s inference pipeline.

Panel of Figure 3 — Figure 3: Qualitative comparison of different 6DoF TOG approaches on objects from the TaskGrasp [ 14 ] dataset. For each method, we show 5 task-oriented grasps per object. Compared to other approaches, our TGrasp6D produces more successful grasps across diverse object types and tasks. For cases like “scoop[ing]” with the “spoon,” most approaches generate feasible grasps. However, performance drops sharply for more challenging cases like doing a “handover” with the “measuring cup.” Of note, compared to the pure vision-language prior (f), grasp-pose refinement (g) appears to visually diversify grasps by generating feasible stable and task-consistent alternates. In some cases, like the “measuring cup,” it even recovers some failures despite noise and missing points due to occlusion in the object point-cloud.

Object	Task	TGrasp6D (Ours)	GraspGPT	F. Grasp
Hammer	pound	10/10	2/10	5/10
Hammer	handover	10/10	6/10	5/10
Jug	pour	8/10	7/10	4/10
Jug	handover	8/10	5/10	7/10
Pan	cook	10/10	6/10	3/10
Pan	handover	10/10	3/10	5/10

Table II : Real-robot task-oriented grasping success rate comparison.

Figure 4: Analysis of failure modes, including the effect of grasp refinement for GPT-5.5. Note the y-axis breaks between 4%-10% and 12%-19% to accommodate Gemini-3.7 (11.0%) and Qwen3.5-35B (19.7%) without compressing the smaller values.

VLM	SR (%) $\uparrow$	# Tokens Used $(\times 10^{3})$
VLM	SR (%) $\uparrow$	Affordance	Localization	Grasp Axis
GPT 5.5	$89.68$	$0.20$ $\pm 0.16$	$0.34$ $\pm 0.34$	$0.61$ $\pm 0.52$
Gemini 3.7	$84.19$	$0.78$ $\pm 0.74$	$1.20$ $\pm 1.32$	$1.80$ $\pm 1.28$
Qwen3.5-35B	$75.16$	$2.12$ $\pm 1.69$	$1.36$ $\pm 1.60$	$5.09$ $\pm 2.08$

Table III: Pre-pose refinement ablation using other vision-language models for affordance reasoning, localization, and grasp-prior generation.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 302 papers

GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin
Sep 24, 2026Daniel J. Evans, Yinlong Dai, Simon Stepputtis +1Robotic Grasping Zero-Shot Robotic Manipulation
Collab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061. · TEA Lab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061.
OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping
Sep 16, 2026Jiyao Zhang, Junhan Wang, Tianyu Wang +4Robotic Grasping Dexterous Manipulation
CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1
Generalizable task-oriented object grasping through LLM-guided ontology and similarity-based planning
Mar 27, 2026Hao Chen, Takuya Kiyokawa, Weiwei Wan +1Robotic Grasping Generalization in Robotic Manipulation
Graduate School of Engineering Science, The University of Osaka, 1-3, Machikaneyamacho, Toyonaka, Osaka, 560-8531, Japan

Sep 24, 2026J/K move · Enter open · S save

GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin

Daniel J. Evans, Yinlong Dai, Simon Stepputtis, Dylan P. Losey

Collab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061. · TEA Lab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061.

Robotic Grasping Zero-Shot Robotic Manipulation Sim-to-Real Transfer

Open paper AskPDF arXiv

As robots transition from structured factory settings into homes, they are required to interact with an ever-increasing variety of objects. Many tasks require grasping, and often it is not sufficient to just pick up the target object. Consider a task like "pouring coffee" --- to facilitate the subsequent pouring, the robot should grasp the mug by its handle. Existing learning-based approaches for grasping either find robust and collision-free grasps that are largely agnostic to the task (e.g., picking up the mug by its rim), or leverage foundation models to propose task-appropriate grasp locations that lack fine-grained physical grounding (e.g., reaching for and missing the handle). In this work, we bridge these approaches with a real-to-sim-to-real framework. Based on a single RGB-D observation, we construct a digital twin of the environment, query a large foundation model to propose grasps that align with the object's affordances and task description, and then optimize the proposals to ensure robustness and plausibility before executing the result on the real robot. Our key insight is that the grasp proposals of the foundation model should be regarded as semantic priors that serve as seeds for local, gradient-free optimization. We leverage Bayesian optimization with Thompson sampling to draw batches of nearby poses, which are subsequently evaluated in parallel under domain-randomized physics rollouts. The resulting grasp is both task-oriented and physically feasible for execution by the robot arm. Our full zero-shot real-world transfer only takes a few minutes and improves task-oriented grasping success by up to 33% as compared to other state-of-the-art pipelines. Our code is available here: https://github.com/VT-Collab/GraspTwin/

Abstract

Robotic Grasping

Explore similar work

GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin

OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping

Generalizable task-oriented object grasping through LLM-guided ontology and similarity-based planning