GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin

Daniel J. Evans; Yinlong Dai; Simon Stepputtis; Dylan P. Losey

cs.ROSep 24, 2026

GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin

Authors: Daniel J. Evans, Yinlong Dai, Simon Stepputtis, Dylan P. Losey

Organizations: Collab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061. · TEA Lab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061.

Abstract

As robots transition from structured factory settings into homes, they are required to interact with an ever-increasing variety of objects. Many tasks require grasping, and often it is not sufficient to just pick up the target object. Consider a task like "pouring coffee" --- to facilitate the subsequent pouring, the robot should grasp the mug by its handle. Existing learning-based approaches for grasping either find robust and collision-free grasps that are largely agnostic to the task (e.g., picking up the mug by its rim), or leverage foundation models to propose task-appropriate grasp locations that lack fine-grained physical grounding (e.g., reaching for and missing the handle). In this work, we bridge these approaches with a real-to-sim-to-real framework. Based on a single RGB-D observation, we construct a digital twin of the environment, query a large foundation model to propose grasps that align with the object's affordances and task description, and then optimize the proposals to ensure robustness and plausibility before executing the result on the real robot. Our key insight is that the grasp proposals of the foundation model should be regarded as semantic priors that serve as seeds for local, gradient-free optimization. We leverage Bayesian optimization with Thompson sampling to draw batches of nearby poses, which are subsequently evaluated in parallel under domain-randomized physics rollouts. The resulting grasp is both task-oriented and physically feasible for execution by the robot arm. Our full zero-shot real-world transfer only takes a few minutes and improves task-oriented grasping success by up to 33% as compared to other state-of-the-art pipelines. Our code is available here: https://github.com/VT-Collab/GraspTwin/

Figures & tables

Figure 1: We present GraspTwin , a zero-shot method for identifying grasps that are both task-oriented and physically feasible. Given a single RGB-D image of the scene, GraspTwin constructs a digital twin. Within that digital twin we query foundation models to propose grasps that facilitate the downstream task. These semantically-aware grasps seed local optimization while enforcing physical constraints (e.g., unintended collisions) and domain randomization. The resulting grasp pose is then executed by the real arm.

Figure 2: Overview of GraspTwin . (Left) We convert from real to sim by constructing a digital twin from an RGB-D image of the robot’s environment. We estimate the scene objects $\hat{\theta}$ and identify $\hat{\theta}_{1}$ , the target object for grasping. (Middle) We then render the target object $\hat{\theta}_{1}$ from multiple perspectives, and use a VLM conditioned on the task $\tau$ to select task-oriented grasp positions $\mathbf{P}_{\mathrm{VLM}}^{(i)}$ and an approach direction $\mathbf{R}_{\mathrm{VLM}}$ . (Right) These proposals become the initial guesses for localized grasp optimization. We simulate grasp poses $\mathbf{T}$ in the digital twin with object perturbations $\Delta\theta$ . Batch Bayesian optimization with Thompson sampling enables parallel evaluations for gradient-free optimization. The minimum-cost grasp pose $\mathbf{T}^{*}$ is executed by the real robot.

#	Task	Source	Sim	Real
1	Hand me the knife	C	✓	✓
2	Cut the bread	A,B,C	✓	✓
3	Grab the mug by the handle	A,B,C	✓	✓
4	Flip the pancake with the spatula	B,C	✓	✓
5	Squeeze the mustard out the bottle	C	✓	✓
6	Grab a wet wipe from the pack	A	✓	✓

Table I: Tasks used in our experiments. Most tasks were taken from related works A [ 19 ] , B [ 22 ] , and C [ 2 ] . All 30 tasks were used in our simulated testing (Table II ), and a subset of 10 tasks was used for real world evaluations (Figure 4 ).

Metric [%]	LERF-TOGO [ 19 ]	GraspMolmo [ 3 ]	Ours-O	Ours-G	GraspTwin
Task-Oriented	$11.3\pm 1.5$	$53.1\pm 2.4$	$75.8\pm 2.0$	$51.3\pm 2.4$	$\textbf{81.1}\pm 1.8$
Lift Object	$9.1\pm 1.4$	$48.0\pm 2.4$	$28.9\pm 2.1$	$48.0\pm 2.4$	$\textbf{54.4}\pm 2.3$
Collision Free	$35.6\pm 2.3$	$76.4\pm 2.0$	$63.8\pm 2.3$	$87.3\pm 1.6$	$\textbf{95.6}\pm 1.0$
Overall	$5.8\pm 1.1$	$33.1\pm 2.2$	$22.9\pm 2.0$	$33.8\pm 2.2$	$\textbf{51.6}\pm 2.4$

Table II: Grasp performance in simulated tasks from Table I . Results are averaged over 30 tasks, each evaluated across 3 base scenes with five object configurations per scene. This yields 450 grasp attempts per method. The values are grasp success % in the form

avg\pm sem

. All pairwise comparisons between GraspTwin and the baselines are statistically significant (

p<.05

Figure 3: Examples of real-world grasps performed using GraspMolmo [ 3 ] (top row) and GraspTwin (bottom row). Each column shows a different task.

Figure 4: Grasp performance in the real-world tasks from Table I . Results are averaged over 10 tasks with 4 trials per method. Error bars show standard error, and an

*

denotes statistical significance (

p<.05

). The solid bars show real-world evaluation, and the hashed bars show performance on the same subset of tasks in simulation. We find that GraspTwin significantly outperforms GraspMolmo in real-world grasp success, and that the gap between GraspTwin’s simulated and real-world performance is not significant.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 302 papers

TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping
Oct 14, 2026Manav Kulshrestha, Apoorva Vashisth, Damon Conover +1Robotic Grasping Zero-Shot Robotic Manipulation
Manav Kulshrestha, S. Apoorva Vashisth, and Aniket Bera are with the IDEAS Lab, Department of Computer Science, Purdue University, West Lafayette, IN, USA. · Damon Conover is with the DEVCOM Army Research Laboratory, Adelphi, MD, USA.
Building Explicit World Model for Zero-Shot Open-World Object Manipulation
Mar 14, 2026Xiaotong Li, Gang Chen, Javier Alonso-MoraZero-Shot Robotic Manipulation Robotic Manipulation
Department of Cognitive Robotics (CoR), Delft University of Technology, 2628CD Delft, The Netherlands.
OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping
Sep 16, 2026Jiyao Zhang, Junhan Wang, Tianyu Wang +4Robotic Grasping Dexterous Manipulation
CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1

Oct 14, 2026J/K move · Enter open · S save

TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping

Manav Kulshrestha, Apoorva Vashisth, Damon Conover, Aniket Bera

Manav Kulshrestha, S. Apoorva Vashisth, and Aniket Bera are with the IDEAS Lab, Department of Computer Science, Purdue University, West Lafayette, IN, USA. · Damon Conover is with the DEVCOM Army Research Laboratory, Adelphi, MD, USA.

Robotic Grasping Zero-Shot Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

Task-Oriented Grasping (TOG) has emerged as an important skill for purposeful robotic manipulation. Given the task of using a knife to "slice bread," a robot must choose to grasp from the handle, whereas for "hand[ing] the knife over," it is preferable for the handle to remain unobstructed for safe handoff. Current approaches for TOG either focus on 2D planar grasping, remain limited by their reliance on task-agnostic grasp generators, or are constrained by reliance on finite, manually-annotated, task-specific grasp datasets that cannot cover all possible grasp-task pairs. To that end, we present TGrasp6D, a vision-language based approach for generating task-oriented 6DoF grasps in a zero-shot manner. Given a task and a single RGB-D image of an associated object, TGrasp6D reasons about the appropriate semantic affordance for the given task and object geometry to produce an initial task- informed prior. This generated prior is further refined via 6DoF pose optimization for collision avoidance while preserving task consistency. Our approach achieves state-of-the-art success rate of 94% on the TaskGrasp dataset, with the best baseline at 74%. We also deploy our approach on a Franka Research 3 robot to demonstrate zero-shot generalization to real world objects

Abstract

Robotic Grasping

Explore similar work

TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping