cs.ROSep 24, 2026

GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin

Authors: Daniel J. Evans, Yinlong Dai, Simon Stepputtis, Dylan P. Losey

Organizations: Collab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061. · TEA Lab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As robots transition from structured factory settings into homes, they are required to interact with an ever-increasing variety of objects. Many tasks require grasping, and often it is not sufficient to just pick up the target object. Consider a task like "pouring coffee" --- to facilitate the subsequent pouring, the robot should grasp the mug by its handle. Existing learning-based approaches for grasping either find robust and collision-free grasps that are largely agnostic to the task (e.g., picking up the mug by its rim), or leverage foundation models to propose task-appropriate grasp locations that lack fine-grained physical grounding (e.g., reaching for and missing the handle). In this work, we bridge these approaches with a real-to-sim-to-real framework. Based on a single RGB-D observation, we construct a digital twin of the environment, query a large foundation model to propose grasps that align with the object's affordances and task description, and then optimize the proposals to ensure robustness and plausibility before executing the result on the real robot. Our key insight is that the grasp proposals of the foundation model should be regarded as semantic priors that serve as seeds for local, gradient-free optimization. We leverage Bayesian optimization with Thompson sampling to draw batches of nearby poses, which are subsequently evaluated in parallel under domain-randomized physics rollouts. The resulting grasp is both task-oriented and physically feasible for execution by the robot arm. Our full zero-shot real-world transfer only takes a few minutes and improves task-oriented grasping success by up to 33% as compared to other state-of-the-art pipelines. Our code is available here: https://github.com/VT-Collab/GraspTwin/

Figures & tables

Figure 1: We present GraspTwin , a zero-shot method for identifying grasps that are both task-oriented and physically feasible. Given a single RGB-D image of the scene, GraspTwin constructs a digital twin. Within that digital twin we query foundation models to propose grasps that facilitate the downstream task. These semantically-aware grasps seed local optimization while enforcing physical constraints (e.g., unintended collisions) and domain randomization. The resulting grasp pose is then executed by the real arm.
Figure 2: Overview of GraspTwin . (Left) We convert from real to sim by constructing a digital twin from an RGB-D image of the robot’s environment. We estimate the scene objects θ^\hat{\theta} and identify θ^1\hat{\theta}_{1} , the target object for grasping. (Middle) We then render the target object θ^1\hat{\theta}_{1} from multiple perspectives, and use a VLM conditioned on the task τ\tau to select task-oriented grasp positions PVLM(i)\mathbf{P}_{\mathrm{VLM}}^{(i)} and an approach direction RVLM\mathbf{R}_{\mathrm{VLM}} . (Right) These proposals become the initial guesses for localized grasp optimization. We simulate grasp poses T\mathbf{T} in the digital twin with object perturbations Δθ\Delta\theta . Batch Bayesian optimization with Thompson sampling enables parallel evaluations for gradient-free optimization. The minimum-cost grasp pose T∗\mathbf{T}^{*} is executed by the real robot.
#TaskSourceSimReal
1Hand me the knifeC✓✓
2Cut the breadA,B,C✓✓
3Grab the mug by the handleA,B,C✓✓
4Flip the pancake with the spatulaB,C✓✓
5Squeeze the mustard out the bottleC✓✓
6Grab a wet wipe from the packA✓✓
Table I: Tasks used in our experiments. Most tasks were taken from related works A [ 19 ] , B [ 22 ] , and C [ 2 ] . All 30 tasks were used in our simulated testing (Table II ), and a subset of 10 tasks was used for real world evaluations (Figure 4 ).
Metric [%]LERF-TOGO [ 19 ]GraspMolmo [ 3 ]Ours-OOurs-GGraspTwin
Task-Oriented11.3±1.511.3\pm 1.553.1±2.453.1\pm 2.475.8±2.075.8\pm 2.051.3±2.451.3\pm 2.481.1±1.8\textbf{81.1}\pm 1.8
Lift Object9.1±1.49.1\pm 1.448.0±2.448.0\pm 2.428.9±2.128.9\pm 2.148.0±2.448.0\pm 2.454.4±2.3\textbf{54.4}\pm 2.3
Collision Free35.6±2.335.6\pm 2.376.4±2.076.4\pm 2.063.8±2.363.8\pm 2.387.3±1.687.3\pm 1.695.6±1.0\textbf{95.6}\pm 1.0
Overall5.8±1.15.8\pm 1.133.1±2.233.1\pm 2.222.9±2.022.9\pm 2.033.8±2.233.8\pm 2.251.6±2.4\textbf{51.6}\pm 2.4
Table II: Grasp performance in simulated tasks from Table I . Results are averaged over 30 tasks, each evaluated across 3 base scenes with five object configurations per scene. This yields 450 grasp attempts per method. The values are grasp success % in the form avg±semavg\pm sem . All pairwise comparisons between GraspTwin and the baselines are statistically significant ( p<.05p<.05 ).
Figure 3: Examples of real-world grasps performed using GraspMolmo [ 3 ] (top row) and GraspTwin (bottom row). Each column shows a different task.
Figure 4: Grasp performance in the real-world tasks from Table I . Results are averaged over 10 tasks with 4 trials per method. Error bars show standard error, and an ∗* denotes statistical significance ( p<.05p<.05 ). The solid bars show real-world evaluation, and the hashed bars show performance on the same subset of tasks in simulation. We find that GraspTwin significantly outperforms GraspMolmo in real-world grasp success, and that the gap between GraspTwin’s simulated and real-world performance is not significant.
More in this topic

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 302 papers

Explore similar work

CardsList
Oct 14, 2026

TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping

Task-Oriented Grasping (TOG) has emerged as an important skill for purposeful robotic manipulation. Given the task of using a knife to "slice bread," a robot must choose to grasp from the handle, whereas for "hand[ing] the knife over," it is preferable for the handle to remain unobstructed for safe handoff. Current approaches for TOG either focus on 2D planar grasping, remain limited by their reliance on task-agnostic grasp generators, or are constrained by reliance on finite, manually-annotated, task-specific grasp datasets that cannot cover all possible grasp-task pairs. To that end, we present TGrasp6D, a vision-language based approach for generating task-oriented 6DoF grasps in a zero-shot manner. Given a task and a single RGB-D image of an associated object, TGrasp6D reasons about the appropriate semantic affordance for the given task and object geometry to produce an initial task- informed prior. This generated prior is further refined via 6DoF pose optimization for collision avoidance while preserving task consistency. Our approach achieves state-of-the-art success rate of 94% on the TaskGrasp dataset, with the best baseline at 74%. We also deploy our approach on a Franka Research 3 robot to demonstrate zero-shot generalization to real world objects

Manav Kulshrestha, Apoorva Vashisth, Damon Conover +1

Manav Kulshrestha, S. Apoorva Vashisth, and Aniket Bera are with the IDEAS Lab, Department of Computer Science, Purdue University, West Lafayette, IN, USA. · Damon Conover is with the DEVCOM Army Research Laboratory, Adelphi, MD, USA.

Robotic GraspingZero-Shot Robotic ManipulationVLMs for Robotics
AskPDF
Mar 14, 2026

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

Open-world object manipulation remains a fundamental challenge in robotics. While Vision-Language-Action (VLA) models have demonstrated promising results, they rely heavily on large-scale robot action demonstrations, which are costly to collect and can hinder out-of-distribution generalization. In this paper, we propose an explicit-world-model-based framework for open-world manipulation that achieves zero-shot generalization by constructing a physically grounded digital twin of the environment. The framework integrates open-set perception, digital-twin reconstruction, sampling and evaluation of interaction strategies. By constructing a digital twin of the environment, our approach efficiently explores and evaluates manipulation strategies in physic-enabled simulator and reliably deploys the chosen strategy to the real world. Experimentally, the proposed framework is able to perform multiple open-set manipulation tasks without any task-specific action demonstrations, proving strong zero-shot generalization on both the task and object levels. Project Page: https://bojack-bj.github.io/projects/thesis/

Xiaotong Li, Gang Chen, Javier Alonso-Mora

Department of Cognitive Robotics (CoR), Delft University of Technology, 2628CD Delft, The Netherlands.

Zero-Shot Robotic ManipulationRobotic ManipulationWorld Models for Robotics
AskPDF
Sep 16, 2026

OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping

Dexterous grasp synthesis has advanced rapidly in generating stable and physically plausible hand poses, but real-world manipulation requires grasps that preserve the function implied by the task. We study open-vocabulary task-oriented dexterous grasp generation, where a robot must infer functional intent from free-form language, ground it in multi-view visual observations and object geometry, and generate an executable high-degree-of-freedom grasp. We present OpenDexGrasp, a unified data and generative modeling framework for this setting. OpenDexVerse provides dual-source supervision organized by the Coverage-to-Alignment (C2A) Recipe: OpenDex-Scale offers large-scale semantic and geometric coverage through automatic grasp synthesis and vision-language annotation, while OpenDex-Align supplies high-quality embodied alignment through human teleoperation and category-level transfer. OpenDexGrasp learns a shared perception-action latent representation that couples open-vocabulary vision-language context with dexterous action generation. Affordance grounding and grasp generation provide complementary supervision over this latent space, enabling direct generation of task-consistent dexterous grasps without a separate affordance-to-pose inference stage. Extensive simulation and real-robot experiments demonstrate improved functional alignment, physical feasibility, generalization to unseen categories, and real-world execution success. Additional details and videos are available at https://opendexgrasp.github.io/.

Jiyao Zhang, Junhan Wang, Tianyu Wang +4

CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1

Robotic GraspingDexterous ManipulationLanguage-Conditioned Robot ManipulationRobot Manipulation Datasets
AskPDF