Organizations: Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden. · Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology, Shenzhen, China.
Building generalist robots capable of performing functional grasping in everyday, open-world environments remains a significant challenge due to the vast diversity of objects and tasks. Existing methods are either constrained to narrow object/task sets or rely on prohibitively large-scale data collection to capture real-world variability. In this work, we present an alternative approach, GraspDreamer, a method that leverages human demonstrations synthesized by visual generative models (VGMs) (e.g., video generation models) to enable zero-shot functional grasping without labor-intensive data collection. The key idea is that VGMs pre-trained on internet-scale human data implicitly encode generalized priors about how humans interact with the physical world, which can be combined with embodiment-specific action optimization to enable functional grasping with minimal effort. Extensive experiments on the public benchmarks with different robot hands demonstrate the superior data efficiency and generalization performance of GraspDreamer compared to previous methods. Real-world evaluations further validate the effectiveness on real robots. Additionally, we showcase that GraspDreamer can (1) be naturally extended to downstream manipulation tasks, and (2) can generate data to support visuomotor policy learning.
Figures & tables
Fig. 1: GraspDreamer leverages human demonstrations synthesized by visual generative models (VGMs), such as video generation models, to enable zero-shot functional grasping. (a) Conceptual illustration of GraspDreamer. (b) Example generated human demonstrations and the corresponding executed dexterous functional grasps.
Fig. 2: An overview of GraspDreamer. The pipeline consists of three stages: (a) Human demonstration generation with VGMs, (b) Human hand motion extraction and optimization, and (c) Human-to-Robot functional retargeting and execution.
Method
Object Generalization
Task Generalization
Success
PIA
Intent
Success
PIA
Intent
LAN-Grasp
68.2%
92.8%
1.9
65.3%
88.5%
2.0
GraspGPT
71.4%
83.4%
2.8
73.4%
81.4%
2.5
FoundationGrasp
73.5%
86.8%
2.9
74.2%
82.0%
2.8
GraspMolmo
77.4%
89.1%
3.3
76.7%
90.1%
3.5
GraspDreamer (Ours)
78.6%
86.1%
3.6
79.5%
85.2%
3.8
TABLE I: Quantitative evaluation on the TaskGrasp dataset.
Object
Success (Gripper)
Object
Success (Dex)
Watering Pot
8/10
Chip Can
7/10
Box
7/10
Toy
9/10
Brush
6/10
Box
8/10
Mug
6/10
Water Bottle
7/10
Power Drill
9/10
Wine Glass
6/10
Basket
7/10
Milk Bottle
8/10
TABLE III: Quantitative results of real-robot experiments: parallel-jaw grasping (left) and dexterous grasping (right).
Fig. 3: Qualitative results on the TaskGrasp dataset. The predicted grasp is shown in blue, and the top-3 matched grasps from the dataset are visualized in green, brown, and red, respectively.
Task Description
Stage
Success
Generation
Grasp
Manipulation
Take Flower from Vase
9/10
7/10
7/10
70%
Open Pot
7/10
6/10
5/10
50%
Pull Tissue
8/10
8/10
8/10
80%
TABLE IV: Quantitative evaluation on three short-horizon manipulation tasks.
Fig. 5: Ablation studies on system components (left) and generation models (right).
Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: https://grasping.io/
Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu +5
New York University · Tsinghua University · University of Michigan
Dexterous grasp synthesis has advanced rapidly in generating stable and physically plausible hand poses, but real-world manipulation requires grasps that preserve the function implied by the task. We study open-vocabulary task-oriented dexterous grasp generation, where a robot must infer functional intent from free-form language, ground it in multi-view visual observations and object geometry, and generate an executable high-degree-of-freedom grasp. We present OpenDexGrasp, a unified data and generative modeling framework for this setting. OpenDexVerse provides dual-source supervision organized by the Coverage-to-Alignment (C2A) Recipe: OpenDex-Scale offers large-scale semantic and geometric coverage through automatic grasp synthesis and vision-language annotation, while OpenDex-Align supplies high-quality embodied alignment through human teleoperation and category-level transfer. OpenDexGrasp learns a shared perception-action latent representation that couples open-vocabulary vision-language context with dexterous action generation. Affordance grounding and grasp generation provide complementary supervision over this latent space, enabling direct generation of task-consistent dexterous grasps without a separate affordance-to-pose inference stage. Extensive simulation and real-robot experiments demonstrate improved functional alignment, physical feasibility, generalization to unseen categories, and real-world execution success. Additional details and videos are available at https://opendexgrasp.github.io/.
Jiyao Zhang, Junhan Wang, Tianyu Wang +4
CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1
Multifingered grasping is a crucial robotic skill, but current deep-learning grasp planners often struggle to generalize to new objects because they are trained on limited, object-specific datasets. We introduce a fundamentally different approach, grounded in the observation that the gripper and the object share identical surface geometry at their mutual contact points. We propose GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation, a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution, enabling the efficient sampling of valid grasp configurations without relying on object-specific training data. We show that by introducing object features only at inference time, our model can effectively retrieve admissible contact areas that are compatible with the gripper's capabilities. We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios, demonstrating its effectiveness with different grippers from the literature. Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%. It offers significantly faster processing when generating numerous grasps, while matching the performance of leading approaches specifically trained on this dataset. Unlike these methods, our approach does not rely on object-specific training data, highlighting the advantages of object-agnostic learning. It effectively addresses the generalization challenges faced by traditional data-driven grasp planners. Code and videos are available on our project website https://cea-list.github.io/goagweb/ .
Julien Merand, Boris Meden, Mathieu Grossard +1
Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Ecole Centrale de Lyon, CNRS, LIRIS, UMR5205, Institut Universitaire de France (IUF), F-69130 Ecully, France