Task-oriented grasping for dexterous robots using postural synergies and reinforcement learning

Dimitrios Dimou; José Santos-Victor; Plinio Moreno

cs.ROFeb 24, 2026

Task-oriented grasping for dexterous robots using postural synergies and reinforcement learning

Authors: Dimitrios Dimou, José Santos-Victor, Plinio Moreno

Organizations: Institute for Systems and Robotics Instituto Superior Tecnico Lisboa, Portugal

Abstract

In this paper, we address the problem of task-oriented grasping for humanoid robots, emphasizing the need to align with human social norms and task-specific objectives. Existing methods, employ a variety of open-loop and closed-loop approaches but lack an end-to-end solution that can grasp several objects while taking into account the downstream task's constraints. Our proposed approach employs reinforcement learning to enhance task-oriented grasping, prioritizing the post-grasp intention of the agent. We extract human grasp preferences from the ContactPose dataset, and train a hand synergy model based on the Variational Autoencoder (VAE) to imitate the participant's grasping actions. Based on this data, we train an agent able to grasp multiple objects while taking into account distinct post-grasp intentions that are task-specific. By combining data-driven insights from human grasping behavior with learning by exploration provided by reinforcement learning, we can develop humanoid robots capable of context-aware manipulation actions, facilitating collaboration in human-centered environments.

Figures & tables

Panel of Figure 1 — Fig. 1: Example of different execution of a grasp according to the post-grasp intention as captured in the dataset presented in [ 1 ] . In the left figure, the person grasps the hammer in order to use it, in the right it grasps it in order to hand it over to another person.

Fig. 2: Example grasping targets for hammer extracted from the ContactPose dataset [ 1 ] .

Fig. 3: Proposed agent structure for task-oriented grasping.

Fig. 4: Rewards for training policies with 1) full joint control, 2) PCA synergy space, and 3) VAE synergy space. The thick line is the average among the two seeds and the shaded part denotes the standard deviation.

Method	Success Rate
Joint action space	66%
PCA action space	71%
VAE action space (ours)	83%

TABLE I: Average success rate for each policy for 5000 trials.

Fig. 6: Rewards for policies that use synergy models of different latent dimensions.

Number of latent dimensions	Success Rate
1	59%
2	83%
3	68%
4	84%
5	81%

TABLE II: Average success rate for policies with synergy models of different latent dimensions.

Fig. 7: Distances of the robot’s hand to each grasp target for each task. The results are for a 1000 grasp trials performed for each object and each post-grasp intention performed using our proposed approach.

Method	Success Rate
Policy without object category observation	82%
Policy with object category observation	83%

TABLE III: Average success rate for each policy for 5000 trials.

Fig. 8: Clustering results for grasp points on hammer object.

Fig. 9: Distances of the robot’s hand to each grasp target for each task. The results are for a 1000 grasp trials performed for each object and each post-grasp intention performed using a model that does not take as observation the object’s category.

Fig. 10: The first row of depicts grasps using the joint angles as action space, while the second row depicts grasps from the policy that uses the synergy space as action space

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 311 papers

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness
Apr 28, 2026Ethan Foong, Yunshuang Li, Hao Jiang +2Robotic Grasping Dexterous Manipulation
Department of Computer Science at Northwestern University · Viterbi School of Engineering at the University of Southern California
Learning Dexterous Grasping from Sparse Taxonomy Guidance
Apr 5, 2026Juhan Park, Taerim Yoon, Seungmin Kim +10RL for Robotics Robotic Grasping
Department of Artificial Intelligence, Korea University, Seoul, Korea · Korea University of Technology and Education, Cheonan, Korea · Naver AI Lab, Korea +4
OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping
Sep 16, 2026Jiyao Zhang, Junhan Wang, Tianyu Wang +4Robotic Grasping Dexterous Manipulation
CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1

Apr 28, 2026J/K move · Enter open · S save

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

Ethan Foong, Yunshuang Li, Hao Jiang, Gaurav S. Sukhatme+1

Department of Computer Science at Northwestern University · Viterbi School of Engineering at the University of Southern California

Robotic Grasping Dexterous Manipulation Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Dexterous robot hands offer rich opportunities for multifunctional manipulation, where a robot must execute multiple skills in sequence while maintaining control over previously grasped objects. Most prior work in dexterous manipulation focuses on single-object, single-skill tasks. In contrast, our insight is that many sequential tasks require resource-aware grasps that conserve fingers for future actions. In this paper, we study sequential grasp-conditioned dexterous manipulation, where a robot first grasps an object and then performs a second, distinct manipulation subtask while preserving the initial grasp. We introduce HANDFUL, a learning framework that models finger usage as a limited resource and encourages exploration of resource-aware grasps through finger-level contact rewards. These grasps are subsequently selected for downstream tasks via curriculum-based policy learning. We further propose HANDFUL-Bench, a simulation benchmark that introduces sequential dexterous manipulation tasks across multiple secondsubtask objectives, including pushing, pulling, and pressing, under a shared grasp-conditioned setup. Extensive simulation results demonstrate that prioritizing resource-aware grasps improves second-subtask success and robustness compared to a baseline that greedily optimizes the initial grasp before attempting the second subtask. We additionally validate our approach on a real dexterous LEAP hand. Together, this work establishes resource-aware grasp planning as a key principle for multifunctional dexterous manipulation. Supplementary material is available on our website: https://handful-dex.github.io.

Abstract

Robotic Grasping

Explore similar work

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

Learning Dexterous Grasping from Sparse Taxonomy Guidance

OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping