MDPO: Multi-constraint Dexterous Policy Optimization with Reinforcement Learning
Authors: Ao Zhou, Zongyou Yu, Ke Wu, Tao Xu, Junqiao Zhao, Wenchao Ding, Alois Knoll, Guang Chen
Organizations: Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Fudan University, Shanghai, China · TARS Robotics, Shanghai, China · Technical University of Munich, Munich, Germany
Dexterous manipulation requires high-dimensional control policies to accomplish complex contact-rich tasks while simultaneously satisfying multiple safety requirements, such as joint limits, unsafe finger configurations, arm collisions, and unstable object interactions. Constrained reinforcement learning (RL) provides a principled framework for incorporating these requirements into dexterous manipulation policy learning. However, existing constrained RL methods for dexterous manipulation primarily focus on cumulative cost constraints and struggle to simultaneously handle multiple heterogeneous constraints during dexterous policy learning. In this work, we propose Multi-constraint Dexterous Policy Optimization (MDPO) for safe dexterous manipulation. We first formulate dexterous manipulation as a multi-constraint policy optimization problem involving both cumulative cost constraints and state-wise cost constraints. Based on this formulation, we integrate multi-constraint projection into the dexterous manipulation policy optimization process. Specifically, we employ constraint-specific policy projections to map the intermediate policy that maximizes task performance onto the feasible regions defined by the cumulative cost constraint and the state-wise cost constraint. Extensive experiments on ReDMan, including safe joint control, finger safety, and grasping tasks, demonstrate that MDPO achieves superior constraint satisfaction while maintaining competitive task performance compared with the baselines. Our code is available at this link.
Figures & tables
Fig. 1: Relationship between the cumulative-cost and state-wise-cost safe regions. (a) In the general case, the two safe regions partially overlap, and their intersection defines the feasible policy region satisfying both constraints. (b) When Dˉ≤(1−γ)cˉ , the state-wise-cost safe region is contained within the cumulative-cost safe region, such that satisfying the state-wise constraint also guarantees the cumulative constraint.
Fig. 2: Performance comparison of MDPO with LAG, P3O, and CRPO on five dexterous manipulation tasks in ReDMan. From top to bottom, the rows correspond to Joint , Finger , Armjoint , Armfinger , and Grasp , while the columns report success rate, cost of episode, cost rate, and reward of episode, respectively. The solid curves denote the mean performance over three random seeds, the shaded regions indicate the corresponding standard deviation, and the black dashed lines represent the cumulative cost limits.
Hyper-parameters
Setting
Hyper-parameters
Setting
Discount factor γ
0.998
GAE trace parameter λGAE
0.95
Clipping coefficient ϵ
0.2
Projection clipping coefficient ϵP
0.2
Stage-2 KL coefficient κ0
0.75
State-wise risk quantile q
0.95
Initial cumulative coefficient zC0
1e-4
Initial state-wise coefficient zD0
1e-5
Cumulative adaptation rate ηC
3e-3
State-wise adaptation rate ηD
6e-5
Cumulative upper bound zCmax
0.5
State-wise upper bound zDmax
0.02
TABLE I: The hyper-parameters of the MDPO algorithm model.
Fig. 3: Ablation study of MDPO on the Joint and Finger tasks. MDPO _ Cumulative and MDPO _ State retain only the cumulative-cost and state-wise safety corrections, respectively. From top to bottom, the rows report success rate, cost of episode, cost rate, and reward of episode.
Fig. 4: Real-world deployment of MDPO on the Grasp task. Left: grasping execution on a Tianji robotic arm equipped with a WujiHand dexterous hand. Right: the corresponding Grasp task in simulation.
Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.
Haoyu Wang, Siyuan Qian, Yanjun Li +4
School of Computer Science, Peking University · AI2 Robotics
Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex contact-rich dynamics. Policies trained purely from demonstrations often suffer from compounding errors during deployment and require large amounts of expert data to achieve reliable performance. To move beyond the limitations of demonstration data, in this work, we propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment. First, DexPIE enables effective exploration coverage through a dexterous-hand-adapted intervention system and multi-stage DAgger-style data collection across initial and intermediate task stages. Meanwhile, we enhance consistency between training and inference to reduce the distribution shift between rollouts and demonstration data, better aligning rollout behavior with demonstrations, allowing the critic to learn a value function induced by a more consistent underlying policy. Together, these components provide reliable supervision for policy evaluation. Finally, DexPIE improves the policy through conditioning on a continuous optimality indicator, allowing the policy to leverage the quality of data in a more fine-grained manner. Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37.3% improvement in success rate over the demonstration-based reference policy, outperforming all baseline methods and demonstrating stronger robustness. The source code and dataset will be made publicly available.
Real-world reinforcement learning (RL) offers a promising route to dexterous manipulation policies that can adapt directly from physical interaction, but learning is hindered by inefficient early exploration and costly failures. We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions. A small set of MPC trajectories is first used to populate an offline replay buffer and to pretrain the actor and critic. During online learning, MPC intermittently guides data collection while an off-policy Soft Actor-Critic learner trains from both prior MPC experience and newly collected physical interaction, with control gradually transitioning to the learned policy. On continuous in-hand rotation with a 16-DoF Allegro hand, initialized from 20 MPC trajectories collected in 12 minutes on hardware, the policy reaches 100% success after 7 minutes of online RL, with about three object drops on average during training. After 20 minutes of online learning, the policy achieves more than five times the rotation speed of the MPC controller and completes 1000 consecutive rotations without a drop. Ablations show complementary benefits from MPC-based pretraining, retained MPC experience, and online MPC guidance, while additional experiments demonstrate rapid adaptation to new object geometries and successful goal-conditioned reorientation.
Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang +3
Honda Research Institute USA, San Jose, CA, USA · Skild AI, San Mateo, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA