Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.
Figures & tables
Fig. 1: DexPolicy overview. (A) Simulation and real-robot tasks. (B) Gaussian noise scheduled by training steps; the PPO example does not represent within-trial phases or other methods’ schedules. (C) Paired updates and deterministic Target success (%), baseline to +DexPolicy. Simulation averages five shared objects over three seeds; hardware averages three objects with 20 trials from one model per condition (Tables I – III ). Policies are object-specific, with distinct simulation and hardware embodiments; GRPO uses continuation budgets (Sec. V-C ).
Fig. 2: Simulation settings. (a) The ViViDex benchmark hand approaches and tracks translucent green target poses on representative objects. (b) Hardware-policy training uses RealMan RM75 arm and Inspire/RH56 right-hand models matched to the real robot (Sec. V-D ). These qualitative rollouts provide task context; the five-object comparison in Table I uses banana instead of clamp.
Method
Mustard bottle
Mug
Banana
Sugar box
Tomato can
Overall (5 objects)
PPO baseline
84.0 ± 5.3
25.3 ± 9.5
8.7 ± 13.3
0.0 ± 0.0
42.0 ± 14.7
32.0 ± 5.3
PPO + DexPolicy
94.7 ± 9.2
40.7 ± 17.1
2.3 ± 2.1
0.0 ± 0.0
40.7 ± 9.1
35.7 ± 3.9
GRPO baseline
30.8 ± 53.4
25.3 ± 15.0
3.0 ± 3.9
4.2 ± 4.3
7.0 ± 3.1
14.1 ± 8.5
GRPO + DexPolicy
34.5 ± 56.7
41.7 ± 3.3
3.2 ± 2.8
88.0 ± 12.3
59.8 ± 4.5
45.4 ± 14.4
FPO baseline
51.7 ± 46.0
25.3 ± 34.8
16.3 ± 8.0
91.3 ± 13.3
62.3 ± 54.0
49.4 ± 21.5
FPO + DexPolicy
100.0 ± 0.0
31.0 ± 53.7
17.7 ± 16.6
99.0 ± 1.7
92.7 ± 4.0
68.1 ± 11.4
TABLE I: Target success: mean (%) ± sample SD (percentage points) over three training seeds; Overall: mean over five shared objects within each seed. Eight-object PPO/FPO means: Sec. V-B . PPO/FPO: 5M steps, 100 episodes/model/mode. GRPO: ∼ 5M initialization + 0.762M steps (tomato: 1.016M), 200 episodes/model; seed 0 is developmental except on banana. All methods execute policy means ( σeval=0 ). Bold compares within pairs; budgets differ across methods.
Evaluation
Mustard
Mug
Banana
Sugar
Tomato
Seeds 1/2, σ=0
−44.5
+23.5
+0.2
+80.5
+49.8
All seeds, σ=0.03
+6.3
+13.3
+0.3
+72.0
+43.0
All seeds, σ=0.10
−6.7
+2.7
+0.7
−32.0
−1.7
TABLE II: GRPO Target change (+DexPolicy minus baseline), in percentage points. Seeds 1/2: replication on the original four objects; subset of untuned banana seeds. Episodes/model: 200 at zero noise, 100 otherwise.
Method
Tomato can
Sugar box
Mustard bottle
Target
Lift
Target
Lift
Target
Lift
PPO baseline
10
35
0
5
15
75
PPO + DexPolicy
45
70
0
10
85
95
GRPO baseline
5
35
10
75
15
30
GRPO + DexPolicy
55
90
90
100
45
95
FPO baseline
25
30
20
45
30
60
TABLE III: Deterministic real-robot success rates (%): 20 trials per object–method condition, one model per condition. Bold marks the higher rate within each pair.
Fig. 3: Grasp-and-lift sequences with a RealMan RM75 arm and an Inspire/RH56 right hand. The physical objects match the (a) tomato-can, (b) sugar-box, and (c) mustard-bottle geometries used in simulation. Each row shows seven uniformly sampled frames illustrating task progression in a demonstration video. These sequences illustrate the tasks rather than evaluation completion times.
Fig. 4: Learning histories on the three hardware object geometries, using separate benchmark policies. PPO/FPO show stochastic evaluation returns; GRPO shows continuation rollout returns. Shading: sample SD over three seeds; GRPO includes development seed 0. Dotted lines mark GRPO phase reloads; PPO checkpoints are nominal. Compare conditions within panels: return measurements and budgets differ across methods.
Condition
Reward
Contact
Lift (m)
Learned-std PPO
19.96
0.325
0.02995
Noise schedule only
27.36
0.427
0.03139
Update contraction only
12.35
0.200
0.00293
Both
19.13
0.217
0.00327
TABLE IV: Mustard seed-0 component screen: normalized AUC over 5M steps. Noise uses the initial curve with the 2M jump. Contact is a fraction; lift is height.
Candidate
Reward
Contact
Lift
Pass
Uniform linear
91.9
86.2
110.6
No
Continuous front-loaded
93.8
92.5
95.8
No
No 2M jump
100.3
95.5
108.3
Yes
No final drop
94.3
91.9
78.2
No
TABLE V: Schedule-shape AUC retention (%) relative to the confirmed stagewise reference. Passing requires at least 95% retention of all three mean AUCs and reward retention on at least two of three seeds.
Object
Reward AUC
Wins
Contact AUC
Lift AUC
Mustard bottle
+43.4
3/3
+35.4
+38.2
Mug
+14.0
2/3
+27.4
+7.6
Banana
+25.1
2/3
+20.1
+55.3
Sugar box
+26.1
3/3
+11.3
+144.4
Pitcher base
+29.5
3/3
+38.4
− 13.4
Wood block
+64.2
3/3
+240.8
+45.2
TABLE VI: Three-seed 5M schedule transfer with separately trained object policies. Entries are relative changes (%) in mean normalized AUC; wins count paired reward-AUC gains.
Window / metric
PPO
Scheduled
Δ (%)
Wins
Reward AUC, 0–10M
36.57
47.79
+30.7
3/3
Reward AUC, 5–10M
53.09
66.44
+25.2
3/3
Reward, last 20% of steps
59.51
68.28
+14.7
3/3
Contact AUC, 0–10M
0.620
0.704
+13.5
3/3
Lift AUC, 0–10M (m)
0.0654
0.0731
+11.7
3/3
TABLE VII: Three-seed 10M mustard-bottle confirmation. Entries are seed means; AUC is divided by its training window. Contact is a 0–1 fraction; lift is height in metres. Δ is relative change; wins count paired improvements.
Object
Final reward
AUC Δ (%)
Baseline
Scheduled
Reward
Contact
Lift
Mustard
40.53±6.78
66.33±1.09
+31.3
+14.2
+100.6
Mug
30.05±5.82
34.56±14.64
+0.7
−20.3
−23.0
Sugar
48.08±4.79
63.09±1.19
+0.8
−15.2
−20.2
Tomato
31.11±6.33
50.15±2.28
+38.8
+47.0
+142.1
Clamp
17.62±1.20
20.63±0.83
+8.1
+0.3
+144.7
TABLE VIII: FPO diagnostics (three seeds): final reward mean ± sample SD and relative AUC changes over 0–5M steps. Contact measures count rather than PPO’s fraction; lift measures height.
Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex contact-rich dynamics. Policies trained purely from demonstrations often suffer from compounding errors during deployment and require large amounts of expert data to achieve reliable performance. To move beyond the limitations of demonstration data, in this work, we propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment. First, DexPIE enables effective exploration coverage through a dexterous-hand-adapted intervention system and multi-stage DAgger-style data collection across initial and intermediate task stages. Meanwhile, we enhance consistency between training and inference to reduce the distribution shift between rollouts and demonstration data, better aligning rollout behavior with demonstrations, allowing the critic to learn a value function induced by a more consistent underlying policy. Together, these components provide reliable supervision for policy evaluation. Finally, DexPIE improves the policy through conditioning on a continuous optimality indicator, allowing the policy to leverage the quality of data in a more fine-grained manner. Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37.3% improvement in success rate over the demonstration-based reference policy, outperforming all baseline methods and demonstrating stronger robustness. The source code and dataset will be made publicly available.
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Jusuk Lee, Sungha Kim, Yeonsoo Park +8
Seoul National University · University of Maryland, College Park · All Purpose AI +1
Dexterous manipulation requires high-dimensional control policies to accomplish complex contact-rich tasks while simultaneously satisfying multiple safety requirements, such as joint limits, unsafe finger configurations, arm collisions, and unstable object interactions. Constrained reinforcement learning (RL) provides a principled framework for incorporating these requirements into dexterous manipulation policy learning. However, existing constrained RL methods for dexterous manipulation primarily focus on cumulative cost constraints and struggle to simultaneously handle multiple heterogeneous constraints during dexterous policy learning. In this work, we propose Multi-constraint Dexterous Policy Optimization (MDPO) for safe dexterous manipulation. We first formulate dexterous manipulation as a multi-constraint policy optimization problem involving both cumulative cost constraints and state-wise cost constraints. Based on this formulation, we integrate multi-constraint projection into the dexterous manipulation policy optimization process. Specifically, we employ constraint-specific policy projections to map the intermediate policy that maximizes task performance onto the feasible regions defined by the cumulative cost constraint and the state-wise cost constraint. Extensive experiments on ReDMan, including safe joint control, finger safety, and grasping tasks, demonstrate that MDPO achieves superior constraint satisfaction while maintaining competitive task performance compared with the baselines. Our code is available at this link.
Ao Zhou, Zongyou Yu, Ke Wu +5
Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Fudan University, Shanghai, China +2