While multi-fingered dexterous robot hands have the potential to provide human-level dexterity, using them for functional manipulation remains difficult because task success depends on precise contact, coordinated arm-hand motion, and reactive recovery from execution errors. Precise contact depends on complex contact dynamics that are difficult to model accurately in simulation, motivating real-world learning. However, real-world demonstrations in the full arm-hand control space are costly and noisy because each trajectory must coordinate arm motion with high-DoF multi-finger contact. Small execution deviations can further cause slipped contact or premature force application, making recovery from off-nominal contact states important for robust execution. To deal with these challenges, we propose HiFun, a hierarchical real-world RL framework for functional dexterous manipulation tasks. Our key insight is to decouple the control complexity into a functional hand skill for precise contact execution and an arm-hand coordination policy for arm control and hand-skill activation. This design enables effective real-time human intervention for reactive recovery learning and efficient data collection with Dynamic Movement Primitive (DMP) augmentation. Across six real-world tasks on two dexterous hands, HiFun achieves 98.3% average success within one hour of online training and remains robust to dynamic disturbances. The project page is available at https://hly-123.github.io/HiFun/.
Figures & tables
Figure 1: HiFun is a hierarchical framework for functional dexterous manipulation. It learns contact-aware skills and arm-hand coordination across precise tool-use (A,E), contact-rich rotational manipulation (D), and constrained-space manipulation (B,C,F), achieving 98.3% average success on six real-world tasks with two dexterous hands within one hour of online training.
Figure 2: Overview of HiFun, a hierarchical framework. It learns (1) a residual-RL hand skill for contact-aware execution and (2) a coordination policy for arm motion and skill activation, initialized by DMP rollouts and refined by value-guided fine-tuning. At execution time, it combines arm motion and skill activation with the frozen hand skill for target approach and fine-grained contact.
Task
IL Data Collection (mins)
RL Data Collection (mins)
RL Online Training (mins)
Success Rate (%)
HIL-SERL (Glove+Gello)
HIL-SERL (Ours Interv.)
HiFun (Ours)
Behavior Cloning (BC)
Diffusion Policy (DP)
Sharpa Main Validation Tasks
Power-Drill Actuation
150
34
60
15
62 — 30 ‡
100
15
30
Angle-Spreader Retrieval
60
13
55
–
40 — 0 ‡
100
30
60
Thin-Handle Retrieval
80
17
60
–
28 — 0 ‡
100
35
40
Power-Drill Bit Removal
80
21
22
–
60 — 20 ‡
100
30
36
Table 1: Main results. Success rates are computed over 50 trials. “–” marks HIL-SERL (Glove+Gello) settings where direct arm-hand teleoperation can not consistently provide reliable corrections for contact-rich tasks. ‡ a — b reports arm approach success ( a ) and final task success with the required hand contact ( b ). Averages use final-task success and count “–” as 0.
Figure 3: Finger-level contact recovery. Following collision-induced key reorientation, the hand restores thumb contact without arm re-alignment. Red and green circles mark disturbed and restored contact states, respectively.
Data Source Comparison
Task
Success Rate (%)
Data Collection Time (mins)
(w/ DMP)
(w/ Human)
(w/ DMP)
(w/ Human)
Angle-Spreader Retrieval
100
85
13
60
Power-Drill Bit Removal
100
100
21
80
Component Ablation (Success Rate %)
Task
Ours (Full)
w/o Multi-head
w/o IAW (Uniform BC)
w/o Interv. BC
Table 2: Coordination policy analysis. Top: DMP rollouts vs. human demonstrations for warm-start data. Bottom: ablations of the multi-head critic and intervention advantage weighting (IAW).
Task
Ours ( πskill )
Kinesthetic Replay
DexPilot ∗
AnyTeleop ∗
Power-Drill Actuation
20/20
0/20
18/20
19/20
Angle-Spreader Retrieval
20/20
0/20
7/20
5/20
Thin-Handle Retrieval
20/20
0/20
0/20
0/20
Power-Drill Bit Removal
20/20
5/20
3/20
3/20
Pipette Dispensing
20/20
0/20
20/20
20/20
Thin-Shaft Tool Retrieval
20/20
0/20
0/20
0/20
Table 3: Hand Motion Control Comparison. Success rate over 20 trials.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Trajectory distribution from DMP rollouts and human demonstrations. Starting from one reference trajectory, DMP augmentation randomizes initial arm poses while preserving the demonstrated goal and activation progress. This broadens approach coverage beyond repeated human demonstrations.
Figure 5: Offline pre-training analysis on a representative disassembly task. (a) The total target Q-value converges without value divergence under SAC. (b) Decomposed Q-values reveal a scale gap between arm alignment and hand activation, motivating the multi-head critic.
Figure 6: Task setup and camera views. The figure shows the six task setups (A–F), the workspace regions used for training and evaluation, and the corresponding wrist and side camera observations.
Table 4: Common policy training hyperparameters shared across tasks.
Task
Reward Classifier Data Collection (mins)
DMP Data Collection (mins)
Coordination Policy Training (mins)
Power-Drill Actuation
4
34
60
Angle-Spreader Retrieval
13
13
55
Thin-Handle Retrieval
11
17
60
Power-Drill Bit Removal
7
21
22
Pipette Dispensing
10
14
60
Thin-Shaft Tool Retrieval
7.5
14
20
Appendix
Table 5: Per-task time cost for coordination learning. All times are wall-clock minutes. DMP data collection corresponds to automated rollout collection, and coordination policy training corresponds to online HIL fine-tuning.
Figure 7: Online training curves across six tasks. For each task, intervention rate (upper) and cycle time (lower) are shown as running averages over 20 episodes, comparing our method (Ours) against the HIL-SERL (Ours Interv.) baseline. Lower intervention rate indicates more autonomous execution, while lower cycle time indicates more efficient completion.
Figure 8: Coordination component analysis. (A) Activation localization on Angle-Spreader Retrieval : the x-axis compares HiFun with the one-head-critic variant, and the y-axes report p95 activation error and far-state activation rate computed from TCP-to-target error et with a 2cm pre-contact band. (B) IAW weighting: left shows imitation weight w(s) for low- and high-weight interventions; right shows ΔQ^π=Q^(shigh_w,aˉπ(shigh_w))−Q^(slow_w,aˉπ(slow_w)) , where negative values indicate high-weight corrections at lower-value policy states.
Figure 9: Failure analysis for hand motion control. Representative hand-control outcomes compare the residual hand skill, kinesthetic replay, and DexPilot ∗ across contact-rich manipulation cases.