GAP: Generalized Adaptation Prior for Dexterous Piano Playing
Authors: Jinyi Chen, Hongyi Li, Lei Jiang, Xianwei Liu, Yanhong Liang, Yulin Chen, Yucheng Tao, Yongbin Jin, Hongtao Wang
Organizations: Center for X-Mechanics, Zhejiang University, Hangzhou, China. · Mirrorme Technology Co., Ltd., Shanghai, China. · ZJU-Hangzhou Global Scientific and Technology Innovation Center, Hangzhou, China.
Dexterous hands enable versatile manipulation, but policies trained in simulation often degrade on hardware owing to non-idealities, and the high-dimensional action space makes real-world adaptation costly. Piano playing, which requires precise and coordinated multifinger motion, offers a representative testbed. We propose GAP (Generalized Adaptation Prior for Dexterous Piano Playing), a sample-efficient adaptation framework that learns how to adapt, rather than how to act, during simulation pretraining and transfers this adaptation prior to hardware. Between repeated executions, GAP uses feedback to adjust the actions of the next one: a gradient guidance network learns from paired executions how adjusting actions affects playing performance, and its gradients guide these adjustments. In simulation, GAP yields approximately 2.4 times the F1 scores improvement of the existing fast-adaptation baseline for dexterous piano playing under the same interaction budget. The learned prior also generalizes to musical pieces held out from pretraining. On a physical dexterous hand, the same simulation-pretrained prior improves F1 scores by 3.6%-23% across three pieces of varying difficulty after approximately 7 min of adaptation each. These results suggest a practical route to reduce the real-world interaction needed to deploy dexterous hands.
Figures & tables
Fig. 1: GAP pretrains a generalized adaptation prior in simulation on multiple pieces under different perturbations, including joint zero offsets, damping, actuator delays, and sensor noise (top), and uses it for fast adaptation on the physical hand (bottom). Green, red, and orange keys denote correct, incorrect, and missed keystrokes.
Fig. 2: Overview of GAP. Both stages share one adaptation process: gradients of the reward change modeled by GGN Fθ iteratively update the action corrections, and paired executions of the initial and adjusted trajectories supervise GGN. The execution and collection module is instantiated in parallel simulation in Stage I and on the physical hand in Stage II.
Perturbation
Hand (18 joints)
Arm (3 joints)
Per paired execution
Zero offset
±0.08 rad †
—
Damping scale
[0.8,1.25]
Actuator delay
[2.5,30] ms
[5,50] ms
Action perturbation
±0.005 rad
±0.005 m
Per time step
TABLE I: Simulation perturbations
Fig. 3: Comparison of GGN design choices. (a) Bidirectional versus unidirectional local observation windows. (b) Different correction frequencies. ΔF10.5 is the median F1 improvement across 512 parallel environments; bars and error bars show the mean and standard deviation over 5 seeds.
Fig. 4: Simulation comparison of GAP. Curves show mean whole-piece F1 over 5 random seeds; shading spans the middle three seeds after removing the best and worst. Time denotes simulated environment interaction time, excluding training and scheduling, with 0 denoting the initial trajectory. Dashed lines mark the GAP few-shot F1 and the interaction required by GAP and GAP w/o pretraining to reach it.
Fig. 5: F1 scores at each whole-piece evaluation during real-world adaptation of three representative pieces. Evaluations 0 and 1 denote the initial and few-shot trajectories, respectively. Time includes execution, training, and scheduling. Dashed lines show direct-deployment F1 without added offsets or adaptation.
Fig. 6: Framewise keystroke results for the three pieces at three stages of real-world adaptation. Columns show the initial, few-shot, and converged trajectories, with F1 and cumulative wall-clock time in each panel title. Green, red, and orange denote correct, incorrect, and missed keystrokes; gray bands denote black keys.
Fig. 7: Real-world adaptation by GAP at 5 instants in Beethoven’s Symphony No. 5. Each column compares the initial (top) and adapted (bottom) keystrokes at one instant. Green, red, and orange denote correct, incorrect, and missed keystrokes; examples include off-center, overly light, and mistimed strikes.
Mastering dexterous manipulation with multi-fingered hands has been a grand challenge in robotics for decades. Despite its potential, the difficulty of collecting high-quality data remains a primary bottleneck for high-precision tasks. While reinforcement learning and simulation-to-real-world transfer offer a promising alternative, the transferred policies often fail for tasks demanding millimeter-scale precision, such as bimanual piano playing. In this work, we introduce HandelBot, a framework that combines a simulation policy and rapid adaptation through a two-stage pipeline. Starting from a simulation-trained policy, we first apply a structured refinement stage to correct spatial alignments by adjusting lateral finger joints based on physical rollouts. Next, we use residual reinforcement learning to autonomously learn fine-grained corrective actions. Through extensive hardware experiments across five recognized songs, we demonstrate that HandelBot can successfully perform precise bimanual piano playing. Our system outperforms direct simulation deployment by a factor of 1.8x and requires only 30 minutes of physical interaction data.
Amber Xie, Haozhi Qi, Dorsa Sadigh
Stanford University · Amazon FAR (Frontier AI & Robotics)
Reinforcement learning can train bimanual dexterous hands to play piano in physics simulation with high note accuracy, but for high-DoF dexterous hands, relying solely on task rewards or IK inversion often leads to unnatural postures and joint overextension. We propose Adversarial Posture Regularization (APR). It avoids expensive, song-aligned expert demonstration data and instead uses a small amount of casual human playing data. By matching the distribution of the posture of the policy with the human prior through an adversarial objective, APR encourages more human-like hand shapes. Meanwhile, we collect and release unstructured hand motion data of piano playing using a consumer-grade Meta Quest 3, and retarget the key motion information to the Shadow Hand. Finally, we achieve significantly better performance than prior methods on all three human-likeness metrics (cPSI, BSE, and FAC) as well as in visual quality. Project repository: https://github.com/APRProject/APRPianist.
Bin Qiu, Yanming Shao, Guanyu Cai +1
Shanghai Jiao Tong University · The University of Hong Kong · Shanghai AI Laboratory
Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.