GAP: Generalized Adaptation Prior for Dexterous Piano Playing
Organizations: Center for X-Mechanics, Zhejiang University, Hangzhou, China. · Mirrorme Technology Co., Ltd., Shanghai, China. · ZJU-Hangzhou Global Scientific and Technology Innovation Center, Hangzhou, China.
Abstract
Dexterous hands enable versatile manipulation, but policies trained in simulation often degrade on hardware owing to non-idealities, and the high-dimensional action space makes real-world adaptation costly. Piano playing, which requires precise and coordinated multifinger motion, offers a representative testbed. We propose GAP (Generalized Adaptation Prior for Dexterous Piano Playing), a sample-efficient adaptation framework that learns how to adapt, rather than how to act, during simulation pretraining and transfers this adaptation prior to hardware. Between repeated executions, GAP uses feedback to adjust the actions of the next one: a gradient guidance network learns from paired executions how adjusting actions affects playing performance, and its gradients guide these adjustments. In simulation, GAP yields approximately 2.4 times the F1 scores improvement of the existing fast-adaptation baseline for dexterous piano playing under the same interaction budget. The learned prior also generalizes to musical pieces held out from pretraining. On a physical dexterous hand, the same simulation-pretrained prior improves F1 scores by 3.6%-23% across three pieces of varying difficulty after approximately 7 min of adaptation each. These results suggest a practical route to reduce the real-world interaction needed to deploy dexterous hands.
Figures & tables
| Perturbation | Hand (18 joints) | Arm (3 joints) |
|---|---|---|
| Per paired execution | ||
| Zero offset | rad † | — |
| Damping scale | ||
| Actuator delay | ms | ms |
| Action perturbation | rad | m |
| Per time step | ||
Robot Policy Adaptation
Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.
See all 186 papers