OmniPiano: Diverse Dexterous Piano-Playing Challenges for Standard, Robust, Safe, and Multi-Agent RL
Authors: Fan Xu, Fan Yang, Yifan Xu, Ruiyuan Zhang, Yimou Wu, Nuo Yi, Muning Wen, Guang Chen, Costas Spanos, Shangding Gu
Organizations: University of California, Berkeley · Shanghai Jiao Tong University · The Chinese University of Hong Kong, Shenzhen · Japan Advanced Institute of Science and Technology · The Chinese University of Hong Kong · Nanjing University · Tongji University
Dexterous robotic manipulation remains a major challenge in robotics, exemplified by piano playing, which requires coordinated control of fingers and provides a demanding testbed for reinforcement learning (RL). Although previous work established a benchmark for robotic piano playing, it is restricted to two hands and does not support unified evaluation. To address this gap, we introduce OmniPiano, a benchmark that provides one to up to five Shadow Hands and supports standard, robust, safe, and multi-agent RL within a shared piano-playing task family. OmniPiano is developed for scalable and diverse piano-playing tasks with configurable perturbations, explicit safety constraints, and decentralized cooperation settings. Particularly, to facilitate usability and extendability, OmniPiano adopts a highly modular design and provides comprehensive tasks to support RL study of task performance, robustness, safety, and cooperation in dexterous control. With at least 912 task settings and 36 baseline algorithms, OmniPiano further incorporates LLM-based agents to broaden the evaluation scope and reveal new insights. Extensive evaluations show that state-of-the-art RL algorithms and frontier LLM-based agents still struggle on challenging piano-playing tasks, even under standard RL settings, highlighting substantial room for improvement in dexterous control. The code, dataset, tutorial, and demo are available at https://sail-research-lab.github.io/omnipiano/.
Figures & tables
Figure 1: The overview of OmniPiano . A unified benchmark for learning dexterous piano playing.
Feature
RoboPianist
Robust-Gym.
Safety-Gym.
OmniPiano
Action dimension
45
1–30
2–17
23–111
Dexterous piano playing
✓
✗
✗
✓
Robust RL track
✗
✓
✗
✓
Safe RL track
✗
✓
✓
✓
Multi-agent RL track
✗
✓
✓
✓
Unified task setting
✗
✗
✗
✓
Table 1: The comparison of OmniPiano with related reinforcement-learning benchmarks.
Figure 2: The overview of the Robust RL task design. Robustness perturbations are introduced through four independent disruptors acting on the agent–environment interaction loop.
Figure 3: The overview of the safety task design in OmniPiano . Four safety semantics, Joint Range, Actuator Power, Injured Finger, and Hand Collision, define the constrained behaviors. Event, Fraction, and Excess provide alternative measures of constraint violations.
Figure 4: The overview of the multi-agent RL task design. The configuration shown is a typical example. In practice, users can freely configure the MIDI score, number of hands and agents, hand-to-agent assignments, and agent-specific observation and action ranges and overlaps.
Figure 5: Average evaluation reward ( ↑ : top row, higher is better) and mean token usage ( ↓ : bottom row, lower is better) per optimization attempt for LLMs across ten keyframe-optimization iterations and RL methods over 5M training steps in the two-hand setting on three piano pieces.
Figure 6: F1-score comparison of PPO, SAC, DeepSeek-V4.1-Flash, and GPT-6 Astra across eight hand settings on three piano pieces. The details of F1 metric can be found in Appendix D.1.2 .
Figure 7: Periodic evaluation reward of PPO across six robustness channels under Gaussian, uniform, and shift perturbations, with the clean environment shown as a reference.
Figure 8: Effect of hand count on safety RL: training reward and safety cost for 2 to 4 hands. Solid lines denote CUP / PPO-Lag; dashed lines denote unconstrained PPO. The dotted line marks the cost budget. In this task, four hands policies achieve lower rewards and show greater difficulty reducing safety costs than two or three hands policies.
Figure 9: Task-wise training performance of representative MARL algorithms across the five settings. Compared with Base, Scalability and Observability variant generally improve performance, while Heterogeneity and especially Coupling variant make policy learning more difficult. Among the variants, Coupling most clearly differentiates the algorithms.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Song
Steps
NPS
F1 Score
Gymnopédie No. 1
657
1.77
0.88
Twinkle Twinkle (Rousseau)
451
1.51
0.86
Clair de Lune
597
2.65
0.63
French Suite No. 1, Allemande
517
10.89
0.59
French Suite No. 5, Sarabande
962
3.56
0.59
Piano Sonata No. 23, 2nd mov.
313
12.80
0.49
Appendix
Table 2: Task statistics and performance across piano pieces.
Figure 10: Periodic evaluation reward during training for PPO and SAC under Restricted and Unrestricted configurations across the three songs.
Figure 11: F1 of LLMs over ten keyframe-optimization iterations and RL method over 5M steps for the two-hand setting on three piano pieces.
Figure 12: Cross-algorithm robustness comparison on Clair de Lune .
Figure 13: Composability of Gaussian perturbations on Für Elise with three hands.
Figure 14: Additional hand count results for safe RL. Solid lines denote CPO / OnCRPO; dashed lines denote PPO. Training reward and cost show means with ±1 standard deviation across three seeds. The dotted line marks the cost budget.
Figure 15: Overall performance on 2 hand safety tasks: evaluation reward and cost. Shading indicates ±1 standard deviation across available seeds; dotted lines mark task-specific cost budgets.
Figure 16: Overall performance on 3 hand safety tasks: evaluation reward and cost. Shading indicates ±1 standard deviation across available seeds; dotted lines mark task-specific cost budgets.
Figure 17: Overall performance on 4 hand safety tasks: evaluation reward and cost. Shading indicates ±1 standard deviation across available seeds; dotted lines mark task-specific cost budgets.
Figure 18: Key-press F1 on the 2-hand safety tasks. Curves show means with ±1 standard deviation across available seeds.
Figure 19: Key-press F1 on the 3-hand safety tasks. Curves show means with ±1 standard deviation across available seeds.
Figure 20: Key-press F1 on the 4-hand safety tasks. Curves show means with ±1 standard deviation across available seeds.
Figure 21: Sensitivity to the episodic cost limit. Evaluation reward and cost are shown for five budgets; shading indicates ±1 standard deviation across three seeds.
Figure 22: Key-press F1 under different cost limits, shown as means with ±1 standard deviation across three seeds.
Setting
Agent
Hands
Action
Observation
Visible hands
Base
A1
0,1
1–44
1–49
2
A2
2,3
45–88
40–88
1
Observability
A1
0,1
1–44
1–70
2
A2
2,3
45–88
19–88
1
Coupling
A1
0,1
1–70
1–49
2
A2
2,3
19–88
40–88
1
Appendix
Table 3: MARL test settings.
Figure 23: Detailed MARL results across the five OmniPiano cooperation settings. The first row compares task-wise team return for each MARL algorithm, while the second and third rows compare team return and musical F1 across algorithms for each task.
Figure 24: IPPO performance in the Base MARL setting with and without Gaussian observation noise ( σ=0.05 ). Reward remains similar, while musical F1 degrades substantially under observation noise.
Figure 25: Rollout reward of llms over ten keyframe-optimization iterations and RL method over 5M steps for the two-hand setting on three piano pieces.
Figure 26: token consumption over ten keyframe-optimization iterations for the two-hand setting on three piano pieces.
Target
Perturbed quantity
Family
Frequency
Low / Medium / High
Action
Canonical actuator target
G/U/S
Step
0.05/0.10/0.15
Observation
Proprioception and piano state
G/U/S
Step
0.05/0.10/0.15
Reward
Scalar reward received by the learner
G/U/S
Step
0.10/0.30/0.50
Gravity
Vertical acceleration gz
G/U/S
Episode/step
0.50/1.00/1.50ms−2
Contact-friction
Fingertip–key sliding coefficient
G/U/S
Episode/step
0.05/0.10/0.15
Hand-pose
Per-hand initial (y,z) position
G/U/S
Episode
(10,5)/(25,10)/(50,20)mm
Appendix
Table 4: OmniPiano robustness axes and canonical low/medium/high levels. Levels are standard deviations for Gaussian noise, half-widths for symmetric uniform noise, and magnitudes for constant shifts. Hand-pose entries report (y,z) offsets.
Hands
Song
Cost formulation
Constraint reference
Tnom
Budget d
Runs
2
Für Elise
Joint-range fraction
Central 50%
399
19.95
15
2
ClairDeLune
Joint-range excess
Central 50%
588
11.76
15
3
PicturesGreatKiev
Total-power excess
Pmax=12
720
14.40
15
3
PolonaiseOp40No1
Total-power excess
Pmax=12
563
11.26
15
4
PicturesGreatKiev
Joint-range excess
Central 50%
720
14.40
15
4
PicturesGreatKiev
Total-power excess
Pmax=16
720
14.40
15
Appendix
Table 5: Main safety experiments settings. Each task uses PPO, PPO-Lag, OnCRPO, CPO, and CUP, with three seeds and 5M training steps per run.
Table 6: Baseline algorithms supported by OmniPiano across the four evaluation tracks. Italic denotes on-policy methods and underlined denotes off-policy methods.
Mastering dexterous manipulation with multi-fingered hands has been a grand challenge in robotics for decades. Despite its potential, the difficulty of collecting high-quality data remains a primary bottleneck for high-precision tasks. While reinforcement learning and simulation-to-real-world transfer offer a promising alternative, the transferred policies often fail for tasks demanding millimeter-scale precision, such as bimanual piano playing. In this work, we introduce HandelBot, a framework that combines a simulation policy and rapid adaptation through a two-stage pipeline. Starting from a simulation-trained policy, we first apply a structured refinement stage to correct spatial alignments by adjusting lateral finger joints based on physical rollouts. Next, we use residual reinforcement learning to autonomously learn fine-grained corrective actions. Through extensive hardware experiments across five recognized songs, we demonstrate that HandelBot can successfully perform precise bimanual piano playing. Our system outperforms direct simulation deployment by a factor of 1.8x and requires only 30 minutes of physical interaction data.
Amber Xie, Haozhi Qi, Dorsa Sadigh
Stanford University · Amazon FAR (Frontier AI & Robotics)
Robotic piano playing has emerged as a standard benchmark for dexterous bimanual manipulation, yet progress on it has been measured almost entirely by note accuracy -- which keys are pressed (pitch) and when (onset) -- leaving the musical dynamics essential for expressive performance neither rewarded nor evaluated. We propose CANTABILE, a dynamics-aware framework for robotic piano performance that (i) closes the score-to-contact loop by conditioning the policy on upcoming velocity goals and mapping each key's angular velocity at onset back to MIDI velocity, (ii) couples a velocity-fidelity reward with an onset-coverage reward, so that dynamics cannot be improved by omitting difficult notes, and (iii) refines a frozen dynamics-aware base policy with an alpha-scaled, finger-only residual that localizes strike-intensity adaptation away from nominal note execution. On EXPRESSIVE-51, a dynamics-rich 51-song subset of RoboPianist, CANTABILE raises Velocity F1 -- jointly measuring pitch, onset, and intensity within a +/-8 MIDI-velocity tolerance -- from 0.06 to 0.34 over the RoboPianist baseline, improves all 51 songs, more than halves matched-note velocity error, and reduces log-mel distance to reference audio by 8%. Intensity-randomized training further enables runtime control of performance intensity without retraining.
Woosik Kim, Wonhyeok Choi, Sunghoon Im
KAIST, Daejeon, Republic of Korea · DGIST, Daegu, Republic of Korea
Reinforcement learning can train bimanual dexterous hands to play piano in physics simulation with high note accuracy, but for high-DoF dexterous hands, relying solely on task rewards or IK inversion often leads to unnatural postures and joint overextension. We propose Adversarial Posture Regularization (APR). It avoids expensive, song-aligned expert demonstration data and instead uses a small amount of casual human playing data. By matching the distribution of the posture of the policy with the human prior through an adversarial objective, APR encourages more human-like hand shapes. Meanwhile, we collect and release unstructured hand motion data of piano playing using a consumer-grade Meta Quest 3, and retarget the key motion information to the Shadow Hand. Finally, we achieve significantly better performance than prior methods on all three human-likeness metrics (cPSI, BSE, and FAC) as well as in visual quality. Project repository: https://github.com/APRProject/APRPianist.
Bin Qiu, Yanming Shao, Guanyu Cai +1
Shanghai Jiao Tong University · The University of Hong Kong · Shanghai AI Laboratory