Mastering dexterous manipulation with multi-fingered hands has been a grand challenge in robotics for decades. Despite its potential, the difficulty of collecting high-quality data remains a primary bottleneck for high-precision tasks. While reinforcement learning and simulation-to-real-world transfer offer a promising alternative, the transferred policies often fail for tasks demanding millimeter-scale precision, such as bimanual piano playing. In this work, we introduce HandelBot, a framework that combines a simulation policy and rapid adaptation through a two-stage pipeline. Starting from a simulation-trained policy, we first apply a structured refinement stage to correct spatial alignments by adjusting lateral finger joints based on physical rollouts. Next, we use residual reinforcement learning to autonomously learn fine-grained corrective actions. Through extensive hardware experiments across five recognized songs, we demonstrate that HandelBot can successfully perform precise bimanual piano playing. Our system outperforms direct simulation deployment by a factor of 1.8x and requires only 30 minutes of physical interaction data.
Figures & tables
Fig. 1: HandelBot Method (0) RL in Sim. We leverage fast, parallel simulators for reinforcement learning. This leads to a coarse base policy, πsim , from which we extract an open-loop rollout, τsim . (1) Policy Refinement. Second, we refine τsim , yielding τsim∗ . We use real-world updates to iteratively update the lateral joints of the fingers, moving the finger horizontally in the direction of the keys it is intended to press. (2) Residual RL. We perform residual RL atop τsim , using the keyboard’s MIDI output as a reward. This allows us to further update our policy for better piano playing.
Fig. 2: Hardware Setup . We use a MIDI keyboard, two Tesollo DG-5F hands, and two Franka arms for piano playing. We use the MIDI output from the piano, which tells us which notes are pressed, in order to calculate rewards. We emphasize that the robot hands are far larger than the average human hand, thus making piano playing difficult. Finally, for RL training, we include a collision checker which prevents fingers from pressing down beyond the keys.
Fig. 3: Main Results. We include F1 score, multiplied by 100, for 5 songs. HandelBot consistently achieves the strongest F1 score, showing the importance of effectively using real-world samples to accomplish precise, dexteorus piano-playing. Methods only using simulated data, such as πsim (CL) and πsim , have weak performance due to the sim-to-real gap.
Fig. 4: Visualization of HandelBot Trajectories. Per each song, we visualize the notes pressed correctly , pressed incorrectly , and missed . The x axis is the timestep of the song, and the y axis are the different notes, with the top half representing keys for the right hand, and the bottom for the left hand. Across easier songs such as Twinkle Twinkle and Ode to Joy, we find that HandelBot makes few mistakes, with occasional timing errors or wrong presses. For harder songs such as Fur Elise, large jumps in the left hand notes (bottom section of each song plot) are challenging for the left hand.
Song
πsim ( CL )
πsim ( Hybrid )
Ode to Joy
5 ± 2.46
12 ± 2.6
Twinkle Twinkle
23 ± 6.2
24 ± 2.5
Hot Cross Buns
8 ± 2.1
9 ± 2.9
Prelude in C
29 ± 2.2
40 ± 1.0
Fur Elise
18 ± 3.2
20 ± 4.9
TABLE I: We report F1 scores x 100 for the closed-loop πsim , which runs inference over real-world inputs, with hybrid execution πsim . Hybrid execution runs a simulation in parallel with the real-world, and uses proprioception from the parallel sim instead of the real-world.
Twinkle
HandelBot (OURS)
81 ± 4.1
HandelBot: γ = 0.75
73 ± 2.5
HandelBot: γ = 0.9
69 ± 0.2
HandelBot: Pr(guided noise) = 0
81 ± 0.7
HandelBot: Pr(guided noise) = 1
77 ± 0.9
TABLE II: Ablations . By default, HandelBot uses γ=0.8 and Pr(guided noise)=0.5 .
Fig. 5: HandelBot Trajectories across Residual RL Training. We include 4 evaluation trajectories during HandelBot training, with the final, best-performing trajectory in fig. 4 . Across these 4 trajectories, we see that HandelBot initially struggles with many keys in the left hand. However, with real-world interactions, the residual policy is able to adapt to real world and press the correct keys.
Reinforcement learning can train bimanual dexterous hands to play piano in physics simulation with high note accuracy, but for high-DoF dexterous hands, relying solely on task rewards or IK inversion often leads to unnatural postures and joint overextension. We propose Adversarial Posture Regularization (APR). It avoids expensive, song-aligned expert demonstration data and instead uses a small amount of casual human playing data. By matching the distribution of the posture of the policy with the human prior through an adversarial objective, APR encourages more human-like hand shapes. Meanwhile, we collect and release unstructured hand motion data of piano playing using a consumer-grade Meta Quest 3, and retarget the key motion information to the Shadow Hand. Finally, we achieve significantly better performance than prior methods on all three human-likeness metrics (cPSI, BSE, and FAC) as well as in visual quality. Project repository: https://github.com/APRProject/APRPianist.
Bin Qiu, Yanming Shao, Guanyu Cai +1
Shanghai Jiao Tong University · The University of Hong Kong · Shanghai AI Laboratory
Performing in-hand, contact-rich, and long-horizon dexterous manipulation remains an unsolved challenge in robotics. Prior hand dexterity works have considered each of these three challenges in isolation, yet do not combine these skills into a single, complex task. To further test the capabilities of dexterity, we propose drumming as a testbed for dexterous manipulation. Drumming naturally integrates all three challenges: it involves in-hand control for stabilizing and adjusting the drumstick with the fingers, contact-rich interaction through repeated striking of the drum surface, and long-horizon coordination when switching between drums and sustaining rhythmic play. We present DexDrummer, a hierarchical object-centric bimanual drumming policy trained in simulation with sim-to-real transfer. The framework reduces the exploration difficulty of pure reinforcement learning by combining trajectory planning with residual RL corrections for fast transitions between drums. A dexterous manipulation policy handles contact-rich dynamics, guided by rewards that explicitly model both finger-stick and stick-drum interactions. In simulation, we show our policy can play two styles of music: multi-drum, bimanual songs and challenging, technical exercises that require increased dexterity. Across simulated bimanual tasks, our dexterous, reactive policy outperforms a fixed grasp policy by 1.87x across easy songs and 1.22x across hard songs F1 scores. In real-world tasks, we show song performance across a multi-drum setup. DexDrummer is able to play our training song and its extended version with an F1 score of 1.0.
Robotic piano playing has emerged as a standard benchmark for dexterous bimanual manipulation, yet progress on it has been measured almost entirely by note accuracy -- which keys are pressed (pitch) and when (onset) -- leaving the musical dynamics essential for expressive performance neither rewarded nor evaluated. We propose CANTABILE, a dynamics-aware framework for robotic piano performance that (i) closes the score-to-contact loop by conditioning the policy on upcoming velocity goals and mapping each key's angular velocity at onset back to MIDI velocity, (ii) couples a velocity-fidelity reward with an onset-coverage reward, so that dynamics cannot be improved by omitting difficult notes, and (iii) refines a frozen dynamics-aware base policy with an alpha-scaled, finger-only residual that localizes strike-intensity adaptation away from nominal note execution. On EXPRESSIVE-51, a dynamics-rich 51-song subset of RoboPianist, CANTABILE raises Velocity F1 -- jointly measuring pitch, onset, and intensity within a +/-8 MIDI-velocity tolerance -- from 0.06 to 0.34 over the RoboPianist baseline, improves all 51 songs, more than halves matched-note velocity error, and reduces log-mel distance to reference audio by 8%. Intensity-randomized training further enables runtime control of performance intensity without retraining.
Woosik Kim, Wonhyeok Choi, Sunghoon Im
KAIST, Daejeon, Republic of Korea · DGIST, Daegu, Republic of Korea