bFaaaP: An Inclusive, Head-Angle Piano-Pedal Interaction that Quantitatively Reproduces a Pianist's Intended Pedalling -- Foot-Free, for Acoustic and Electronic Pianos
Expressive piano performance depends on the sustain (damper) pedal, operated by foot, excluding players who cannot readily use their feet: wheelchair users and others with lower-limb impairments, small children, and some elderly or disabled players. We present bFaaaP (barrier-Free assist as a Pedal), an inclusive, foot-free interaction that operates the pedal from the angle of the player's head: a smartphone tracks head pose with on-device augmented-reality (AR) face tracking and streams a compact command over Bluetooth Low Energy (BLE) to a pedal device. Supported by patent examination, our central claim is not the head-to-pedal architecture (anticipated by prior art) but a quantitative, user-tunable control law -- the patentable "key" to a natural, expressive result: the player presets a small angular dead-zone (offset 3-10 degrees) and a multiplier (10-50), which together fix a secondary, pre-adjustable response speed that reproduces the pianist's intended pedalling. An engineering trick decouples the fast AR sampling from the slower BLE rate. Two co-equal realizations share one controller: a non-destructive robotic actuator for acoustic pianos (Pro), anchored by a pneumatic "airback" (our coined term for an inflatable air-braced anchor) that absorbs the reaction force without modifying the instrument; and an electronic sustain switch for digital pianos (Switch). In a human-subject Auxiliary Pedal Effect Evaluation (APEE) with 15 participants, bFaaaP significantly increased sustained-tone energy (p<0.01) and was statistically indistinguishable from a player's own foot (p>0.05), with no significant difference across classes; one participant with a leg disability and a tracheostomy performed successfully. With nothing worn on the face and fast setup, bFaaaP has run in formal public concerts (2018-2025). We release the full hardware and software as open source.
Reinforcement learning can train bimanual dexterous hands to play piano in physics simulation with high note accuracy, but for high-DoF dexterous hands, relying solely on task rewards or IK inversion often leads to unnatural postures and joint overextension. We propose \textit{Adversarial Posture Regularization (APR)}. It avoids expensive, song-aligned expert demonstration data and instead uses a small amount of casual human playing data. By matching the distribution of the posture of the policy with the human prior through an adversarial objective, APR encourages more human-like hand shapes. Meanwhile, we collect and release unstructured hand motion data of piano playing using a consumer-grade Meta Quest 3, and retarget the key motion information to the Shadow Hand. Finally, we achieve significantly better performance than prior methods on all three human-likeness metrics (cPSI, BSE, and FAC) as well as in visual quality. Project repository: https://github.com/APRProject/APRPianist.
Robotic piano playing has emerged as a standard benchmark for dexterous bimanual manipulation, yet progress on it has been measured almost entirely by note accuracy -- which keys are pressed (pitch) and when (onset) -- leaving the musical dynamics essential for expressive performance neither rewarded nor evaluated. We propose CANTABILE, a dynamics-aware framework for robotic piano performance that (i) closes the score-to-contact loop by conditioning the policy on upcoming velocity goals and mapping each key's angular velocity at onset back to MIDI velocity, (ii) couples a velocity-fidelity reward with an onset-coverage reward, so that dynamics cannot be improved by omitting difficult notes, and (iii) refines a frozen dynamics-aware base policy with an alpha-scaled, finger-only residual that localizes strike-intensity adaptation away from nominal note execution. On EXPRESSIVE-51, a dynamics-rich 51-song subset of RoboPianist, CANTABILE raises Velocity F1 -- jointly measuring pitch, onset, and intensity within a +/-8 MIDI-velocity tolerance -- from 0.06 to 0.34 over the RoboPianist baseline, improves all 51 songs, more than halves matched-note velocity error, and reduces log-mel distance to reference audio by 8%. Intensity-randomized training further enables runtime control of performance intensity without retraining.
Expressive performance rendering (EPR) aims to generate realistic performances constrained on sequences of notes. However, flow matching audio editing models manipulate only synchronized music samples of the same duration, limiting their understanding of expressive timing. We introduce PianoKontext, a flow matching rendering model for classical piano music that generates variable-length performances in the latent space of a pretrained Music2Latent model. We synthesize MIDI scores into deadpan audio and employ Dynamic Time Warping (DTW) in the latent space to construct paired data for training. The aligned embeddings are concatenated in DiT blocks, allowing for a simple and effective learning of the dependencies between the score and performances. Audio samples are available at our demo page: https://realfolkcode.github.io/pianokontext_demo/.