PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination

Xuan Wang; Kai Ruan; Jiayi Han; Kaiyue Zhou; Gaoang Wang

cs.CVApr 14, 2026

PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination

Authors: Xuan Wang, Kai Ruan, Jiayi Han, Kaiyue Zhou, Gaoang Wang

Organizations: Zhejiang University Hangzhou, Zhejiang, China · Renmin University of China Beijing, China · Beijing Forestry University Beijing, China · Chengdu Minto Tech Chengdu, Sichuan, China

Abstract

Audio-driven bimanual piano motion generation requires precise modeling of complex musical structures and dynamic cross-hand coordination. However, existing methods often rely on acoustic-only representations lacking symbolic priors, employ inflexible interaction mechanisms, and are limited to computationally expensive short-sequence generation. To address these limitations, we propose PianoFlow, a flow-matching framework for precise and coordinated bimanual piano motion synthesis. Our approach strategically leverages MIDI as a privileged modality during training, distilling these structured musical priors to achieve deep semantic understanding while maintaining audio-only inference. Furthermore, we introduce an asymmetric role-gated interaction module to explicitly capture dynamic cross-hand coordination through role-aware attention and temporal gating. To enable real-time streaming generation for arbitrarily long sequences, we design an autoregressive flow continuation scheme that ensures seamless cross-chunk temporal coherence. Extensive experiments on the PianoMotion10M dataset demonstrate that PianoFlow achieves superior quantitative and qualitative performance, while accelerating inference by over 9\times compared to previous methods.

Figures & tables

Figure 1. PianoFlow enables high-fidelity and real-time bimanual piano motion generation. (Left) Our audio-driven framework distills privileged musical priors and employs Asymmetric Role-Gated Interaction for coordinated bimanual synthesis. (Right) PianoFlow achieves state-of-the-art performance across all metrics, supporting seamless real-time generation.

Figure 2. Overview of the PianoFlow architecture. The framework operates in two stages: (1) Music-Aware Wrist Trajectory Generation utilizes a decoupled cross-modal distillation scheme to transfer structured harmonic knowledge from a multimodal teacher (MIDI and audio) to an audio-only student. (2) Bimanual Coordinated Gesture Generation synthesizes fine-grained kinematics via conditional flow-matching. Specifically, a factorized spatio-temporal backbone estimates the velocity field, while an Asymmetric Role-Gated Interaction (ARGI) module explicitly models bimanual coordination at the network bottleneck.

Method	Venue	Left Hand				Right Hand				FID $\downarrow$	RTF $\downarrow$
Method	Venue	FGD $\downarrow$	WGD $\downarrow$	PD $\downarrow$	Smooth $\downarrow$	FGD $\downarrow$	WGD $\downarrow$	PD $\downarrow$	Smooth $\downarrow$	FID $\downarrow$	RTF $\downarrow$
EmoTalk ( Peng et al., 2023 )	ICCV 2023	0.445	0.232	0.044	0.353	0.360	0.259	0.033	0.313	4.645	-
LivelySpeaker ( Zhi et al., 2023 )	ICCV 2023	0.538	0.220	0.038	0.406	0.535	0.249	0.030	0.334	4.157	-
PianoMotion ( Gan et al., 2024 )	ICLR 2025	0.372	0.217	0.037	0.248	0.351	0.244	0.030	0.205	3.281	1.746
S2C ( Liu et al., 2025c )	MM 2025	0.309	0.215	0.035	0.229	0.293	0.242	0.028	0.198	3.012	5.723
PianoFlow (Ours)	-	0.266	0.210	0.029	0.200	0.263	0.238	0.027	0.184	2.674	0.186

Table 1. Quantitative evaluation on the PianoMotion10M dataset under the standard short-sequence setting.

Method	Left Hand					Right Hand
Method	FGD $\downarrow$	WGD $\downarrow$	PD $\downarrow$	Smooth $\downarrow$	FDE $\downarrow$	FGD $\downarrow$	WGD $\downarrow$	PD $\downarrow$	Smooth $\downarrow$	FDE $\downarrow$
PianoMotion ( Gan et al., 2024 ) + SWF	0.432	0.220	0.045	0.253	0.334	0.424	0.250	0.037	0.208	0.269
S2C ( Liu et al., 2025c ) + SWF	0.443	0.223	0.041	0.251	0.350	0.397	0.252	0.035	0.206	0.320
PianoFlow + SWF	0.317	0.223	0.040	0.236	0.324	0.296	0.251	0.032	0.203	0.272
PianoFlow + AFC	0.309	0.222	0.040	0.232	0.312	0.290	0.249	0.032	0.199	0.255

Table 2. Quantitative comparison of piano motion generation under the long-sequence setting on the PianoMotion10M dataset.

Figure 3. Qualitative comparison of synthesized motions, with red and green boxes highlighting kinematic inaccuracies and precise alignments, respectively. Overall, PianoFlow achieves superior spatial accuracy and bimanual coordination.

Method	Left Hand				Right Hand				FID $\downarrow$
Method	FGD $\downarrow$	WGD $\downarrow$	PD $\downarrow$	Smooth $\downarrow$	FGD $\downarrow$	WGD $\downarrow$	PD $\downarrow$	Smooth $\downarrow$	FID $\downarrow$
PianoFlow	0.266	0.210	0.029	0.200	0.263	0.238	0.027	0.184	2.674
w/ Hubert	0.293	0.214	0.032	0.251	0.286	0.240	0.029	0.196	2.868
w/o MIDI	0.277	0.211	0.030	0.203	0.281	0.241	0.028	0.194	2.733
w/o Decoupling	0.283	0.211	0.038	0.221	0.293	0.241	0.035	0.198	2.877
w/o Stage1	0.285	0.216	0.095	0.243	0.275	0.244	0.085	0.217	2.841

Table 3. Ablation Study on the PianoMotion10M dataset under the standard short-sequence setting.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors
Apr 6, 2026Joonhyung Bae, Kirak Kim, Hyeyoon Cho +9Human Motion Generation
Korea Advanced Institute of Science and Technology (KAIST), South Korea · Seoul National University, South Korea · YAMAHA, Japan
PianoKontext: Expressive Performance Rendering from Deadpan Context
Jun 10, 2026Dmitrii GavrilevControllable Music Generation
Applied AI Institute
Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training
Dec 2, 2025Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang +3Efficient Transformer Inference Self-Supervised Pre-Training
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China

Apr 6, 2026J/K move · Enter open · S save

Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors

Joonhyung Bae, Kirak Kim, Hyeyoon Cho, Sein Lee+8

Korea Advanced Institute of Science and Technology (KAIST), South Korea · Seoul National University, South Korea · YAMAHA, Japan

Human Motion Generation

Open paper AskPDF arXiv

Synthesizing realistic piano hand motions requires both precision and naturalness. Physics-based methods achieve precision but produce stiff motions; data-driven models learn natural dynamics but struggle with positional accuracy. Piano motion exhibits a natural hierarchy: fingertip positions are nearly deterministic given piano geometry and fingering, while wrist and intermediate joints offer stylistic freedom. We present [OURS], a four-stage framework exploiting this hierarchy: (1) statistics-based fingertip positioning, (2) FiLM-conditioned trajectory refinement, (3) wrist estimation, and (4) STGCN-based pose synthesis. We contribute expert-annotated fingerings for the FürElise dataset (153 pieces, ~10 hours). Experiments demonstrate F1 = 0.910, substantially outperforming diffusion baselines (F1 = 0.121), with user study (N=41) confirming quality approaching motion capture. Expert evaluation by professional pianists (N=5) identified anticipatory motion as the key remaining gap, providing concrete directions for future improvement.

Abstract

Human Motion Generation

Explore similar work

Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors

PianoKontext: Expressive Performance Rendering from Deadpan Context

Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training