Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors

Joonhyung Bae; Kirak Kim; Hyeyoon Cho; Sein Lee; Yoon-Seok Choi; Hyeon Hur; Gyubin Lee; Akira Maezawa

cs.AIApr 6, 2026

Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors

Authors: Joonhyung Bae, Kirak Kim, Hyeyoon Cho, Sein Lee, Yoon-Seok Choi, Hyeon Hur, Gyubin Lee, Akira Maezawa,

Organizations: Korea Advanced Institute of Science and Technology (KAIST), South Korea · Seoul National University, South Korea · YAMAHA, Japan

Abstract

Synthesizing realistic piano hand motions requires both precision and naturalness. Physics-based methods achieve precision but produce stiff motions; data-driven models learn natural dynamics but struggle with positional accuracy. Piano motion exhibits a natural hierarchy: fingertip positions are nearly deterministic given piano geometry and fingering, while wrist and intermediate joints offer stylistic freedom. We present [OURS], a four-stage framework exploiting this hierarchy: (1) statistics-based fingertip positioning, (2) FiLM-conditioned trajectory refinement, (3) wrist estimation, and (4) STGCN-based pose synthesis. We contribute expert-annotated fingerings for the FürElise dataset (153 pieces, ~10 hours). Experiments demonstrate F1 = 0.910, substantially outperforming diffusion baselines (F1 = 0.121), with user study (N=41) confirming quality approaching motion capture. Expert evaluation by professional pianists (N=5) identified anticipatory motion as the key remaining gap, providing concrete directions for future improvement.

Figures & tables

Figure 1. Overview of Tipiano. Our four-stage cascade exploits decreasing ambiguity from fingertips to intermediate joints.

Dataset	Pieces	Notes	Hours	Performer	Annotation	Boundary
PIG a	150	100K	—	Expert	Expert	✗
ThumbSet b	2.5K	—	—	—	Amateur	✗
PianoVAM c	106	1.05M	21	Amateur	Researcher	✗
Tipiano	153	340K	10	Expert	Expert	✓

Table 1. Piano fingering datasets comparison. Fingering provides per-note finger-key contact assignments, enabling position prior construction.

Figure 2. The Tipiano pipeline. From MIDI and fingering, four cascaded stages synthesize hand motion: (1) statistics-based fingertip positioning, (2) FiLM-conditioned trajectory refinement with geometric constraints, (3) wrist estimation via offset priors, and (4) STGCN-based full pose synthesis.

Method	Task Accuracy			Position (mm) $\downarrow$									Accel. Ratio ∗
	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Fingertip			Wrist			Full			Fingertip			Wrist			Full
	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Left	Right	Both	Left	Right	Both	Left	Right	Both	Left	Right	Both	Left	Right	Both	Left	Right	Both
FürElise Dataset	.839	.670	.739	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
S2C a	.100	.163	.121	74.1	64.3	69.2	65.4	58.5	61.9	69.6	59.9	64.8	0.74	0.67	0.70	0.75	0.79	0.77	0.77	0.72	0.75
RoboPianist b	.996 †	.783 †	.877 †	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
Tipiano (Stage 1)	.798	.991	.882	50.0	49.7	49.8	—	—	—	—	—	—	3.06	3.45	3.79	—	—	—	—	—	—

Table 2. Quantitative comparison.

\downarrow

: lower is better,

\uparrow

: higher is better. Accel. Ratio ∗ : ratio of predicted to GT acceleration magnitude;

{\approx}1.0

matches GT dynamics,

{<}1.0

lower acceleration,

{>}1.0

higher acceleration. F1 scores are evaluated on Stage 2.2 output (post-refinement, pre-smoothing) across all stages to ensure consistent key contact measurement; smoothing improves motion quality but slightly shifts press timing. Position and Accel. metrics reflect each stage’s actual output.

Figure 3. User study results (

N=41

). (a) Preference for Tipiano over baselines. (b) Mean preference gap from FürElise dataset on 7-point Likert scale (lower is better; 0 = no preference). Tipiano achieves the smallest gap across all dimensions. Error bars: 95% CI. All

p<.001

Configuration	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	$\Delta$ F1
Full (proposed)	0.847	0.988	0.912	—
w/o Y-axis masking	0.846	0.909	0.876	$-$ 0.036
Raw MIDI features	0.829	0.986	0.901	$-$ 0.011
Concat (no FiLM)	0.838	0.987	0.907	$-$ 0.005

Table 3. Design choice ablation on the test set. All variants are compared at the Stage 2.2 output.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination
Apr 14, 2026Xuan Wang, Kai Ruan, Jiayi Han +2Human Motion Generation Flow-Based Generative Modeling
Zhejiang University Hangzhou, Zhejiang, China · Renmin University of China Beijing, China · Beijing Forestry University Beijing, China +1
Statistical Models for Automatic Fingering-Annotated Piano Sheet Music Transcription
Sep 23, 2026Daniel Penner, Abram HindleAutomatic Music Transcription
Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada
SKY-Piano: A Multimodal Piano Performance Dataset
Jul 29, 2026Joonhyung Bae, Dawon Park, Taegyun Kwon +10Music Information Retrieval
KAIST · Seoul National University · Yamaha Corporation

Apr 14, 2026J/K move · Enter open · S save

PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination

Xuan Wang, Kai Ruan, Jiayi Han, Kaiyue Zhou+1

Zhejiang University Hangzhou, Zhejiang, China · Renmin University of China Beijing, China · Beijing Forestry University Beijing, China+1

Human Motion Generation Flow-Based Generative Modeling

Open paper AskPDF arXiv

Audio-driven bimanual piano motion generation requires precise modeling of complex musical structures and dynamic cross-hand coordination. However, existing methods often rely on acoustic-only representations lacking symbolic priors, employ inflexible interaction mechanisms, and are limited to computationally expensive short-sequence generation. To address these limitations, we propose PianoFlow, a flow-matching framework for precise and coordinated bimanual piano motion synthesis. Our approach strategically leverages MIDI as a privileged modality during training, distilling these structured musical priors to achieve deep semantic understanding while maintaining audio-only inference. Furthermore, we introduce an asymmetric role-gated interaction module to explicitly capture dynamic cross-hand coordination through role-aware attention and temporal gating. To enable real-time streaming generation for arbitrarily long sequences, we design an autoregressive flow continuation scheme that ensures seamless cross-chunk temporal coherence. Extensive experiments on the PianoMotion10M dataset demonstrate that PianoFlow achieves superior quantitative and qualitative performance, while accelerating inference by over 9\times compared to previous methods.

Abstract

Human Motion Generation

Explore similar work

PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination

Statistical Models for Automatic Fingering-Annotated Piano Sheet Music Transcription

SKY-Piano: A Multimodal Piano Performance Dataset