Phase-Based Onset Enhancement for Piano Transcription in Ensemble Settings
Authors: Hyemi Kim, Jiyun Park, Juhan Nam
Organizations: Electronics and Telecommunications Research Institute (ETRI), Daejeon, South Korea · Kim Jaechul AI Graduate School, KAIST, South Korea · Graduate School of Culture Technology, KAIST, South Korea
Piano transcription models have achieved very high performance in recent state-of-the-art systems, but their performance tends to degrade in ensemble settings where the piano is played alongside other instruments. While some approaches attempt to address this issue by incorporating multi-instrumental datasets, the problem remains challenging. In particular, when other instruments produce high-energy sustained tones while relatively low-energy piano notes occur simultaneously, the magnitude components of their frequency representations overlap, making it difficult to accurately distinguish piano notes. To address this issue, we propose an improved piano transcription model that jointly leverages magnitude and phase information, enabling more robust piano transcription performance in ensemble settings. We evaluate the proposed model on two piano ensemble datasets with precise MIDI note annotations, which we obtained by revisiting and extending existing datasets. Notably, we constructed a new piano concerto dataset, PCD-K, by playing a piano part on a Disklavier and recording the audio and MIDI pairs with pre-recorded orchestral tracks. Experimental results demonstrate that phase information is highly effective in improving performance, outperforming magnitude-based approaches in ensemble settings.
Figures & tables
Figure 1: An excerpt from the score of Rachmaninoff’s Piano Concerto No.2, with piano (upper) and orchestra (lower).
Figure 2: Overview of the proposed model. The STFT magnitude and phase are used to enhance interval scoring for the 88 piano keys, as indicated by the red arrows and the red boxed region.
Title
Composer
Split
Dur.
Piano Trio in D Minor, Op. 32
Anton Arensky
Train
28:57
Piano Trio in D Minor, Op. 32
Anton Arensky
Train
26:54
Piano Trio in D Minor, Op. 32
Anton Arensky
Train
28:35
Piano Trio in D Major, Op. 70 No. 1
Ludwig van Beethoven
Train
20:55
Piano Trio in D Major, Op. 70 No. 1
Ludwig van Beethoven
Train
19:41
Piano Trio No. 1 in D Major, Op. 49
Felix Mendelssohn
Test
28:42
Table 1: The list of Piano Trios in the MAESTRO v2 dataset. (durations in mm:ss)
Figure 3: Pitch histogram of piano notes in the PCD-K dataset
Model
Additional Datasets
Input
Lowerbound Transkun [ 2 ]
−
Mixture
Kong [ 4 ]
Piano Trio
Mixture
YourMT3+ [ 16 ]
Multi Inst.
Mixture
Transkun_Aug [ 2 ]
Environmental Sound
Mixture
Transkun [ 2 ]
Symphony
Mixture
Transkun-Phase (Ours)
Symphony
Mixture
Table 2: Comparison of models with their additional training datasets and input audio type at inference time.
Piano Note Transcription
Sustain Pedal Transcription
Method
Frame
Onset
On+Off
On+Off+Vel
Frame
Onset
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Lowerbound
74.45
82.51
78.13
56.67
90.98
69.81
43.42
69.60
53.46
37.18
59.54
45.76
92.53
67.51
77.83
43.57
44.15
43.79
YourMT3+ [ 16 ]
82.72
78.88
80.71
91.06
85.38
88.12
68.48
64.24
66.28
30.54
28.66
29.56
-
-
-
-
-
-
Transkun_Aug [ 2 ]
85.49
86.62
86.01
88.83
90.61
89.69
75.52
77.11
76.29
73.19
74.72
73.92
94.03
87.99
90.88
75.72
62.99
68.53
Transkun [ 2 ]
87.66
91.63
89.54
97.73
89.84
93.61
85.82
78.96
82.24
84.36
77.62
80.84
94.97
89.89
92.34
75.13
60.47
66.65
Table 3: Piano note and sustain pedal transcription results on the MAESTRO v2 Trio subset
Piano Note Transcription
Sustain Pedal Transcription
Method
Frame
Onset
On+Off
On+Off+Vel
Frame
Onset
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Lowerbound
59.60
76.52
65.93
50.63
93.09
64.19
35.18
62.66
44.18
26.78
46.84
33.45
87.85
80.90
83.31
49.55
57.03
51.33
Kong
48.01
82.03
58.81
67.05
87.67
74.89
44.25
57.01
49.18
33.67
43.23
37.40
81.65
85.15
82.01
47.17
60.32
51.57
YourMT3+ [ 16 ]
73.89
62.39
66.24
85.42
85.26
85.04
45.08
44.81
44.81
21.79
21.60
21.63
-
-
-
-
-
-
Transkun_Aug [ 2 ]
76.61
77.34
76.02
79.60
92.10
84.76
62.36
71.97
66.37
50.91
58.25
53.98
87.82
87.32
86.33
61.97
65.28
62.12
Table 4: Piano note and sustain pedal transcription results on the PCD-K dataset
Figure 4: An example of piano transcription results (the Rachmaninoff’s Piano Concerto No.2 excerpt in Figure 1 ). True positives (TP), false positives (FP), and false negatives (FN) are shown in purple, red, and gray, respectively, while recovered true positives are highlighted in sky blue.
Piano Note
Sustain Pedal
Dataset
Input
Fr
O
OF
OFV
Fr
O
MAESTRO v2 Trio
A×∣Δϕ∣
90.82
94.83
84.33
83.27
93.00
69.46
A×∣Δ2ϕ∣
90.50
94.58
84.13
83.13
92.54
68.59
A
90.18
93.96
83.03
81.96
91.39
66.94
PCD-K
A×∣Δϕ∣
75.68
90.33
71.11
57.41
84.44
63.31
A×∣Δ2ϕ∣
74.81
89.66
69.98
57.27
82.86
63.10
Table 5: Ablation results on STFT input representations measured by F1-score. Fr, O, OF, and OFV denote frame, onset, onset with offset, and onset with offset and velocity, respectively.
Machine learning tools have significantly aided automatic piano music transcription; however, this domain has focused primarily on accurately predicting the pitches and timings of played notes. To produce sheet music for the piano, notes must be separated into two staves, one for each hand, and good sheet music often contains fingering annotations to guide the player when sight-reading or learning fast or complex pieces. We propose 8 statistical approaches for combined hand and fingering annotation of transcribed piano notes, including baseline hidden Markov models, rule-based methods, a synthesis of existing approaches, and N-gram language models. Furthermore, we develop a pipeline for complete transcription from piano audio to fingering-annotated sheet music. Evaluations with the PIG dataset demonstrate that our Synthesis model achieves a hand separation accuracy of 90.8% and a joint hand and finger annotation accuracy of 56.6%. These approaches serve as a new baseline for further research into this problem, while our pipeline demonstrates the feasibility of a combined system for automated note transcription, hand separation, and fingering annotation.
Daniel Penner, Abram Hindle
Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada
This paper describes a novel paradigm that formalizes automatic piano transcription (APT) as an optimal transport (OT) problem, not as a frame-level multi-label binary classification problem. Our method learns to minimize the cost of transporting a predicted distribution of note events to the ground-truth distribution over time and frequency. The OT loss can thus accommodate temporal misalignment, leading to perceptually relevant optimization. We also propose a convolutional recurrent neural network (CRNN) with a harmonics-aware attention mechanism to capture the spectro-temporal dependencies inherent in music.Our experiments using the MAESTRO dataset showed that our method attained a state-of-the-art performance in onset detection. We confirmed the versatility of the OT loss in application to existing models.
Weixing Wei, Raynaldi Lalang, Dichucheng Li +1
Graduate School of Informatics, Kyoto University, Japan · Graduate School of Engineering, Kyoto University, Japan · Independent Researcher, Hong Kong, China
Symbolic music datasets with matched scores and performances are essential for many music information retrieval (MIR) tasks. Yet, existing resources often cover a narrow range of composers, lack performance variety, omit note-level alignments, or use inconsistent naming formats. This work presents PianoCoRe, a large-scale piano MIDI dataset that unifies and refines major open-source piano corpora. The dataset contains 250,046 performances of 5,625 pieces written by 483 composers, totaling 21,763 h of performed music. PianoCoRe is released in tiered subsets to support different applications: from large-scale analysis and pre-training (PianoCoRe-C and deduplicated PianoCoRe-B) to expressive performance modeling with note-level score alignment (PianoCoRe-A/A*). The note-aligned subset, PianoCoRe-A, provides the largest open-source collection of 157,207 performances aligned to 1,591 scores to date. In addition to the dataset, the contributions are: (1) a MIDI quality classifier for detecting corrupted and score-like transcriptions and (2) RAScoP, an alignment refinement pipeline that cleans temporal alignment errors and interpolates missing notes. The analysis shows that the refinement reduces temporal noise and eliminates tempo outliers. Moreover, an expressive performance rendering model trained on PianoCoRe demonstrates improved robustness to unseen pieces compared to models trained on raw or smaller datasets. PianoCoRe provides a ready-to-use foundation for the next generation of expressive piano performance research.
Ilya Borovik
*Skolkovo Institute of Science and Technology, Moscow, Russia