Phase-Based Onset Enhancement for Piano Transcription in Ensemble Settings
Authors: Hyemi Kim, Jiyun Park, Juhan Nam
Organizations: Electronics and Telecommunications Research Institute (ETRI), Daejeon, South Korea · Kim Jaechul AI Graduate School, KAIST, South Korea · Graduate School of Culture Technology, KAIST, South Korea
Piano transcription models have achieved very high performance in recent state-of-the-art systems, but their performance tends to degrade in ensemble settings where the piano is played alongside other instruments. While some approaches attempt to address this issue by incorporating multi-instrumental datasets, the problem remains challenging. In particular, when other instruments produce high-energy sustained tones while relatively low-energy piano notes occur simultaneously, the magnitude components of their frequency representations overlap, making it difficult to accurately distinguish piano notes. To address this issue, we propose an improved piano transcription model that jointly leverages magnitude and phase information, enabling more robust piano transcription performance in ensemble settings. We evaluate the proposed model on two piano ensemble datasets with precise MIDI note annotations, which we obtained by revisiting and extending existing datasets. Notably, we constructed a new piano concerto dataset, PCD-K, by playing a piano part on a Disklavier and recording the audio and MIDI pairs with pre-recorded orchestral tracks. Experimental results demonstrate that phase information is highly effective in improving performance, outperforming magnitude-based approaches in ensemble settings.
Figures & tables
Figure 1: An excerpt from the score of Rachmaninoff’s Piano Concerto No.2, with piano (upper) and orchestra (lower).
Figure 2: Overview of the proposed model. The STFT magnitude and phase are used to enhance interval scoring for the 88 piano keys, as indicated by the red arrows and the red boxed region.
Title
Composer
Split
Dur.
Piano Trio in D Minor, Op. 32
Anton Arensky
Train
28:57
Piano Trio in D Minor, Op. 32
Anton Arensky
Train
26:54
Piano Trio in D Minor, Op. 32
Anton Arensky
Train
28:35
Piano Trio in D Major, Op. 70 No. 1
Ludwig van Beethoven
Train
20:55
Piano Trio in D Major, Op. 70 No. 1
Ludwig van Beethoven
Train
19:41
Piano Trio No. 1 in D Major, Op. 49
Felix Mendelssohn
Test
28:42
Table 1: The list of Piano Trios in the MAESTRO v2 dataset. (durations in mm:ss)
Figure 3: Pitch histogram of piano notes in the PCD-K dataset
Model
Additional Datasets
Input
Lowerbound Transkun [ 2 ]
−
Mixture
Kong [ 4 ]
Piano Trio
Mixture
YourMT3+ [ 16 ]
Multi Inst.
Mixture
Transkun_Aug [ 2 ]
Environmental Sound
Mixture
Transkun [ 2 ]
Symphony
Mixture
Transkun-Phase (Ours)
Symphony
Mixture
Table 2: Comparison of models with their additional training datasets and input audio type at inference time.
Piano Note Transcription
Sustain Pedal Transcription
Method
Frame
Onset
On+Off
On+Off+Vel
Frame
Onset
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Lowerbound
74.45
82.51
78.13
56.67
90.98
69.81
43.42
69.60
53.46
37.18
59.54
45.76
92.53
67.51
77.83
43.57
44.15
43.79
YourMT3+ [ 16 ]
82.72
78.88
80.71
91.06
85.38
88.12
68.48
64.24
66.28
30.54
28.66
29.56
-
-
-
-
-
-
Transkun_Aug [ 2 ]
85.49
86.62
86.01
88.83
90.61
89.69
75.52
77.11
76.29
73.19
74.72
73.92
94.03
87.99
90.88
75.72
62.99
68.53
Transkun [ 2 ]
87.66
91.63
89.54
97.73
89.84
93.61
85.82
78.96
82.24
84.36
77.62
80.84
94.97
89.89
92.34
75.13
60.47
66.65
Table 3: Piano note and sustain pedal transcription results on the MAESTRO v2 Trio subset
Piano Note Transcription
Sustain Pedal Transcription
Method
Frame
Onset
On+Off
On+Off+Vel
Frame
Onset
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Lowerbound
59.60
76.52
65.93
50.63
93.09
64.19
35.18
62.66
44.18
26.78
46.84
33.45
87.85
80.90
83.31
49.55
57.03
51.33
Kong
48.01
82.03
58.81
67.05
87.67
74.89
44.25
57.01
49.18
33.67
43.23
37.40
81.65
85.15
82.01
47.17
60.32
51.57
YourMT3+ [ 16 ]
73.89
62.39
66.24
85.42
85.26
85.04
45.08
44.81
44.81
21.79
21.60
21.63
-
-
-
-
-
-
Transkun_Aug [ 2 ]
76.61
77.34
76.02
79.60
92.10
84.76
62.36
71.97
66.37
50.91
58.25
53.98
87.82
87.32
86.33
61.97
65.28
62.12
Table 4: Piano note and sustain pedal transcription results on the PCD-K dataset
Figure 4: An example of piano transcription results (the Rachmaninoff’s Piano Concerto No.2 excerpt in Figure 1 ). True positives (TP), false positives (FP), and false negatives (FN) are shown in purple, red, and gray, respectively, while recovered true positives are highlighted in sky blue.
Piano Note
Sustain Pedal
Dataset
Input
Fr
O
OF
OFV
Fr
O
MAESTRO v2 Trio
A×∣Δϕ∣
90.82
94.83
84.33
83.27
93.00
69.46
A×∣Δ2ϕ∣
90.50
94.58
84.13
83.13
92.54
68.59
A
90.18
93.96
83.03
81.96
91.39
66.94
PCD-K
A×∣Δϕ∣
75.68
90.33
71.11
57.41
84.44
63.31
A×∣Δ2ϕ∣
74.81
89.66
69.98
57.27
82.86
63.10
Table 5: Ablation results on STFT input representations measured by F1-score. Fr, O, OF, and OFV denote frame, onset, onset with offset, and onset with offset and velocity, respectively.
Graduate School of Informatics, Kyoto University, Japan · Graduate School of Engineering, Kyoto University, Japan · Independent Researcher, Hong Kong, China