A learning-based method of automatic music transcription
Authors: Phu-Anh-Tai Do, Huu-Hoang Nguyen, Duc-Hoang Nguyen, Thanh-Hai Le
Organizations: Department of Information Technology, FPT University, Ho Chi Minh City, Vietnam · Faculty of Information Technology, Ho Chi Minh City Open University, Ho Chi Minh City, Vietnam
Automatic Music Transcription (AMT) seeks to convert acoustic music recordings into symbolic musical notation, a process that remains remarkably difficult for polyphonic piano due to concurrent polyphony, harmonic partial interference, expressive timing, and severe pitch activation sparsity. In this paper, we formulate polyphonic piano transcription as a frame-level multi-label classification problem across the standard 88 piano pitches (MIDI notes 21 to 108). Raw audio waveforms are processed using the Constant-Q Transform (CQT) to construct a geometrically spaced log-frequency time-frequency representation that aligns with the chromatic musical scale across seven octaves (252 frequency bins per frame). We systematically investigate recurrent sequence models, contrasting unidirectional Long Short-Term Memory (LSTM) with Bidirectional LSTM (BiLSTM) backbones, and propose two architectural adaptations tailored to recurrent temporal representations: a modified additive attention module and a modified multihead attention mechanism equipped with hyperbolic tangent activation and head-wise summation. Furthermore, to mitigate the severe class imbalance between inactive and active notes (approximately 41:1), we evaluate Weighted Binary Cross-Entropy (WBCE) alongside standard Binary Cross-Entropy (BCE). Rigorous empirical evaluations on the MAPS and MAESTRO datasets demonstrate steady, compounding gains from bidirectionality and attention mechanisms. Our proposed Multihead Attention-BiLSTM architecture attains an F1-score of 88.20%, an accuracy of 78.90%, a precision of 0.89, and a recall of 0.86, substantially outperforming the baseline unidirectional LSTM (F1-score of 71.20%) while maintaining a compact parameter budget of 4.43M. Qualitative piano-roll visualizations confirm that our model generates faithful transcriptions even in complex harmonic passages.
Figures & tables
Figure 1: Data representation pipeline in an Automatic Music Transcription (AMT) system. The input acoustic waveform is transformed into a time-frequency representation, parameterized into frame-level feature vectors, processed by a sequence-modeling acoustic and temporal network, and decoded into active pitch activations (MIDI/piano roll representation) [ 1 ] .
Dataset
Train
Validation
Test
MAESTRO [ 10 ]
962
174
–
MAPS [ 9 ]
23,928
2,991
2,991
Table 1: Data split distribution for the MAPS and MAESTRO benchmark datasets.
Onset Time (s)
Offset Time (s)
MIDI Pitch
0.988281
1.111979
34 (A#1)
1.109375
1.135417
53 (F3)
1.109375
1.161458
62 (D4)
1.153646
1.196615
58 (A#3)
1.518229
1.549479
74 (D5)
1.527344
1.563802
62 (D4)
Table 2: Sample extract of ground truth note event annotations pairing onset time, offset time, and MIDI pitch.
Figure 2: Piano keyboard layout spanning 88 keys and 7 complete octaves. Fundamental frequencies range from A0 (27.5 Hz) to C8 (4186.01 Hz), geometrically spaced according to the equal-tempered scale.
Figure 3: Constant-Q Transform (CQT) spectrogram. The log-frequency axis maintains constant musical interval spacing across octaves, providing sharp resolution for fundamental frequencies in the lower register.
Figure 4: Visual illustration of frame-level multi-label ground truth matrices yt∈{0,1}88 . Rows correspond to the 88 piano pitches and columns represent consecutive time frames.
Figure 5: Class distribution imbalance in the training corpus. Inactive pitch labels (0) total 5.36 billion instances, whereas active pitch labels (1) total 0.129 billion instances, establishing a severe ratio of approximately 41:1.
Figure 6: Internal architecture of the Long Short-Term Memory (LSTM) cell. The cell state ct serves as a constant-error conveyor belt regulated by the forget gate ft , input gate it , and output gate ot [ 8 ] .
Figure 7: Bidirectional Long Short-Term Memory architecture. The input sequence is processed concurrently in both chronological (forward) and reverse (backward) temporal directions.
Figure 8: Proposed Modified Multihead Attention module. Parallel attention heads compute subspace alignment scores, which are summed, projected via WO with bias bO , and activated by hyperbolic tangent before softmax normalization.
Figure 9: Architectural transition from baseline to modified base model. Replacing MSE loss with WBCE and ReLU with Sigmoid eliminates saturation and enables effective multi-label learning.
Table 3: Comprehensive benchmark evaluation of transcription models. Precision, Recall, Accuracy (%), and F1-score (%) on the polyphonic piano test dataset.
Figure 11: Training dynamics of Base Model (LSTM) versus Best Model (Multihead Attention–BiLSTM). The proposed best model converges to a substantially lower training loss while maintaining rapid, steady accuracy improvements without over-fitting.
Figure 12: Visual comparison of transcribed piano rolls against Ground Truth. The Base Model exhibits substantial temporal fragmentation and missed notes, whereas our Best Model closely reconstructs note onsets, sustained durations, and polyphonic chords.
Figure 13: Qualitative ablation of loss formulation: WBCE versus standard BCE. WBCE promotes higher sensitivity but introduces occasional background noise, whereas BCE achieves cleaner background suppression with high harmonic fidelity.
Figure 14: Qualitative ablation of dataset diversity: Combined Datasets versus Single Dataset. Training on both MAPS and MAESTRO provides broader acoustic exposure, reducing false insertions in higher octaves.
This paper presents the results of the 2025 Automatic Music Transcription (AMT) Challenge, an online competition to benchmark progress in multi-instrument transcription. Eight teams submitted valid solutions; two outperformed the baseline MT3 model. The results highlight both advances in transcription accuracy and the remaining difficulties in handling polyphony and timbre variation. We conclude with directions for future challenges: broader genre coverage and stronger emphasis on instrument detection.
This paper describes a novel paradigm that formalizes automatic piano transcription (APT) as an optimal transport (OT) problem, not as a frame-level multi-label binary classification problem. Our method learns to minimize the cost of transporting a predicted distribution of note events to the ground-truth distribution over time and frequency. The OT loss can thus accommodate temporal misalignment, leading to perceptually relevant optimization. We also propose a convolutional recurrent neural network (CRNN) with a harmonics-aware attention mechanism to capture the spectro-temporal dependencies inherent in music.Our experiments using the MAESTRO dataset showed that our method attained a state-of-the-art performance in onset detection. We confirmed the versatility of the OT loss in application to existing models.
Weixing Wei, Raynaldi Lalang, Dichucheng Li +1
Graduate School of Informatics, Kyoto University, Japan · Graduate School of Engineering, Kyoto University, Japan · Independent Researcher, Hong Kong, China
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
Simon Rouard, Michael Krause, Axel Roebel +2
Kyutai · Mirelo AI · UMR STMS, IRCAM-CNRS Sorbonne Univ.