Assessing Alignment in Automatic Piano Transcription Datasets: A Benchmark for Scoring Methods
Authors: Lucas de Souza Brandão, Lucas Gris, Joaquim Cavalcante, Chukwuemeka Nkama, Andrea Poltronieri
Organizations: Universidade de São Paulo, Brazil · Universidade Federal de Goiás, Brazil · Universidade Federal da Paraíba, Brazil · Universitat Pompeu Fabra, Spain
Automatic Music Transcription (AMT) relies on paired audio--MIDI datasets, implicitly assuming near-perfect temporal alignment. However, widely used piano datasets have been shown to contain alignment errors that may bias model evaluation and training. We propose a framework for evaluating audio--MIDI alignment scoring methods through controlled temporal, pitch, and structural MIDI corruptions. We also introduce STI, a scoring method built on AMT models, and benchmark it against existing alignment scoring approaches. Our results show that STI, together with methods based on CENS and spectrograms, achieves the strongest overall performance, with STI exhibiting the highest sensitivity to fine-grained temporal misalignments, the main alignment error in AMT datasets. We then apply the best-performing methods to widely used piano AMT datasets and identify trends in their relative alignment quality. These results support more informed dataset curation, model training, and AMT evaluation.
Figures & tables
Structural
Onset Shift
Pitch Swap
Metric
Mono
Stab
Sens
Cons
Mono
Stab
Sens
Cons
Mono
Stab
Sens
Cons
CLaMP 3
0.89
0.97
0.09
0.83
0.55
0.96
0.02
0.46
0.93
0.96
0.07
0.87
CLAP
0.77
0.96
0.03
0.69
0.25
0.96
0.00
0.26
0.22
0.96
0.00
0.41
CS-C
0.99
0.93
0.49
0.87
0.97
0.95
0.23
0.95
1.00
0.94
0.49
0.99
CS-D
0.98
0.68
0.78
0.98
0.72
0.65
0.66
0.42
0.99
0.69
0.75
0.98
CS-S
0.99
0.93
0.34
0.87
0.97
0.94
0.21
0.89
1.00
0.93
0.37
1.00
Table 1: Performance of each method per corruption type averaged across datasets (Mono: Monotonicity, Stab: Stability, Sens: Sensitivity, Cons: Consistency). Bold marks the best value per column.
MAESTRO
SMD
MAPS-S
Metric
Mono
Stab
Sens
Cons
Mono
Stab
Sens
Cons
Mono
Stab
Sens
Cons
CS-C
0.94
0.99
0.02
0.94
0.64
0.93
0.01
0.83
0.94
0.99
0.02
0.91
CS-S
0.95
0.97
0.04
0.97
0.75
0.92
0.02
0.88
0.97
0.96
0.04
0.94
DTW-C
0.94
0.99
0.02
0.94
0.93
0.99
0.02
0.93
0.95
0.99
0.02
0.91
DTW-D
0.28
0.76
0.06
0.59
0.55
0.71
0.08
0.64
-0.18
0.67
0.00
0.25
STI-BP
1.00
0.78
0.47
1.00
1.00
0.63
0.34
0.98
1.00
0.85
0.48
0.99
Table 2: Evaluation of well-performing methods on fine-grained onset corruption per dataset (Mono: Monotonicity, Stab: Stability, Sens: Sensitivity, Cons: Consistency). Bold marks the best value per column.
Dataset
CS-S
DTW-C
STI-BP
STI-BD
STI-TK
SMD
71±6
92±1
31±19
58±31
61±32
MAESTRO
74±3
91±1
34±10
82±8
87±5
MAPS-S
78±3
93±1
47±8
90±4
93±2
ENSTDkAm
72±2
89±1
33±6
51±8
67±8
ENSTDkCl
75±2
89±2
39±9
64±9
73±8
MusicNet
71±2
87±4
25±8
42±12
45±11
Table 3: Dataset alignment quality according to the methods with highest monotonicity performance ( mean±std , rescaled to [0,100] ). Bold marks the best performance and underline marks the second-best performance per column.
In recent years, thanks to advances in automatic music transcription (AMT), several large-scale datasets of automatically transcribed piano solo music have been released. While these datasets undoubtedly offer extensive material for performance studies, they vary substantially in quality. In the case of classical music, performances often differ not only in expressive aspects such as tempo, but also in their structural interpretation of the score (including repeat patterns and edition-specific variants). To meaningfully use large-scale transcribed datasets for performance research, transcriptions of the same piece must be grouped according to their underlying structural realisation to support valid comparison. We address this by applying sequence-to-sequence alignment followed by hierarchical clustering: we create pairwise alignments for all pairs of transcriptions of a given piece, and use the alignment cost and (dis)similarity of performed sequence lengths to resolve structural mismatches as features for grouping. We propose this approach as a first step towards automatically evaluating large-scale transcribed datasets that lack ground-truth score and/or audio, shifting the evaluation criterion from truth-based accuracy to musical coherence and plausibility. We demonstrate our score-agnostic approach on around 1,500 transcriptions of 88 compositions from a recently published large-scale transcribed piano performance dataset.
Patricia Hu, Silvan Peter, Gerhard Widmer
Institute of Computational Perception, Johannes Kepler University, Linz, Austria · LIT AI Lab, Linz Institute of Technology, Linz, Austria
Precise note-level annotations are critical for training automatic music transcription (AMT) systems, in particular note-onset labels, which form a core component of many recent AMT systems. However, high-quality annotations for real-world recordings are scarce. Sequence-level score--audio alignment methods such as dynamic time warping provide only coarse correspondence, making a local refinement step necessary. This refinement step, known as snapping, adjusts aligned score onsets using peaks in a neural onset posteriorgram and often determines whether weakly aligned score--audio pairs become usable training data at all. Despite its practical importance, snapping is typically treated as a simple post-processing heuristic and implemented with greedy local decisions. We present a systematic analysis of snapping strategies for training instrument-agnostic transcribers, demonstrating that snapping is essential for learning from weakly aligned data. Building on this, we formulate snapping as a per-pitch assignment problem and solve it via bipartite graph matching, yielding context-aware onset decisions under overlapping refinement windows and uncertain initial alignments. Extensive cross-dataset experiments across piano, chamber, and orchestral recordings show improved onset alignment and transcription accuracy over greedy snapping, with gains increasing for wider snapping windows and coarser initial alignments. Qualitative examples are provided on our project page: https://abhirupsaha8.github.io
Audio-to-score alignment is a long-standing challenge in music information retrieval and arguably the most widely applicable alignment task for music research. Alignment algorithms match two versions of a piece of music, and for this to work these versions need to be in comparable formats. Audio-to-audio alignment matches audio features; when matching audio files to scores, they must either synthesize the score or derive audio-like features by means of piano rolls or similar feature sequences. Symbolic alignment, by contrast, matches symbolically encoded notes; in an audio-to-score scenario these would be obtained by a transcription of the audio file. In this article, we present an algorithm that bridges audio-like and symbol-level features directly. Sequential audio features encoding onset and spectral activation are matched to score positions by a bespoke dynamic programming-based matching algorithm derived from symbolic alignment methods. The resulting method is both precise - surpassing widely used audio-to-audio approaches based on synthesized scores -, and remains flexible in its digital signal processing components, i.e., the method is adaptable to diverse timbral characteristics without requiring a separate transcription model. Furthermore it inherits some of the symbolic alignment runtime advantages with an algorithmic complexity that is at worst linear in the length of the (typically short) symbolic score and (typically long) audio feature sequence. In the following sections, we provide a detailed algorithm description and evaluate its alignment quality on a large-scale dataset of solo piano recordings.
Silvan Peter, Patricia Hu, Gerhard Widmer
Institute of Computational Perception, Johannes Kepler University, Linz, Austria · LIT AI Lab, Linz Institute of Technology, Linz, Austria