cs.SDOct 15, 2026

DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription

Authors: Wei Zeng, Ye Wang

Organizations: Integrative Sciences and Engineering Programme, NUS Graduate School · School of Computing, National University of Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transcribing expressive piano performances into scores requires interpreting individual note timings within an underlying metrical structure. However, score attributes are associated with notes, whereas beats may occur without coincident note onsets. We introduce DuET, a Dual-Encoder Transformer that jointly learns note-level performance MIDI-to-score (PM2S) transcription and frame-level beat and downbeat tracking. Bidirectional cross-attention connects note and frame encoders, while an autoregressive decoder alternates between their representations to predict structured score attributes. We further introduce TimeRoPE and Time-Pitch Rotary Positional Encoding (TPRoPE) to establish a shared performance-time reference and explicitly encode relative temporal and pitch relationships. Evaluations on ASAP, a classical piano dataset, and our curated Gangqinpu dataset, which spans multiple genres with a focus on popular piano music, demonstrate improvements in score timing and notated duration. Compared with the strongest baseline on each dataset, DuET reduces MUSTER mean error from 6.56% to 3.64% on Gangqinpu and from 11.30% to 11.22% on ASAP. Ablations support the benefits of frame-level context, time-based positional encoding, and decoder-history masking. Code, weights, and demos are available at https://github.com/wei-zeng98/DuET.

Figures & tables

AttributeNameValuesDescription
pip_{i}Pitch{0,…,127}\{0,\ldots,127\}MIDI pitch copied from the input note
bib_{i}Bar indexbi−a∈{−256,…,255}b_{i}-a\in\{-256,\ldots,255\}Encoded relative to window anchor aa
qiq_{i}Onset integer part{0,…,63}\{0,\ldots,63\}Whole quarter-note units from bar start
fif_{i}Onset fractional partF⊂[0,1)\mathcal{F}\subset[0,1)Fractional quarter-note offset
ℓi\ell_{i}DurationD⊂[0,51]\mathcal{D}\subset[0,51]Notated duration; zero denotes a grace note
sis_{i}Staff{0,1}\{0,1\}Upper or lower staff, respectively
Table 1: Attributes in each note-aligned score record. Score-time values are measured in quarter-note units.
Figure 1: Overview of DuET within the audio-to-score pipeline. Each dual-encoder layer combines branch-specific self-attention, bidirectional cross-attention, and feed-forward networks. Successive decoder layers attend to note and frame memories in alternation, predicting one score record per input note. Rotary positional encoding is applied to queries and keys. Residual connections and layer normalization are omitted for clarity.
Figure 2: Positional coordinates for the same four notes. IndexRoPE uses sequence indices, TimeRoPE uses scaled performance timestamps, and TPRoPE uses both performance timestamps and pitch coordinates. The dashed vertical lines indicate the 20 ms frame grid; note onsets need not coincide with this grid.
 MUSTER [ 31 , 20 ]ScoreSimilarity [ 32 , 33 ]
MethodPitchMissExtraOnsetOffsetMeanMissExtraDurationStaff
ASAP
MuseScore [ 16 ]2.417.359.6447.9049.4423.3516.1716.7455.2321.87
HMMs (J-pop) [ 19 ]2.096.388.6725.0229.2114.2710.8011.3971.38–
HMMs (classical) [ 19 ]2.116.478.7522.5829.8413.9510.7411.2864.73–
Neural beat tracking [ 10 ]2.026.819.0168.2854.1128.0417.1017.6766.986.86
Table 2: PM2S transcription errors (%). Lower is better. Bold and underlined values indicate the best and second-best results within each dataset, ranked before rounding where full-precision results are available. ASAP baseline results are taken from Beyer and Dai [11] .
 MUSTER [ 31 , 20 ]ScoreSimilarity [ 32 , 33 ]
ModelPitchMissExtraOnsetOffsetMeanMissExtraDurationStaff
ASAP
DuET2.255.9511.107.6616.428.689.2812.6930.583.99
Note encoder only2.355.9410.9410.8522.4210.509.7113.1152.234.74
Index / Alternating †2.356.9212.418.6719.009.8710.2713.6736.624.73
Time / Alternating2.8411.4116.9513.2124.0013.6815.8819.2842.984.93
Table 3: PM2S ablations on fixed common-valid subsets of 39 ASAP and 72 Gangqinpu performances. Errors are percentages; lower is better. Bold and underlined values indicate the best and second-best results within each dataset, ranked before rounding. Positional/memory variants are named as “RoPE / decoder memory”. DuET uses Time–Pitch RoPE with alternating memory access. Thin rules separate encoder, positional/memory, and masking controls.
 ASAPGangqinpu
ModelBeatDownbeatBeatDownbeat
DuET68.0165.5577.3479.94
Frame-only tracker61.1955.5178.8681.37
Dual-encoder tracker62.8953.9780.2480.73
Index / Alternating †62.3461.6670.6073.09
Time / Alternating65.7360.4581.2482.50
Table 4: Beat-tracking ablations on 59 ASAP and 100 Gangqinpu performances. Values are macro F1 scores (%); higher is better. Bold and underlined values indicate the best and second-best results in each column. Positional/memory variants are named as “RoPE / decoder memory”. DuET uses Time–Pitch RoPE with alternating memory access.
Figure 3: Qualitative comparison of score excerpts reconstructed from the same performance MIDI: (a) DuET, (b) Beyer and Dai, and (c) Liu’s PM2S. All outputs are rendered with MuseScore 4.
More in this topic

Automatic Music Transcription

Automatic music transcription converts recorded music into symbolic notes, rhythms, instruments, and sometimes readable scores or tablature.

See all 34 papers

Explore similar work

CardsList
Oct 21, 2026

A Learned Musical-Time Coordinate for Piano Audio-to-Score Transcription

End-to-end piano audio-to-score (A2S) transcription is the task of reconstructing a notated score from an audio recording in a single system. Existing A2S approaches usually rely on pre-segmented excerpts, and few handle whole pieces without external segmentation. This raises the question of whether a musical-time coordinate can guide score transcription while also helping the system segment whole pieces. Previous work on jazz saxophone A2S uses an external beat tracker to obtain such a coordinate for score prediction. We introduce downbeat phase, a musical-time coordinate learned jointly with score transcription that represents each audio frame's relative position within a notated bar. With this coordinate, our system can not only condition its score decoder on downbeat phase but also self-segment at bar boundaries and stitch the segments into a whole piece. Evaluation suggests that the coordinate improves metrical structure and supports whole-piece transcription with comparable notation quality.

Min-Hsing Wang, Tsung-Ren Huang

National Taiwan University

Automatic Music Transcription
AskPDF
May 22, 2026

Rubato: Transcribing Piano Music with Timestamps

We consider the conversion of musical recordings into human-readable sheet music annotated with timestamps. Such output lets a listener clearly visualize rubato (temporally expressive playing), a learner diagnose ensemble precision and timing choices against the written music, and a musicology scholar compare performance styles across recordings of the same work. We introduce (1) a prompt-conditioned encoder-decoder model, named Rubato, trained to output (2) a new textual representation for polyphonic music, named InterMo, which we designed for compatibility with sequence-to-sequence training. Our experiments demonstrate that Rubato produces timestamped piano sheet music from audio with higher notational accuracy than the best existing approaches, which are based on cascades. We find that even if the cascade is given ground-truth MIDI instead of audio, Rubato performs better, suggesting that the ceiling of existing approaches is primarily representational, not acoustic. Further, because Rubato is trained on several related tasks (with prompts), it competes with or outperforms the best single-task systems on related but simpler tasks like MIDI note grounding and beat/downbeat detection. A demo is available at https://nctamer.github.io/rubato-transcription .

Nazif Can Tamer, Victoria Ebert, Guang Yang +1

Paul G. Allen School of Computer Science & Engineering, University of Washington · Allen Institute for AI

Symbolic Music GenerationSequence-to-Sequence ModelsAutomatic Music TranscriptionMusic Information RetrievalSequence Modeling
AskPDF
Oct 26, 2026

Gangqinpu: A Multi-Genre, Multi-Task Piano Dataset with Aligned Audio, MIDI, and Musical Scores

Aligned audio, performance MIDI, and musical scores provide valuable supervision for music analysis and performance modeling, yet datasets that jointly align all three modalities remain scarce and limited in scale and genre diversity. We introduce Gangqinpu, a multi-genre piano dataset containing 18,786 score-performance pairs and 1,007.1 hours of source audio, with automatic note-level alignments and derived beat/downbeat annotations. It spans diverse genres, including pop, classical, jazz, folk/traditional, rock, and blues. Our automated pipeline transcribes piano audio, aligns the resulting MIDI with musical scores, assigns genre labels, and filters out non-expressive performances with a score-informed binary classifier. We further construct joint splits with ASAP using conservative score-similarity grouping to reduce data leakage. We evaluate Gangqinpu on performance MIDI-to-score transcription, expressive performance rendering, and beat tracking. Downstream experiments demonstrate the value of Gangqinpu as complementary training data, with cross-dataset gains in beat tracking and primarily in-domain benefits for transcription and expressive performance rendering. The code and dataset are publicly available at https://github.com/wei-zeng98/gangqinpu.

Wei Zeng, Anvay Mathur, Ye Wang

Integrative Sciences and Engineering Programme, NUS Graduate School · School of Computing, National University of Singapore · Singapore American School

Multimodal DatasetMusic Information RetrievalAutomatic Music Transcription
AskPDF