MIDI velocity is crucial for capturing expressive dynamics in human performances. In practical scenarios, a music score with inaccurate velocities may be available alongside the performance audio (e.g., music education and free online archives), enabling the task of score-informed MIDI velocity estimation. In this work, we propose a modular, lightweight score-informed Transformer correction module that refines the velocity estimates of Automatic Music Transcription (AMT) systems. Leveraging high-quality expressive datasets for rigorous evaluation, we focus on piano transcription, integrating our module into multiple existing systems. Trained exclusively on the MAESTRO dataset, our method consistently reduces velocity errors and improves cross-dataset generalization to SMD and MAPS datasets. Under this training protocol, integrating our score-informed module with HPT (named Score-HPT) establishes a new state-of-the-art, outperforming existing score-informed methods and piano transcription models while adding only 1 M parameters. Furthermore, our framework remains robust to temporal misalignments caused by imperfect audio-score synchronization, a common real-world challenge.
Figures & tables
Figure 1: Comparison between the proposed and previous score-informed MIDI velocity estimation approaches.
Figure 2: Proposed score-informed MIDI velocity estimation framework. The system includes a velocity estimation module, which is identical to the velocity branch of the piano AMT system. The correction module then rectifies the preliminary velocity estimates using features extracted from the MIDI score.
Table 1: Comparison of HPT and Score-HPT on the MAESTRO test set. ↓ indicate a lower value is better.
Figure 4: Velocity estimates from the HPT baseline and the proposed Score-HPT. Referencing the given MIDI note features, the refinement effect is evident. Both HPT and Score-HPT provide a fully-populated velocity estimation.
Figure 5: Comparison between HPT and Score-HPT variants using different score features. Models were trained on the MAESTRO train set for 120k iterations and evaluated every 10k iterations across three validation sets.
Model
SMD dataset
MAE ↓
STD ↓
# Params
Score-Inf. Methods
DiffVel [ 27 ]
19.7
13.1
–
FiLM Conv [ 15 ]
15.1
12.3
–
FiLM U-Net [ 14 ]
9.9
7.8
8.8 M
Proposed
Table 2: Comparison of different methods on the SMD dataset. ↓ indicate a lower value is better.
Figure 6: Robustness evaluation on data derived from the SMD dataset, using audio-score misalignments of ± 0.1, 0.2, 0.3, 0.4, 0.5, 1, 1.5, 2, and 2.5 s.
Automatic Music Transcription (AMT) relies on paired audio--MIDI datasets, implicitly assuming near-perfect temporal alignment. However, widely used piano datasets have been shown to contain alignment errors that may bias model evaluation and training. We propose a framework for evaluating audio--MIDI alignment scoring methods through controlled temporal, pitch, and structural MIDI corruptions. We also introduce STI, a scoring method built on AMT models, and benchmark it against existing alignment scoring approaches. Our results show that STI, together with methods based on CENS and spectrograms, achieves the strongest overall performance, with STI exhibiting the highest sensitivity to fine-grained temporal misalignments, the main alignment error in AMT datasets. We then apply the best-performing methods to widely used piano AMT datasets and identify trends in their relative alignment quality. These results support more informed dataset curation, model training, and AMT evaluation.
Lucas de Souza Brandão, Lucas Gris, Joaquim Cavalcante +2
Universidade de São Paulo, Brazil · Universidade Federal de Goiás, Brazil · Universidade Federal da Paraíba, Brazil +1
Transcribing expressive piano performances into scores requires interpreting individual note timings within an underlying metrical structure. However, score attributes are associated with notes, whereas beats may occur without coincident note onsets. We introduce DuET, a Dual-Encoder Transformer that jointly learns note-level performance MIDI-to-score (PM2S) transcription and frame-level beat and downbeat tracking. Bidirectional cross-attention connects note and frame encoders, while an autoregressive decoder alternates between their representations to predict structured score attributes. We further introduce TimeRoPE and Time-Pitch Rotary Positional Encoding (TPRoPE) to establish a shared performance-time reference and explicitly encode relative temporal and pitch relationships. Evaluations on ASAP, a classical piano dataset, and our curated Gangqinpu dataset, which spans multiple genres with a focus on popular piano music, demonstrate improvements in score timing and notated duration. Compared with the strongest baseline on each dataset, DuET reduces MUSTER mean error from 6.56% to 3.64% on Gangqinpu and from 11.30% to 11.22% on ASAP. Ablations support the benefits of frame-level context, time-based positional encoding, and decoder-history masking. Code, weights, and demos are available at https://github.com/wei-zeng98/DuET.
Wei Zeng, Ye Wang
Integrative Sciences and Engineering Programme, NUS Graduate School · School of Computing, National University of Singapore
Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
Yonghyun Kim, Hoyeol Sohn, Juhan Nam +1
Music Informatics Group, Georgia Institute of Technology, USA · Graduate School of Culture Technology, KAIST, South Korea