cs.CVNov 3, 2026

KeyWise: Efficient Video-Only Piano Transcription with a Lightweight Per-Key Architecture

Authors: Anastasios Chatzigiannis, Natalia Panagou, Maria Koziri

Organizations: Department of Informatics and Telecommunications, University of Thessaly, Lamia, Greece.

Abstract

Audio-based piano transcription is hampered by the sustain pedal and by noisy or missing soundtracks, problems that video-only transcription avoids, since a top-view video shows directly when each key is pressed and released. Most visual transcription networks, however, describe the whole keyboard with one global feature vector and must learn where each key is. In this paper, we propose KeyWise, a lightweight per-key architecture that exploits the known geometry of the keyboard instead of learning it. The rectified keyboard is divided into 88 equal tiles that serve as regions of interest (ROIs), and one small recurrent network, shared by all keys, follows each key over time. On PianoVAM, KeyWise reaches the onset accuracy of the state-of-the-art V2N (95.0% vs. 94.7% onset F1 at 50 ms and 97.8% vs. 97.0% at 100 ms) with 1.3 M instead of 28.2 M parameters, and it runs 4.7 times faster on the same GPU. At 100 ms, it is also significantly more accurate when the key releases (97.0% vs. 95.5% F1) or the velocities (89.4% vs. 84.5% F1) must be correct as well. On PianoYT, KeyWise pre-trained on PianoVAM reaches 69.4% onset F1 at 100 ms, comparable to the best published result, although it needs fine-tuning for new recording conditions.

Figures & tables

Explore similar work

Aug 4, 2026

Multi-Task Multi-Frame Visual Piano Transcription

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
Oct 29, 2026

A learning-based method of automatic music transcription

Automatic Music Transcription (AMT) seeks to convert acoustic music recordings into symbolic musical notation, a process that remains remarkably difficult for polyphonic piano due to concurrent polyphony, harmonic partial interference, expressive timing, and severe pitch activation sparsity. In this paper, we formulate polyphonic piano transcription as a frame-level multi-label classification problem across the standard 88 piano pitches (MIDI notes 21 to 108). Raw audio waveforms are processed using the Constant-Q Transform (CQT) to construct a geometrically spaced log-frequency time-frequency representation that aligns with the chromatic musical scale across seven octaves (252 frequency bins per frame). We systematically investigate recurrent sequence models, contrasting unidirectional Long Short-Term Memory (LSTM) with Bidirectional LSTM (BiLSTM) backbones, and propose two architectural adaptations tailored to recurrent temporal representations: a modified additive attention module and a modified multihead attention mechanism equipped with hyperbolic tangent activation and head-wise summation. Furthermore, to mitigate the severe class imbalance between inactive and active notes (approximately 41:1), we evaluate Weighted Binary Cross-Entropy (WBCE) alongside standard Binary Cross-Entropy (BCE). Rigorous empirical evaluations on the MAPS and MAESTRO datasets demonstrate steady, compounding gains from bidirectionality and attention mechanisms. Our proposed Multihead Attention-BiLSTM architecture attains an F1-score of 88.20%, an accuracy of 78.90%, a precision of 0.89, and a recall of 0.86, substantially outperforming the baseline unidirectional LSTM (F1-score of 71.20%) while maintaining a compact parameter budget of 4.43M. Qualitative piano-roll visualizations confirm that our model generates faithful transcriptions even in complex harmonic passages.
Sep 23, 2026

Statistical Models for Automatic Fingering-Annotated Piano Sheet Music Transcription

Machine learning tools have significantly aided automatic piano music transcription; however, this domain has focused primarily on accurately predicting the pitches and timings of played notes. To produce sheet music for the piano, notes must be separated into two staves, one for each hand, and good sheet music often contains fingering annotations to guide the player when sight-reading or learning fast or complex pieces. We propose 8 statistical approaches for combined hand and fingering annotation of transcribed piano notes, including baseline hidden Markov models, rule-based methods, a synthesis of existing approaches, and N-gram language models. Furthermore, we develop a pipeline for complete transcription from piano audio to fingering-annotated sheet music. Evaluations with the PIG dataset demonstrate that our Synthesis model achieves a hand separation accuracy of 90.8% and a joint hand and finger annotation accuracy of 56.6%. These approaches serve as a new baseline for further research into this problem, while our pipeline demonstrates the feasibility of a combined system for automated note transcription, hand separation, and fingering annotation.