KeyWise: Efficient Video-Only Piano Transcription with a Lightweight Per-Key Architecture
Organizations: Department of Informatics and Telecommunications, University of Thessaly, Lamia, Greece.
Abstract
Audio-based piano transcription is hampered by the sustain pedal and by noisy or missing soundtracks, problems that video-only transcription avoids, since a top-view video shows directly when each key is pressed and released. Most visual transcription networks, however, describe the whole keyboard with one global feature vector and must learn where each key is. In this paper, we propose KeyWise, a lightweight per-key architecture that exploits the known geometry of the keyboard instead of learning it. The rectified keyboard is divided into 88 equal tiles that serve as regions of interest (ROIs), and one small recurrent network, shared by all keys, follows each key over time. On PianoVAM, KeyWise reaches the onset accuracy of the state-of-the-art V2N (95.0% vs. 94.7% onset F1 at 50 ms and 97.8% vs. 97.0% at 100 ms) with 1.3 M instead of 28.2 M parameters, and it runs 4.7 times faster on the same GPU. At 100 ms, it is also significantly more accurate when the key releases (97.0% vs. 95.5% F1) or the velocities (89.4% vs. 84.5% F1) must be correct as well. On PianoYT, KeyWise pre-trained on PianoVAM reaches 69.4% onset F1 at 100 ms, comparable to the best published result, although it needs fine-tuning for new recording conditions.
Figures & tables
| Onset | +Off | +Vel | +Off+Vel | |||||||
| Model | Params | Frame | 50 | 100 | 50 | 100 | 50 | 100 | 50 | 100 |
| S2S Koepke et al. (2020) a | – | 41.7 | 60.1 | 94.5 | 21.8 | 49.6 | – | – | – | – |
| V2R Su et al. (2020) a | – | 82.2 | 86.5 | 89.8 | 55.4 | 81.8 | – | – | – | – |
| Li et al. Li et al. (2024) a,b | – | (25.2) | 94.2 | 97.4 | (21.0) | (40.0) | – | – | – | – |
| PPAN Zivanovic et al. (2025) a | – | 80.4 | 84.9 | 94.1 | 45.9 | 82.8 | – | – | – | – |
| V2N Kim et al. (2026) , published | 28.2 M | 90.9 | 94.7 | 97.0 | 89.5 | 95.5 | 82.8 | 84.5 | 78.3 | 83.2 |
| Model | Extra data | 50 ms | 100 ms |
|---|---|---|---|
| S2S Koepke et al. (2020) a | R3 | – | 64 |
| V2R Su et al. (2020) a | R3 | – | 64 |
| PPAN Zivanovic et al. (2025) | video + R3 | – | 68 |
| KeyWise | none | ||
| KeyWise, zero-shot b | PianoVAM | 6.5 | 15.3 |
| KeyWise, fine-tuned | PianoVAM |
| KeyWise | Global | G-88 | |||
| Parameters | 1.3 M | 3.7 M | 7.2 M | ||
| Onset F1, 50 ms | 95.0 | 94.1 | 92.4 | 0.250 | 0.078 |
| Onset F1, 100 ms | 97.8 | 95.6 | 95.2 | 0.020 | 0.020 |
| +Off F1, 50 ms | 89.9 | 86.9 | 83.9 | 0.148 | 0.078 |
| +Off F1, 100 ms | 97.0 | 93.9 | 93.3 | 0.020 | 0.020 |
| Frame F1 | 91.4 | 90.0 | 89.1 | 0.117 | 0.059 |
Automatic Music Transcription
Automatic music transcription converts recorded music into symbolic notes, rhythms, instruments, and sometimes readable scores or tablature.
See all 39 papers