We introduce Non-musical Audio to Symbolic Music Transformation (NASMT), a novel task that converts non-musical audio into symbolic music representations. Unlike automatic music transcription, which assumes musical input and a unique correct score, NASMT is inherently one-to-many and requires evaluation along two axes: input correspondence and musicality. We present Sori, an encoder--decoder Transformer that addresses this task through domain adversarial training (DAT), which bridges the gap between musical and non-musical audio in the encoder's feature space, and classifier-free guidance (CFG) with a continuous parameter~γ that controls the trade-off between correspondence and musicality at inference time. Across objective metrics and a listening study, Sori provides effective continuous control via γ and compares favorably with instrument-agnostic transcription models. Sori supports real-time streaming on a consumer CPU. Audio examples are available at https://soridemo0320.github.io/sori-demo/.
Figures & tables
Figure 1: Illustration of the NASMT task. A CQT spectrogram of non-musical input (a cough sound) is overlaid with the MIDI piano roll generated by Sori. The output aligns with the temporal and harmonic structure of the input while forming musically coherent note patterns. Note that the CQT display is used for visualization only; the model operates on log-Mel spectrograms.
Figure 2: Sori architecture. The encoder maps log-Mel frames to latent features; the Temporal Decoder cross-attends over encoder output to produce per-timestep context vectors; the Event Decoder generates note events autoregressively within each timestep.
Sori
Sori (w/o DAT)
Reference
γ
0.0
0.5
1.0
2.0
3.0
0.0
0.5
1.0
2.0
3.0
Basic Pitch
Deep Salience
TimbreTrap
Objective
OnF1, ↑
0.087
0.236
0.327
0.363
0.368
0.282
0.340
0.381
0.390
0.386
0.210
0.300
0.167
MSS, ↓
57.2
32.7
18.3
13.1
11.8
25.4
17.1
11.3
10.1
10.2
32.8
21.7
33.1
FMD, ↓
143.6
33.6
51.3
161.8
248.0
96.7
96.4
191.3
288.7
324.0
181.8
195.2
472.7
Subjective
Cor., ↑
1.20
1.64
2.61
2.97
2.52
1.39
1.79
2.50
2.66
2.54
2.35
2.63
2.51
Mus., ↑
2.19
3.34
3.27
2.59
2.20
2.78
3.32
2.72
2.07
1.98
2.15
2.37
1.51
Table 1: Objective and subjective evaluation on the AudioSet evaluation set. OnF1: onset F1 averaged over 11 DSP-based onset detectors with a ±100 ms tolerance. MSS: Multi-Scale Spectrogram distance (lower = more similar). FMD: Fréchet Music Distance via CLaMP 2. Cor./Mus.: mean listener ratings of input correspondence and musicality. Instrument-agnostic single-track AMT models are used as reference methods.
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
Simon Rouard, Michael Krause, Axel Roebel +2
Kyutai · Mirelo AI · UMR STMS, IRCAM-CNRS Sorbonne Univ.
Generalizable Audio-to-Score (A2S) transcription is fundamentally constrained by the severe scarcity of high-quality, real-world paired data. Relying solely on existing human-annotated datasets often restricts the generalization of A2S models, limiting their efficacy primarily to single-instrumentation domains. To break this dependency on scarce real-world data, we introduce TUTTI (Transformer for Unified audio-To-score Transcription trained on Synthetic multi-Instrumentation Data), a pre-training paradigm driven by a purely synthetic, large-scale dataset. Rather than using human-composed scores, we leverage a symbolic music generation model to generate a massive, highly scalable multi-instrumentation corpus and create audio-score pairs with expressive acoustic characteristics. Capitalizing on the generated data, we employ a standard Transformer encoder-decoder architecture. We empirically demonstrate that pre-training a unified attention-based model on generated, multi-instrumentation data yields a consistently stronger foundational representation than single-instrumentation training. When fine-tuned with downstream real-world datasets, TUTTI outperforms previous approaches, establishing new overall state-of-the-art results across various A2S baselines. Notably, TUTTI shows remarkable cross-instrument transferability, effectively adapting to unseen instruments with highly competitive performance. The source code and the TuttiCorpus dataset will be made publicly available at https://github.com/a-musiclover/TUTTI.
Jianhuai Hu, Yashan Wang, Shangda Wu +7
Central Conservatory of Music · Independent Researcher · Wuhan Conservatory of Music +2
Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarce, and accurate local predictions can still produce inconsistent musical sequences. We present SheetSage2, a unified music transcription framework that combines synthetic data, task-specific structured decoding, and autoregressive distillation. Automatically annotated MIDI, rendered into audio, provides scalable supervision across music understanding tasks. Task-specific structured decoders integrate complementary musical cues and their temporal dependencies to produce musically coherent scores. Autoregressive distillation further retains transcription accuracy without task-specific dynamic programming at inference. Across eight benchmark collections, a single SheetSage2-AR model exceeds the listed prior systems on 12 of 15 benchmark--metric pairs in our evaluation, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks. Model weights and inference code are publicly available.