Sori: Streamable and Controllable Model for Non-musical Audio to Symbolic Music Transformation

We introduce Non-musical Audio to Symbolic Music Transformation (NASMT), a novel task that converts non-musical audio into symbolic music representations. Unlike automatic music transcription, which assumes musical input and a unique correct score, NASMT is inherently one-to-many and requires evaluation along two axes: input correspondence and musicality. We present Sori, an encoder--decoder Transformer that addresses this task through domain adversarial training (DAT), which bridges the gap between musical and non-musical audio in the encoder's feature space, and classifier-free guidance (CFG) with a continuous parameter~$γ$ that controls the trade-off between correspondence and musicality at inference time. Across objective metrics and a listening study, Sori provides effective continuous control via $γ$ and compares favorably with instrument-agnostic transcription models. Sori supports real-time streaming on a consumer CPU. Audio examples are available at https://soridemo0320.github.io/sori-demo/.