Timbre Analysis of the Hulusi, a Southwestern Chinese Free-Reed Instrument, using Machine Learning

Multi-pitch estimation (MPE) typically predicts which pitches are active in a mixture, but not which instrument or source produced them. This paper investigates a lightweight slot-attention framework for multi-instrument MPE (MI-MPE), where a mixture CQT is mapped to an unordered set of source-like pitch maps. The model uses permutation-invariant Hungarian matching to avoid fixed output semantics and treats the number of slots as an upper bound on the number of active sources. We further study two modular extensions: a self-supervised timbre encoder that provides training-time targets for slot-level timbre embeddings, and a polyphony branch that regularizes the pitch density of mixture- and slot-level predictions. Experiments show that Hungarian matching substantially improves instrument family decomposition on URMP. Stem-level prediction remains more challenging: timbre and polyphony supervision improve selected configurations, but do not consistently resolve source assignment. The results suggest that slot-based architectures are a promising direction for source-aware MPE, while highlighting the need to couple auxiliary musical cues to slot identity more carefully.

In this article, we present a method of spectral serialization envisioned by G. Mahler and A. Schoenberg between the late 19th and early 20th centuries that enables the selection of timbres rooted in musical structure. Based on information geometry, we evaluate the Wasserstein distance between polyphonic timbre spectra, as demonstrated in our previous paper, in terms of the second moment of spectral chords comprising three or more notes, expressed in squared hertz. By embedding these quantities in a Banach space, we analyze specific passages from works by R. Strauss, Mahler, A. Berg, and A. Webern from the perspective of timbre, showing how the same chord exhibits different affinities and divergences depending on orchestration. Taking the wah wah mute as an example, we propose a model that treats instrumental timbre and speech within the same Banach space, and we attempt to quantify the "vowel triangle" in Hz. These efforts constitute a partial but integrated solution to the historical three problems of Schoenberg. Serialism, Klangfarbenmelodie, and Sprechgesang which neither Mahler nor Schoenberg in their lifetimes, nor the postwar generation of composers including Nono, Boulez, and Stockhausen, was able to resolve fully. We aim to reconsider musical thought as a whole from the perspective of timbre and to open new possibilities for interpretation, performance, and composition. All tools are publicly available on GitHub, so that anyone can reproduce and extend these analyses.

Instrument timbre transfer re-voices a performance using the timbre of another instrument. Extracting the target timbre from an audio reference capture more nuances than inferring it from a text prompt. Systems that read timbre from such a clip train a dedicated model for the task, which captures the timbre cleanly but stays a narrow, single-purpose system. More versatile approaches add control to a pretrained music generator, yet a reference clip entangles timbre with genre and melody, so these systems fall back on text to name the timbre. We present MuseTimbre, the first system, to our best knowledge, that transfers timbre from an audio reference to a polyphonic source through conditioning a pretrained music generator. This system employs a multi-pitch estimator to extract pitch information from the source and finetune a CLAP encoder to extract timbre information from the reference audio. Experiments show that across four datasets of real polyphonic recordings, MuseTimbre achieves pitch alignment on par with the baselines while matching the reference timbre far more closely. Results also show that the finetuned CLAP-based timbre extractor is robust to pitch variations, making it useful in timbre similarity measures.