Point-and-Play: Instrument Timbre Cloning from Music Mixtures

Baotong Tian; Zhiyao Duan

cs.SDOct 14, 2026

Point-and-Play: Instrument Timbre Cloning from Music Mixtures

Authors: Baotong Tian, Zhiyao Duan

Organizations: Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA

Abstract

Neural instrument synthesizers can clone the timbre of a solo instrument, letting music producers make music with the sound they desire. However, an instrument of interest is often embedded in a fully produced, multi-instrument recording, beyond the reach of existing methods. Source separation before timbre cloning is an approach, but it may suffer from separation artifacts and is also likely a detour for the task. In this work, we propose Point-and-Play, a system that clones the timbre of a cued source directly from a polyphonic mixture and uses it to synthesize new pitches and velocities. To achieve this, we introduce a Cue-Gated Statistics Pooling (CGSP) module that extracts the timbre code of the cued instrument from adapted CLAP embeddings of the music mixture. This extracted timbre code is then used to condition a separately trained note synthesizer together with pitch and velocity inputs. Experiments show that our proposed system outperforms separate-then-clone baselines in terms of timbre fidelity and synthesis quality, achieving comparable performance with that cloned from a clean solo instrument.

Figures & tables

Figure 1: Overview of Point-and-Play. Given a polyphonic mixture and an audio or text cue, the system extracts the cued source’s timbre from the mixture and uses it to render new pitches and velocities.

Figure 2: Point-and-Play architecture. A CLAP encoder fine-tuned with LoRA encodes the mixture, while a frozen CLAP encoder encodes the source cue. Cue-Gated Statistics Pooling combines both representations into a timbre code of the cued source in the mixture and uses it to condition the note synthesizer. During training, the isolated source of the mixture provides supervision; it is not required at inference time.

Oracle Timbre
System and conditioning input	CLAP $\uparrow$	FAD $\downarrow$	MSS $\downarrow$
Target note	0.811	1.78	2.04
Target-instrument other note	0.762	1.85	2.38
Cue Timbre
Text cue	0.548	2.27	3.84
Target-category audio cue	0.584	1.82	3.25

Table 1: Generation results on 27 held-out NSynth identities. CLAP score is the generated-to-target audio similarity. Arrows show the preferred direction, and bold marks the best Point-and-Play value.

Figure 3: Decoder-free paired retrieval Recall@1 versus mixture duration on NSynth and MoisesDB. Only the centered crop duration varies. Solid and dashed lines denote text and same-category audio cues, respectively; colors and markers distinguish datasets.

Objective	Txt $\uparrow$	Aud $\uparrow$	W ${}_{t}\downarrow$	W ${}_{a}\downarrow$
Full objective (ours)	0.699	0.708	0.418	0.430
$-\,\mathcal{L}_{\rm inv}$	0.654	0.640	0.399	0.428
$-\,\mathcal{L}_{\rm algn}^{a}$	0.661	0.417	0.399	0.400
$-\,\mathcal{L}_{\rm algn}^{t}$	0.578	0.646	0.425	0.403
$-\,\mathcal{L}_{\rm sel}^{a}$	0.664	0.664	0.424	0.469

Table 2: Ablation of the encoder objective in Eq. ( 3 ). All reported values are CLAP scores. Txt and Aud use the matched category text and a clean same-category audio cue from a different instrument, respectively; W t and W a use mismatched text and audio cues. Lower wrong-cue scores indicate stronger cue selectivity.

Audio Generation

Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.

See all 133 papers

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator
Sep 24, 2026Yuan-Chiao Cheng, Zhiyao DuanTimbre-Conditioned Music Generation Controllable Music Generation
Music Informatics Group Georgia Institute of Technology Atlanta, GA, USA · Audio Information Research (AIR) Lab University of Rochester Rochester, NY, USA
LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument
Oct 9, 2026Kyungsu Kim, Yejin Kim, Kyogu LeeMusic Generation Controllable Music Generation
Music and Audio Research Group (MARG) · Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance
Jul 13, 2026Chong Jing, Junan Zhang, Jing Yang +3Diffusion Model Guidance Controllable Music Generation
Chinese University of Hong Kong · Central Media Technology Institute, Huawei

Sep 24, 2026J/K move · Enter open · S save

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator

Yuan-Chiao Cheng, Zhiyao Duan

Music Informatics Group Georgia Institute of Technology Atlanta, GA, USA · Audio Information Research (AIR) Lab University of Rochester Rochester, NY, USA

Timbre-Conditioned Music Generation Controllable Music Generation Audio Representation Learning Music Generation

Open paper AskPDF arXiv

Instrument timbre transfer re-voices a performance using the timbre of another instrument. Extracting the target timbre from an audio reference capture more nuances than inferring it from a text prompt. Systems that read timbre from such a clip train a dedicated model for the task, which captures the timbre cleanly but stays a narrow, single-purpose system. More versatile approaches add control to a pretrained music generator, yet a reference clip entangles timbre with genre and melody, so these systems fall back on text to name the timbre. We present MuseTimbre, the first system, to our best knowledge, that transfers timbre from an audio reference to a polyphonic source through conditioning a pretrained music generator. This system employs a multi-pitch estimator to extract pitch information from the source and finetune a CLAP encoder to extract timbre information from the reference audio. Experiments show that across four datasets of real polyphonic recordings, MuseTimbre achieves pitch alignment on par with the baselines while matching the reference timbre far more closely. Results also show that the finetuned CLAP-based timbre extractor is robust to pitch variations, making it useful in timbre similarity measures.

Abstract

Audio Generation

Explore similar work

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator

LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument

Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance