Neural instrument synthesizers can clone the timbre of a solo instrument, letting music producers make music with the sound they desire. However, an instrument of interest is often embedded in a fully produced, multi-instrument recording, beyond the reach of existing methods. Source separation before timbre cloning is an approach, but it may suffer from separation artifacts and is also likely a detour for the task. In this work, we propose Point-and-Play, a system that clones the timbre of a cued source directly from a polyphonic mixture and uses it to synthesize new pitches and velocities. To achieve this, we introduce a Cue-Gated Statistics Pooling (CGSP) module that extracts the timbre code of the cued instrument from adapted CLAP embeddings of the music mixture. This extracted timbre code is then used to condition a separately trained note synthesizer together with pitch and velocity inputs. Experiments show that our proposed system outperforms separate-then-clone baselines in terms of timbre fidelity and synthesis quality, achieving comparable performance with that cloned from a clean solo instrument.
Figures & tables
Figure 1: Overview of Point-and-Play. Given a polyphonic mixture and an audio or text cue, the system extracts the cued source’s timbre from the mixture and uses it to render new pitches and velocities.
Figure 2: Point-and-Play architecture. A CLAP encoder fine-tuned with LoRA encodes the mixture, while a frozen CLAP encoder encodes the source cue. Cue-Gated Statistics Pooling combines both representations into a timbre code of the cued source in the mixture and uses it to condition the note synthesizer. During training, the isolated source of the mixture provides supervision; it is not required at inference time.
System and conditioning input
CLAP ↑
FAD ↓
MSS ↓
Oracle Timbre
Target note
0.811
1.78
2.04
Target-instrument other note
0.762
1.85
2.38
Cue Timbre
Text cue
0.548
2.27
3.84
Target-category audio cue
0.584
1.82
3.25
Table 1: Generation results on 27 held-out NSynth identities. CLAP score is the generated-to-target audio similarity. Arrows show the preferred direction, and bold marks the best Point-and-Play value.
Figure 3: Decoder-free paired retrieval Recall@1 versus mixture duration on NSynth and MoisesDB. Only the centered crop duration varies. Solid and dashed lines denote text and same-category audio cues, respectively; colors and markers distinguish datasets.
Objective
Txt ↑
Aud ↑
W t↓
W a↓
Full objective (ours)
0.699
0.708
0.418
0.430
−Linv
0.654
0.640
0.399
0.428
−Lalgna
0.661
0.417
0.399
0.400
−Lalgnt
0.578
0.646
0.425
0.403
−Lsela
0.664
0.664
0.424
0.469
Table 2: Ablation of the encoder objective in Eq. ( 3 ). All reported values are CLAP scores. Txt and Aud use the matched category text and a clean same-category audio cue from a different instrument, respectively; W t and W a use mismatched text and audio cues. Lower wrong-cue scores indicate stronger cue selectivity.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Instrument timbre transfer re-voices a performance using the timbre of another instrument. Extracting the target timbre from an audio reference capture more nuances than inferring it from a text prompt. Systems that read timbre from such a clip train a dedicated model for the task, which captures the timbre cleanly but stays a narrow, single-purpose system. More versatile approaches add control to a pretrained music generator, yet a reference clip entangles timbre with genre and melody, so these systems fall back on text to name the timbre. We present MuseTimbre, the first system, to our best knowledge, that transfers timbre from an audio reference to a polyphonic source through conditioning a pretrained music generator. This system employs a multi-pitch estimator to extract pitch information from the source and finetune a CLAP encoder to extract timbre information from the reference audio. Experiments show that across four datasets of real polyphonic recordings, MuseTimbre achieves pitch alignment on par with the baselines while matching the reference timbre far more closely. Results also show that the finetuned CLAP-based timbre extractor is robust to pitch variations, making it useful in timbre similarity measures.
Yuan-Chiao Cheng, Zhiyao Duan
Music Informatics Group Georgia Institute of Technology Atlanta, GA, USA · Audio Information Research (AIR) Lab University of Rochester Rochester, NY, USA
Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.
Kyungsu Kim, Yejin Kim, Kyogu Lee
Music and Audio Research Group (MARG) · Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
Chong Jing, Junan Zhang, Jing Yang +3
Chinese University of Hong Kong · Central Media Technology Institute, Huawei