LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument

Kyungsu Kim; Yejin Kim; Kyogu Lee

cs.SDOct 9, 2026

LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument

Authors: Kyungsu Kim, Yejin Kim, Kyogu Lee

Organizations: Music and Audio Research Group (MARG) · Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University

Abstract

Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.

Figures & tables

Figure 1: Overview of LiveSynth. (a) Conditional adversarial training of the feedback-free continuous-latent generator. (b) Frame-wise streaming synthesis conditioned on MIDI and audio- or text-derived timbre embeddings, followed by causal VAE decoding.

	NSynth						Slakh
Method	MERT $\uparrow$	MSS $\downarrow$	PANNs $\uparrow$	F1 $\uparrow$	CE $\uparrow$	PQ $\uparrow$	MERT $\uparrow$	MSS $\downarrow$	PANNs $\uparrow$	F1 $\uparrow$	CE $\uparrow$	PQ $\uparrow$
Ground truth	1.000	0.0	0.912	0.476	4.52	6.77	1.000	0.0	0.800	0.785	5.22	7.45
VAE reconstruction	0.773	9.5	0.871	0.288	4.53	6.43	0.797	7.7	0.783	0.564	5.03	7.24
TokenSynth [ 13 ]	0.465	26.0	0.794	0.317	4.00	6.88	0.498	28.7	0.713	0.391	3.97	7.10
CTD [ 5 ]	0.438	19.5	0.755	0.361	4.98	6.78	0.495	16.1	0.735	0.339	5.03	7.08
LiveSynth (ours)	0.540	13.1	0.819	0.210	4.16	5.95	0.506	12.4	0.743	0.305	4.21	6.61

Table 1: Objective evaluation of zero-shot instrument cloning on 530 NSynth and 480 Slakh clips. Ground truth and VAE reconstruction are shown as references. † AnySynth results are self-reported on different instrument and sample sets and averaged across multiple reference durations; they are included for context and excluded from ranking.

Method	MusicCoCa $\uparrow$	F1 $\uparrow$	CE $\uparrow$	PQ $\uparrow$
Real instrument	0.152	0.238	3.78	6.99
TokenSynth [ 13 ]	0.165	0.202	3.74	6.97
w/ Procrustes	0.177	0.219	3.76	7.00
LiveSynth (ours)	0.149	0.181	4.27	6.61
w/ Procrustes	0.165	0.279	4.64	6.60

Table 2: Objective evaluation of text-to-instrument.

Figure 2: PANNs similarity to endpoint timbres

A

and

B

during a streamed 10 s timbre morph. Similarities are computed over 2 s sliding windows with a 1 s hop; curves show the mean and 95% CI across 64 instrument pairs.

Condition	PCH-JSD	IOI-JSD	CE $\uparrow$	PQ $\uparrow$
True continuation	0.251	0.204	4.73	6.60
LiveSynth (matched)	0.229	0.529	4.42	5.68
LiveSynth (shuffled pairing)	0.655	0.561	–	–

Table 3: Objective evaluation of autonomous 5 s performance continuation.

Music Generation

Music generation studies computational methods for composing, synthesizing, and editing music from symbolic, audio, textual, and multimodal inputs.

See all 107 papers

Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance
Jul 13, 2026Chong Jing, Junan Zhang, Jing Yang +3Diffusion Model Guidance Controllable Music Generation
Chinese University of Hong Kong · Central Media Technology Institute, Huawei
AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text
Oct 7, 2026Tristan Wu, Daniel Chin, Liwei Lin +2Music Generation
The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +1
MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator
Sep 24, 2026Yuan-Chiao Cheng, Zhiyao DuanTimbre-Conditioned Music Generation Controllable Music Generation
Music Informatics Group Georgia Institute of Technology Atlanta, GA, USA · Audio Information Research (AIR) Lab University of Rochester Rochester, NY, USA

Jul 13, 2026J/K move · Enter open · S save

Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance

Chong Jing, Junan Zhang, Jing Yang, Yulun Wu+2

Chinese University of Hong Kong · Central Media Technology Institute, Huawei

Diffusion Model Guidance Controllable Music Generation

Open paper AskPDF arXiv

Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/

Abstract

Music Generation

Explore similar work

Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance

AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator