AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text

Tristan Wu; Daniel Chin; Liwei Lin; Junan Zhang; Gus Xia

cs.SDOct 7, 2026

AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text

Authors: Tristan Wu, Daniel Chin, Liwei Lin, Junan Zhang, Gus Xia

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Audio generation models can translate natural-language descriptions into sound, but their outputs are typically waveforms. Their audio quality is constrained by audio compression, and their outputs do not readily support direct edits to notes, timbral parameters, or modulation relationships. We present AutoSynth, which represents MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes as a unified sequence for a synthesizer, and learns their dependencies with an audio-conditioned autoregressive model. A single model supports both tasks. Given reference audio, the model directly predicts a synthesizer program; given text, it uses a pretrained audio generation model and converts the generated audio into a program. Training consists of two stages: supervised learning on large-scale audio-program pairs automatically constructed from a small set of native presets, followed by group-relative policy optimization with a mixed reward combining semantic similarity, pitch-related features, acoustic similarity, and sound usefulness. The pipeline requires neither paired text-target-program annotations nor a differentiable synthesizer. Experiments show that AutoSynth produces complete, editable synthesizer programs and achieves competitive results in both synthesizer inversion and text-driven generation. Audio demos and source code are available at https://auto-synth.github.io/.

Figures & tables

Figure 1: Overview of AutoSynth.

Figure 2: Autoregressive program generation and decoder architecture.

Gemini–Woosh
System	Audio CLAP $\downarrow$	LanguageBind $\downarrow$	wMFCC $\downarrow$	SOT $\downarrow$	RMS $\uparrow$	MSS $\downarrow$
AutoSynth	0.37	0.397	10.22	0.15	0.82	6.94
Synth Permutations	0.54	0.584	22.75	0.17	0.80	9.99
DDSynth-RL	0.49	0.482	10.34	0.16	0.87	7.04
FSD50K
AutoSynth	0.67	0.649	14.95	0.16	0.632	13.00

Table 1: Synthesizer inversion on Gemini–Woosh and FSD50K. Bold denotes the best value for each metric within each test set.

Gemini
System	Text CLAP $\downarrow$	LanguageBind $\downarrow$	CE $\uparrow$	CU $\uparrow$	PQ $\uparrow$	FAD $\downarrow$	KL $\downarrow$
AutoSynth	0.550	0.638	3.759	7.23	7.433	—	—
Synth Permutations	0.642	0.718	3.409	6.88	7.372	—	—
DDSynth-RL	0.607	0.678	3.253	6.81	7.276	—	—
CTAG	0.519	0.645	3.020	6.58	6.888	—	—
Stable Audio 3 ∗	0.546	0.614	3.891	7.16	7.373	—	—

Table 2: Text-driven generation on Gemini and FSD50K descriptions, corresponding to the audio in Table 1 .

Figure 3: Subjective synthesizer-inversion evaluation on Gemini–Woosh (left) and FSD50K (right). Error bars indicate 95% bootstrap confidence intervals for the mean.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Audio encoder	Encoder architecture	Parameters	Frozen
AutoSynth encoder	Stereo mel spectrograms, convolutional frontend, 12-layer audio Transformer	28.02M	No
SAME-S	Waveform compression frontend, 6 backbone blocks of width 768	53.73M	Yes
SAME-L	Waveform compression frontend, 12 backbone blocks of width 1536	426.06M	Yes

Appendix

Table A1: Encoder sizes. Counts exclude the program decoder and the SAME waveform decoder.

Figure A1: Validation curves for three encoders across two training stages. Left: in-domain SFT loss, shown up to 500,000 updates. Right: Stable Audio 3 validation reward. Horizontal axes count optimizer updates within each stage.

Condition	PQ $\uparrow$	CLAP distance $\downarrow$
Uncorrupted Woosh	7.18	0.000
Uncorrupted input $\rightarrow$ AutoSynth	7.43	0.372
Noisy input	5.07	0.306
Noisy input $\rightarrow$ AutoSynth	7.40	0.451
Low-pass input	7.10	0.134
Low-pass input $\rightarrow$ AutoSynth	7.34	0.375

Appendix

Table B1: Corrupted-input inversion and resynthesis, averaged over all 128 clips. CLAP distance uses the uncorrupted Woosh clip as reference; lower is better. Higher PQ is better.

Test set	Audio CLAP $\downarrow$	LanguageBind $\downarrow$	wMFCC $\downarrow$	SOT $\downarrow$	RMS $\uparrow$	MSS $\downarrow$
Gemini–Woosh	0.47 $\rightarrow$ 0.37	0.535 $\rightarrow$ 0.397	17.05 $\rightarrow$ 10.22	0.33 $\rightarrow$ 0.15	0.65 $\rightarrow$ 0.82	19.41 $\rightarrow$ 6.94
FSD50K	0.73 $\rightarrow$ 0.67	0.746 $\rightarrow$ 0.649	22.05 $\rightarrow$ 14.95	0.31 $\rightarrow$ 0.16	0.472 $\rightarrow$ 0.632	24.33 $\rightarrow$ 13.00
Helm in-domain	0.17 $\rightarrow$ 0.13	0.221 $\rightarrow$ 0.174	8.02 $\rightarrow$ 6.93	0.057 $\rightarrow$ 0.048	0.91 $\rightarrow$ 0.92	7.36 $\rightarrow$ 3.77

Appendix

Table C1: SFT/GRPO comparison for synthesizer inversion.

Test set	Text CLAP $\downarrow$	LanguageBind $\downarrow$	CE $\uparrow$	CU $\uparrow$	PQ $\uparrow$	FAD $\downarrow$	KL $\downarrow$
Gemini	0.601 $\rightarrow$ 0.550	0.671 $\rightarrow$ 0.638	3.469 $\rightarrow$ 3.759	6.92 $\rightarrow$ 7.23	7.278 $\rightarrow$ 7.433	—	—
FSD50K	0.768 $\rightarrow$ 0.771	0.776 $\rightarrow$ 0.793	3.24 $\rightarrow$ 3.48	6.73 $\rightarrow$ 6.91	6.97 $\rightarrow$ 7.07	8.83 $\rightarrow$ 7.00	5.04 $\rightarrow$ 5.27

Appendix

Table C2: SFT/GRPO comparison for text-driven generation.

Figure D1: In-domain validation loss for different autoregressive orders over the first 100,000 steps.

Position	Group	Parameters in generation order
1–7	Global performance	beats_per_minute , polyphony , legato , portamento_type , portamento , pitch_bend_range , velocity_track
8–14	Oscillator 1	osc_1_waveform , osc_1_transpose , osc_1_tune , osc_1_unison_voices , osc_1_unison_detune , unison_1_harmonize , osc_1_volume
15–21	Oscillator 2	osc_2_waveform , osc_2_transpose , osc_2_tune , osc_2_unison_voices , osc_2_unison_detune , unison_2_harmonize , osc_2_volume
22–30	Cross-modulation and sources	cross_modulation , sub_waveform , sub_octave , sub_shuffle , sub_volume , noise_volume , osc_feedback_amount , osc_feedback_transpose , osc_feedback_tune
31–34	Amplitude envelope	amp_attack , amp_decay , amp_sustain , amp_release
35–47	Filter and envelope	filter_on , filter_style , filter_shelf , filter_blend , filter_drive , cutoff , resonance , keytrack , fil_env_depth , fil_attack , fil_decay , fil_sustain , fil_release

Appendix

Table D1: Autoregressive order of fixed parameters, read left to right within each row.

Item	Setting
Audio input	44.1 kHz, 5 seconds, stereo
Mel spectrograms	128 bands per channel, 25 ms Hamming window, 100 frames/s
Encoder / decoder layers	12 / 12
Hidden width / heads / FFN width	512 / 8 / 1024
Normalization and activation	Pre-LayerNorm, GELU, dropout 0.1
Value output layer	Shared Linear(512, 128), with field-specific legal categories

Appendix

Table E1: Main model and optimization settings.

Component	Direction	Coefficient
CLAP distance	Negative	8
Basic Pitch hidden-feature distance	Negative	12
AudioBox CE, normalized to $[0,1]$	Positive	1
AudioBox CU, normalized to $[0,1]$	Positive	1
wMFCC	Negative	0.07
SOT	Negative	3.58

Appendix

Table E2: Coefficients used in Equation ( 3 ). All coefficients are positive; each metric’s direction determines its sign in the reward.

Dimension	Question	Evaluation criteria
Alignment: synthesizer inversion	Does this sound faithfully reproduce the reference audio?	Consider timbre, pitch, articulation, and changes over time; 1 means not at all and 5 means very faithfully.
Alignment: text-driven generation	Does this sound faithfully follow the text description?	Consider the described timbre, pitch, articulation, and temporal changes; 1 means not at all and 5 means very faithfully.
Content quality: both tasks	How good is this sound as material for music production or sound design?	Consider audio quality, unintended artifacts, and creative usefulness; 1 means very poor and 5 means excellent. Intentional distortion is not a defect, and quality is rated independently of alignment.

Appendix

Table 15

Explore similar work

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning
Aug 4, 2026Tristan Wu, Daniel Chin, Junan Zhang +3Diffusion-Based Inverse Problems Diffusion Models
Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +2
SongCraft: Unified Song Generation and Editing with Reconstructive Learning
Sep 14, 2026Haohe Liu, Varun Nagaraja, Gael Le Lan +5Text-to-Music Generation Audio Editing
Meta AI
DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
Sep 14, 2026Zhongjie Duan, Shengchuan Gao, Hong Zhang +1Controllable Music Generation Audio Diffusion Models
Alibaba Group · Shanghai Jiao Tong University

Aug 4, 2026, cs.SDJ/K move · Enter open · S save

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning

Tristan Wu, Daniel Chin, Junan Zhang, Junyan Jiang+2

Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen+2

Diffusion-Based Inverse Problems Diffusion Models

Open paper AskPDF arXiv

Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to reflect rendered audio similarity, while the synthesizer being a non-differentiable black box prevents simple audio-domain supervision. To address the one-to-many mapping induced by the first challenge, we formulate synthesizer inversion as conditional generation over discrete synthesizer parameters and use masked discrete diffusion as the generator. This treatment additionally avoids the fixed-order assumption of autoregressive models and the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls. To address the second challenge, we further fine-tune the model with GRPO-style audio-domain rewards computed from rendered outputs. Experiments on Dexed show that, after supervised training, the discrete diffusion model is competitive with autoregressive and flow-matching baselines, and reward-based fine-tuning further improves out-of-domain audio matching performance. Code and demos are available at: https://github.com/DDSynth-RL/DDSynthRL.

Abstract

Explore similar work

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning

SongCraft: Unified Song Generation and Editing with Reconstructive Learning

DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation