Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (e.g., guidance) that can also be computationally demanding. By examining the bottlenecks of existing guidance-based controls, in particular their high cost-per-step due to decoder backpropagation, we introduce a guidance-based approach through selective TFG and Latent-Control Heads (LatCHs), which enables controlling latent audio diffusion models with low computational overhead. LatCHs operate directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and ≈ 4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. Our method balances precision and audio fidelity with far lower computational costs than standard end-to-end guidance. Demo examples can be found at https://zacharynovack.github.io/latch/latch.html.
Figures & tables
Figure 1: Left. End-to-end guidance can be slow and VRAM intensive as it requires backpropagating through the VAE decoder. Center. LatCH is compute efficient as it directly predicts control features from the latent space. Right. Selective TFG is also compute efficient as it allows applying TFG guidance only on selected sampling steps.
Quality metrics
Control alignment metrics ( ↓ )
Computational costs ( ↓ )
Qualitative results (MOS ↑ )
FD openl3
KL passt
CLAP
Intensity
Pitch
Beats
runtime on
VRAM
Audio
Prompt
Control
Control(s)
Method
( ↓ )
( ↓ )
( ↑ )
MSE
BCE
BCE
GPU (s)
usage (GB)
quality
adherence
alignment
–
SAO [ 2 ]
96.51
0.55
0.41
32.91
0.070
0.351
11.3
5.51
–
–
–
Beats
LatCH-B
89.43
0.55
36.77
–
–
0.138
17.6
5.59
4.5
4.6
4.0
Beats
LatCH-F
101.24
0.7
33.49
–
–
0.161
17.6
5.59
4.1
4.1
2.9
Beats
Readout
97.79
0.61
36.96
–
–
0.209
15.7
5.59
4.4
4.7
2.4
Table 1: Qualitative and quantitative results. ↓ the lower the better or ↑ the higher the better.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Stable Audio 3 is a family of fast latent diffusion models (small, medium, large) for variable-length audio generation and editing. Since our models can generate several minutes of audio, variable-length generations are key to avoid the cost of producing full-length generations for short sounds. We also support inpainting, enabling targeted audio editing and the continuation of short recordings. Our latent diffusion models operate on top of a novel semantic-acoustic autoencoder that projects audio into a compact latent space, enabling efficient diffusion-based generation while preserving audio fidelity and encouraging semantic structure in the latent. Finally, we run adversarial post-training to both accelerate inference and improve generation quality, reducing the number of inference steps while improving fidelity and prompt adherence. Stable Audio 3 models are trained on licensed and Creative Commons data to generate music and sounds in less than a 2s on an H200 GPU and less than a few seconds on a MacBook Pro M4. We release the weights of small and medium, that can run on consumer-grade hardware, together with their training and inference pipeline.
Recent work on controllable music generation has focused on autoregressive models, leaving diffusion-based systems comparatively underexplored. We present a lightweight method for steering the pitch content of audio produced by Stable Audio Open, a latent diffusion model for music synthesis. A small convolutional probe containing approximately 125k parameters is trained to decode frame-level pitch-class activations from the model's variational autoencoder latent space, using paired audio and MIDI data. At inference time, the frozen probe serves as a differentiable loss function: its gradient with respect to the denoising latent is used to nudge generation toward a user-specified pitch-class sequence, requiring no retraining or architectural modification of the base model. Across 27 evaluation trials spanning 9 text prompts and 3 target melodies, probe-guided generation increases melodic coherence by 2.4x over the unguided baseline (p < 1e-5, Wilcoxon signed-rank test), demonstrating that musically meaningful structure is both recoverable and steerable in diffusion-based music latent spaces.
Yushi Ye, Wilson Zheng, Yongyi Zang
Carnegie Mellon University · Independent Researcher
Latent audio generative models are typically trained in two stages: an audio codec is learned first, followed by a latent generative model. This decomposition leads to a decoder train-generation mismatch: the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents at inference time. Across diverse datasets and latent generative models, we observe clear reconstruction-generation gaps under both FD and FAD, showing that strong reconstruction quality does not necessarily translate into strong end-to-end generation quality. A natural remedy is to adapt the decoder on generation-produced latents, but generated latents lack correspondence with source audio and therefore cannot directly provide the paired supervision used for decoder fine-tuning. We introduce AudioGAR, which constructs intermediate latents by perturbing encoder latents and denoising them through the frozen latent diffusion model. These latents form a trajectory from reconstruction toward generation, with lower-noise latents retaining source correspondence and supporting paired decoder fine-tuning. We fine-tune only the codec decoder on these latents, while keeping the codec encoder and latent generative model frozen. When applied to AudioX, AudioGAR substantially improves generative performance. It requires only 1.5% of the original training audio hours and 0.26% of the original training cost.