Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes

Yushi Ye; Wilson Zheng; Yongyi Zang

cs.SDSep 3, 2026

Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes

Authors: Yushi Ye, Wilson Zheng, Yongyi Zang

Organizations: Carnegie Mellon University · Independent Researcher

Abstract

Recent work on controllable music generation has focused on autoregressive models, leaving diffusion-based systems comparatively underexplored. We present a lightweight method for steering the pitch content of audio produced by Stable Audio Open, a latent diffusion model for music synthesis. A small convolutional probe containing approximately 125k parameters is trained to decode frame-level pitch-class activations from the model's variational autoencoder latent space, using paired audio and MIDI data. At inference time, the frozen probe serves as a differentiable loss function: its gradient with respect to the denoising latent is used to nudge generation toward a user-specified pitch-class sequence, requiring no retraining or architectural modification of the base model. Across 27 evaluation trials spanning 9 text prompts and 3 target melodies, probe-guided generation increases melodic coherence by 2.4x over the unguided baseline (p < 1e-5, Wilcoxon signed-rank test), demonstrating that musically meaningful structure is both recoverable and steerable in diffusion-based music latent spaces.

Figures & tables

Panel of Figure 1 — Fig. 1 : Probe-guided latent steering at inference time. The base model is frozen (grey); the only learned component is the pitch-class probe $f_{\theta}$ (blue), a small $\sim$ 125k-parameter network trained offline on paired audio and MIDI and kept frozen at inference, which decodes a frame-level pitch-class map $\hat{\mathbf{y}}$ from the denoising latent $\mathbf{z}_{t}$ . The binary cross-entropy loss against the user-supplied target $\mathbf{y}^{*}$ is backpropagated to $\mathbf{z}_{t}$ and applied as an RMS-normalized, clamped update. Both pitch-class maps are drawn schematically.

Noise $\sigma$	F1 (micro)	Positive recall
0.0	0.663	0.596
0.1	0.662	0.595
0.3	0.657	0.588
0.5	0.642	0.569
0.7	0.621	0.542
1.0	0.579	0.493

Table 1: Probe decoding performance under additive Gaussian noise. The probe maintains usable accuracy across noise levels encountered during diffusion denoising.

Method	Mean $\pm$ Std	Shuffled	$p_{\mathrm{base}}$	$p_{\mathrm{shuf}}$
Baseline	0.112 $\pm$ 0.104	0.081	—	0.14
Probe ( $\lambda{=}$ 0.03)	0.236 $\pm$ 0.104	0.069	$4.6\!\times\!10^{-4}$	$1.9\!\times\!10^{-7}$
Probe ( $\lambda{=}$ 0.05)	0.274 $\pm$ 0.084	0.066	$5.7\!\times\!10^{-6}$	$7.5\!\times\!10^{-9}$
Probe ( $\lambda{=}$ 0.10)	0.266 $\pm$ 0.071	0.067	$8.0\!\times\!10^{-6}$	$7.5\!\times\!10^{-9}$
Probe ( $\lambda{=}$ 0.20)	0.273 $\pm$ 0.064	0.066	$2.3\!\times\!10^{-6}$	$7.5\!\times\!10^{-9}$

Table 2: Melodic coherence across probe guidance scales, with the shuffled-target control.

p_{\mathrm{base}}

tests each condition against the unguided baseline;

p_{\mathrm{shuf}}

tests it against its own shuffled control (one-sided Wilcoxon signed-rank test,

n{=}27

paired trials).

Fig. 2 : Mean melodic coherence with standard deviation error bars across guidance scales. The baseline (gray) uses no probe intervention. All probe-guided conditions (blue) substantially improve pitch-class alignment.

Melody	Baseline	Probe ( $\lambda{=}$ 0.05)
Ascending	0.095 $\pm$ 0.081	0.268 $\pm$ 0.074
Descending	0.144 $\pm$ 0.108	0.234 $\pm$ 0.064
Alternating	0.096 $\pm$ 0.112	0.320 $\pm$ 0.090

Table 3: Per-melody coherence breakdown at

\lambda{=}0.05

(

n{=}9

prompts per melody type). All melody types show substantial improvement.

Controllable Music Generation

Controllable music generation steers composition and audio synthesis through prompts, musical structure, references, and interactive controls.

See all 88 papers

Low-Resource Guidance for Controllable Latent Audio Diffusion
Mar 4, 2026Zachary Novack, Zack Zukowski, CJ Carr +6Audio Generation Diffusion Model Guidance
UC – San Diego · Stability AI
TADA! Tuning Audio Diffusion Models through Activation Steering
Feb 12, 2026Łukasz Staniszewski, Katarzyna Zaleska, Mateusz Modrzejewski +1Controllable Music Generation Audio Diffusion Models
Warsaw University of Technology · IDEAS Research Institute
DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
Sep 11, 2026Zhongjie Duan, Shengchuan Gao, Hong Zhang +1Controllable Music Generation Audio Diffusion Models
Alibaba Group · Shanghai Jiao Tong University

Mar 4, 2026J/K move · Enter open · S save

Low-Resource Guidance for Controllable Latent Audio Diffusion

Zachary Novack, Zack Zukowski, CJ Carr, Julian Parker+5

UC – San Diego · Stability AI

Audio Generation Diffusion Model Guidance Audio Diffusion Models Latent Diffusion Models

Open paper AskPDF arXiv

Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (e.g., guidance) that can also be computationally demanding. By examining the bottlenecks of existing guidance-based controls, in particular their high cost-per-step due to decoder backpropagation, we introduce a guidance-based approach through selective TFG and Latent-Control Heads (LatCHs), which enables controlling latent audio diffusion models with low computational overhead. LatCHs operate directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and

\approx

4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. Our method balances precision and audio fidelity with far lower computational costs than standard end-to-end guidance. Demo examples can be found at https://zacharynovack.github.io/latch/latch.html.

Abstract

Controllable Music Generation

Explore similar work

Low-Resource Guidance for Controllable Latent Audio Diffusion

TADA! Tuning Audio Diffusion Models through Activation Steering

DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation