SAGE: Semantic Audio Generative Encoder

Francesco Brigante; Luca Cerovaz; Davide Marincione; Giorgio Strano; Luca Zhou; Emanuele Rodolà; Michele Mancusi

cs.SDSep 26, 2026

SAGE: Semantic Audio Generative Encoder

Authors: Francesco Brigante, Luca Cerovaz, Davide Marincione, Giorgio Strano, Luca Zhou, Emanuele Rodolà, Michele Mancusi

Organizations: Sapienza University of Rome, Italy · Paradigma · Moises Systems, Inc.

Abstract

Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This 105M-parameter model runs about 3x faster than Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder 8x larger and over 2x slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.

Figures & tables

Figure 1: Distributional fidelity against inference cost on the MoisesDB mixtures; marker area is the parameter count.

Figure 2: Probing on the nineteen tasks, one panel per block; higher is better on every task.

Figure 3: The SAGE encoder: (top) the tensor pipeline through the three SwinV2 stages to the latent bottleneck; (bottom) the internals of one stage.

Panel of Figure 4 — Figure 4: The two training phases of SAGE.

						MoisesDB, $10$ s		FMA, $30$ s
Model	Params	Data (h)	Domain	Latent	Comp.	ms $\downarrow$	vs SAGE	ms $\downarrow$	vs SAGE
SAGE (ours)	$105$ M	$10{,}460$	STFT	$64$	$\times 64$	$\underline{12.2}$	$\times 1.00$	$\underline{30.6}$	$\times 1.00$
SAME-L	$852$ M	$\approx 19{,}500$	waveform	$256$	$\times 32$	$25.9$	$\times 2.12$	$78.3$	$\times 2.56$
SAME-S	$108$ M	$\approx 19{,}500$	waveform	$256$	$\times 32$	$\mathbf{5.77}$	$\times 0.47$	$\mathbf{14.7}$	$\times 0.48$
Stable Audio Open	$156$ M	$\approx 7{,}300$	waveform	$64$	$\times 64$	$33.9$	$\times 2.78$	$100.1$	$\times 3.28$
CoDiCodec	$\approx 150$ M	$\approx 4{,}030$	STFT	$64$	$\times 128$	$148.2$	$\times 12.1$	$413.8$	$\times 13.5$

Table 1: Compiled TF32 time per clip on one RTX PRO 6000 Blackwell (Appendix C ); best in bold, second underlined. Data hours as reported; compression is waveform samples per latent scalar.

Figure 5: STFT power spectrograms of a selected music excerpt. Top: reference and reconstructions. Bottom: the same highlighted region enlarged for each model.

	Model	MERT	CLAP	PANN	music	audio	SDR $\uparrow$	dSTFT $\downarrow$
		FAD $\downarrow$			CLAP $\uparrow$
FMA test	SAGE (ours)	$\mathbf{0.113}$	$\mathbf{0.016}$	$\mathbf{1.422}$	$\mathbf{0.956}$	$\mathbf{0.960}$	$5.28$	$\mathbf{0.929}$
	SAME-L	$0.210$	$0.038$	$3.470$	$0.897$	$0.927$	$\mathbf{10.91}$	$1.746$
	Stable Audio Open	$0.319$	$0.044$	$2.891$	$0.892$	$0.911$	$5.64$	$1.342$
	CoDiCodec	$0.283$	$0.026$	$3.243$	$0.907$	$0.934$	$1.78$	$1.379$
	SAME-S	$0.649$	$0.082$	$7.169$	$0.839$	$0.870$	$8.80$	$1.675$

Table 2: Reconstruction on the five evaluation sets, best per column within each set in bold.

SAGE (ours) $105$ M / $0.0012$	SAME-L $852$ M / $0.0026$	Stable Audio Open $156$ M / $0.0034$	CoDiCodec $150$ M / $0.0148$
$\mathbf{81.6\pm 2.7}$	$\mathbf{81.8\pm 2.6}$	$64.6\pm 3.7$	$66.4\pm 3.5$

Table 3: Mean MUSHRA score

\pm

95% CI, with parameters / RTF (TF32, compiled, as in Table 1 ) under each model;

38

participants,

21

after filtering. The hidden reference scored

97.9\pm 0.7

and the

3.5

kHz low-pass anchor

15.4\pm 2.0

Block	SAGE (ours)	SAME-L	SAME-S	Stable Audio Open	CoDiCodec	Music2Latent
FMA (6)	$\mathbf{0.563}$	$0.470$	$0.468$	$0.490$	$0.474$	$0.484$
MoisesDB (7)	$\mathbf{0.544}$	$0.493$	$0.487$	$0.469$	$0.456$	$0.479$
Upstream MAEB (6)	$\mathbf{0.622}$	$0.473$	$0.472$	$0.481$	$0.471$	$0.487$

Table 4: Average score over the tasks of each block.

Gate and strength, with the WavTokenizer discriminator
	FAD $\downarrow$		CLAP $\uparrow$
Configuration	MERT	CLAP	music	audio	SI-SDR $\uparrow$	STFT $\downarrow$	Probes $\uparrow$
$s_{0}=25$ k, $\lambda_{\mathrm{sem}}=1$ (adopted)	$0.240$	$0.051$	$0.881$	$0.879$	$0.76$	$\mathbf{1.099}$	$0.637$
$s_{0}=50$ k, $\lambda_{\mathrm{sem}}=1$	$0.248$	$\mathbf{0.046}$	$\mathbf{0.889}$	$\mathbf{0.885}$	$0.35$	$1.117$	$0.629$
$s_{0}=100$ k, $\lambda_{\mathrm{sem}}=1$	$\mathbf{0.218}$	$0.051$	$0.878$	$0.881$	$\mathbf{1.46}$	$1.121$	$0.614$
$s_{0}=25$ k, $\lambda_{\mathrm{sem}}=0.5$	$0.250$	$0.052$	$0.885$	$0.882$	$1.01$	$1.102$	$0.631$

Table 5: Semantic branch on FMA-medium, probes averaged over the six FMA tasks. Gate steps are optimizer steps, one generator update every three of them.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Input representation
Sample rate, channels	$44.1$ kHz, stereo
STFT window / hop	$2048$ (Hann) / $512$
Frequency bins, frames	$F=1024$ (Nyquist dropped), $T=128$
Training segment	$65{,}024$ samples per channel ( $\approx 1.5$ s)
Power-law compression $\alpha$ , scale $\beta$	$0.65$ , $0.35$
Architecture

Appendix

Table 6: Full configuration of SAGE and of the two training phases.

Pre-training	Side level (dB)	$d_{\mathrm{width}}$ $\downarrow$	SI-SDR side $\uparrow$	SI-SDR mid $\uparrow$
without $\mathcal{L}_{\mathrm{SD}}$ , epoch $319$	$-12.40$	$0.092$	$-14.81$	$4.87$
without $\mathcal{L}_{\mathrm{SD}}$ , epoch $1499$	$-11.99$	$0.091$	$-14.32$	$4.87$
with $\mathcal{L}_{\mathrm{SD}}$ , epoch $499$	$-0.20$	$0.060$	$-5.09$	$4.84$

Appendix

Table 7: Stereo image on the MoisesDB mixtures, with and without

\mathcal{L}_{\mathrm{SD}}

in pretraining.

	MoisesDB $10$ s			FMA $30$ s
Model	Pkg. defaults	TF32	BF16	Pkg. defaults	TF32	BF16
SAME-L	$\times 3.75$ [3.65,3.85]	$\times 2.12$ [2.07,2.18]	$\times 1.67$ [1.63,1.72]	$\times 4.19$ [4.08,4.31]	$\times 2.56$ [2.49,2.63]	$\times 2.03$ [1.97,2.08]
SAME-S	$\times 0.60$ [0.58,0.61]	$\times 0.47$ [0.46,0.49]	$\times 0.40$ [0.39,0.42]	$\times 0.60$ [0.58,0.61]	$\times 0.48$ [0.47,0.49]	$\times 0.42$ [0.41,0.44]
Stable Audio Open	$\times 1.47$ [1.43,1.51]	$\times 2.78$ [2.70,2.85]	$\times 2.02$ [1.97,2.08]	$\times 1.73$ [1.68,1.78]	$\times 3.28$ [3.18,3.37]	$\times 3.16$ [3.07,3.25]

Appendix

Table 8: Time relative to SAGE at the same precision, each model at its faster of eager and compiled, with 95% intervals from clip sampling and the between-run spread. Below one: faster than SAGE.

MoisesDB mixtures, $10$ s
	Package defaults		TF32		BF16
Model	Eager	Comp.	Eager	Comp.	Eager	Comp.
SAGE (ours)	$34.2$	$23.3$	$20.3$	$12.2$	$23.6$	$12.5$
SAME-L	$123.4$	$87.4$	$33.4$	$25.9^{*}$	$27.1^{\dagger}$	$21.0^{\dagger}$
SAME-S	$21.1$	$13.9$	$8.54$	$5.77$	$9.89^{\dagger}$	$5.07^{*}$
Stable Audio Open	$75.1$	$34.2$	$74.9$	$33.9$	$67.6^{\dagger}$	$25.4^{\dagger}$

Appendix

Table 9: Inference time per clip in ms for every precision, median over

2

–

4

runs.

\dagger

: quality worse than package defaults in eager mode;

*

: loss between

0.1\%

and

1\%

FMA test split, in domain
Task	Metric	SAGE	SAME-L ∗	SAME-S ∗	Stable Audio Open	CoDiCodec	Music2- Latent	CLAP- oracle †
Genre classification	accuracy	$\mathbf{0.447}$	$0.285$	$0.279$	$0.287$	$0.305$	$0.307$	$\underline{0.529}$
Genre clustering	V-measure	$\mathbf{0.215}$	$0.146$	$0.145$	$0.171$	$0.168$	$0.185$	$\underline{0.385}$
Artist clustering	V-measure	$\mathbf{0.706}$	$0.634$	$0.634$	$0.655$	$0.617$	$0.632$	$\underline{0.719}$
Artist retrieval	nDCG@10	$\mathbf{0.583}$	$0.442$	$0.438$	$0.486$	$0.448$	$0.435$	$\underline{0.587}$
Genre reranking	MAP@1000	$\mathbf{0.535}$	$0.491$	$0.491$	$0.493$	$0.493$	$0.513$	$\underline{0.626}$

Appendix

Table 10: Semantic probing on the nineteen tasks, best per row among the six autoencoders in bold, block averages in the last row of each block. ∗ SAME-L and SAME-S are probed at their native

256

dimensions, four times the width of every other descriptor. † CLAP-oracle, underlined throughout, probes the frozen

512

-dimensional teacher of Section 2.2 under the same protocol; it cannot be inverted back to audio and is a reference, not a seventh autoencoder, so it is excluded from the bolding.

Type, present from the first step
	FAD $\downarrow$		CLAP $\uparrow$
Discriminator	MERT	CLAP	music	audio	SI-SDR $\uparrow$
WavTokenizer-RpGAN	$\mathbf{0.203}$	$\mathbf{0.056}$	$\mathbf{0.881}$	$\mathbf{0.879}$	$1.55$
EnCodec-RpGAN	$0.387$	$0.122$	$0.821$	$0.804$	$0.55$
EnCodec-hinge	$1.048$	$0.265$	$0.693$	$0.660$	$-2.21$
none	$1.453$	$0.414$	$0.645$	$0.624$	$\mathbf{2.47}$

Appendix

Table 11: Discriminator type and placement on FMA-medium; only the first two rows of the lower block distill a teacher.

Width, $\lambda_{\mathrm{mel}}=0.5$
		FAD $\downarrow$		CLAP $\uparrow$
Configuration	Params	MERT	CLAP	music	audio	SI-SDR $\uparrow$
$C=512$ , $(2,6,2)$	$388$ M	$1.050$	$\mathbf{0.259}$	$\mathbf{0.700}$	$\mathbf{0.720}$	$1.58$
$C=256$ , $(2,6,2)$	$97$ M	$\mathbf{0.963}$	$0.347$	$0.680$	$0.674$	$\mathbf{1.68}$
Depth, $\lambda_{\mathrm{mel}}=0.3$
$C=256$ , $(2,18,2)$	$180$ M	$\mathbf{1.298}$	$\mathbf{0.38}$	$\mathbf{0.667}$	$\mathbf{0.650}$	$\mathbf{2.94}$

Appendix

Table 12: Capacity on FMA-medium, without the adversarial branch; values compare inside a study, not across the two.

Autoencoders

Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.

See all 152 papers

SAME: A Semantically-Aligned Music Autoencoder
May 18, 2026Julian D. Parker, Zach Evans, CJ Carr +4Audio Representation Learning Neural Audio Codecs
Stability AI
A Generative-First Neural Audio Autoencoder
Feb 17, 2026Jonah Casebeer, Ge Zhu, Zhepei Wang +1Autoencoders Audio Representation Learning
Adobe Research
Stable Audio 3
May 18, 2026Zach Evans, Julian D. Parker, Matthew Rice +4Text-to-Audio Generation Latent Diffusion Models
Stability AI

May 18, 2026J/K move · Enter open · S save

SAME: A Semantically-Aligned Music Autoencoder

Julian D. Parker, Zach Evans, CJ Carr, Zachary Zukowski+3

Stability AI

Audio Representation Learning Neural Audio Codecs Autoencoders

Open paper AskPDF arXiv

Latent representations are at the heart of the majority of modern generative models. In the audio domain they are typically produced by a neural-audio-codec autoencoder. In this work we introduce SAME (Semantically-Aligned Music autoEncoder), an autoencoder for stereo music and general audio that reaches a 4096

\times

temporal compression ratio while maintaining reconstruction quality and downstream generative performance. We achieve this by combining a tranformer-based backbone with set of semantic regularisation approaches, phase-aware reconstruction losses and improved discriminator designs. The architecture delivers substantial computational cost benefits, through both its high compression ratio and its reliance on well-optimised transformer primitives. Two variants (a large SAME-L and a CPU-deployable SAME-S) are released in open-weights form.

Abstract

Autoencoders

Explore similar work

SAME: A Semantically-Aligned Music Autoencoder

A Generative-First Neural Audio Autoencoder

Stable Audio 3