Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This 105M-parameter model runs about 3x faster than Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder 8x larger and over 2x slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.
Figures & tables
Figure 1: Distributional fidelity against inference cost on the MoisesDB mixtures; marker area is the parameter count.
Figure 2: Probing on the nineteen tasks, one panel per block; higher is better on every task.
Figure 3: The SAGE encoder: (top) the tensor pipeline through the three SwinV2 stages to the latent bottleneck; (bottom) the internals of one stage.
Figure 4: The two training phases of SAGE.
MoisesDB, 10 s
FMA, 30 s
Model
Params
Data (h)
Domain
Latent
Comp.
ms ↓
vs SAGE
ms ↓
vs SAGE
SAGE (ours)
105 M
10,460
STFT
64
×64
12.2
×1.00
30.6
×1.00
SAME-L
852 M
≈19,500
waveform
256
×32
25.9
×2.12
78.3
×2.56
SAME-S
108 M
≈19,500
waveform
256
×32
5.77
×0.47
14.7
×0.48
Stable Audio Open
156 M
≈7,300
waveform
64
×64
33.9
×2.78
100.1
×3.28
CoDiCodec
≈150 M
≈4,030
STFT
64
×128
148.2
×12.1
413.8
×13.5
Table 1: Compiled TF32 time per clip on one RTX PRO 6000 Blackwell (Appendix C ); best in bold, second underlined. Data hours as reported; compression is waveform samples per latent scalar.
Figure 5: STFT power spectrograms of a selected music excerpt. Top: reference and reconstructions. Bottom: the same highlighted region enlarged for each model.
FAD ↓
CLAP ↑
Model
MERT
CLAP
PANN
music
audio
SDR ↑
dSTFT ↓
FMA test
SAGE (ours)
0.113
0.016
1.422
0.956
0.960
5.28
0.929
SAME-L
0.210
0.038
3.470
0.897
0.927
10.91
1.746
Stable Audio Open
0.319
0.044
2.891
0.892
0.911
5.64
1.342
CoDiCodec
0.283
0.026
3.243
0.907
0.934
1.78
1.379
SAME-S
0.649
0.082
7.169
0.839
0.870
8.80
1.675
Table 2: Reconstruction on the five evaluation sets, best per column within each set in bold.
SAGE (ours) 105 M / 0.0012
SAME-L 852 M / 0.0026
Stable Audio Open 156 M / 0.0034
CoDiCodec 150 M / 0.0148
81.6±2.7
81.8±2.6
64.6±3.7
66.4±3.5
Table 3: Mean MUSHRA score ± 95% CI, with parameters / RTF (TF32, compiled, as in Table 1 ) under each model; 38 participants, 21 after filtering. The hidden reference scored 97.9±0.7 and the 3.5 kHz low-pass anchor 15.4±2.0 .
Block
SAGE (ours)
SAME-L
SAME-S
Stable Audio Open
CoDiCodec
Music2Latent
FMA (6)
0.563
0.470
0.468
0.490
0.474
0.484
MoisesDB (7)
0.544
0.493
0.487
0.469
0.456
0.479
Upstream MAEB (6)
0.622
0.473
0.472
0.481
0.471
0.487
Table 4: Average score over the tasks of each block.
FAD ↓
CLAP ↑
Configuration
MERT
CLAP
music
audio
SI-SDR ↑
STFT ↓
Probes ↑
Gate and strength, with the WavTokenizer discriminator
s0=25 k, λsem=1 (adopted)
0.240
0.051
0.881
0.879
0.76
1.099
0.637
s0=50 k, λsem=1
0.248
0.046
0.889
0.885
0.35
1.117
0.629
s0=100 k, λsem=1
0.218
0.051
0.878
0.881
1.46
1.121
0.614
s0=25 k, λsem=0.5
0.250
0.052
0.885
0.882
1.01
1.102
0.631
Table 5: Semantic branch on FMA-medium, probes averaged over the six FMA tasks. Gate steps are optimizer steps, one generator update every three of them.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Input representation
Sample rate, channels
44.1 kHz, stereo
STFT window / hop
2048 (Hann) / 512
Frequency bins, frames
F=1024 (Nyquist dropped), T=128
Training segment
65,024 samples per channel ( ≈1.5 s)
Power-law compression α , scale β
0.65 , 0.35
Architecture
Appendix
Table 6: Full configuration of SAGE and of the two training phases.
Pre-training
Side level (dB)
dwidth↓
SI-SDR side ↑
SI-SDR mid ↑
without LSD , epoch 319
−12.40
0.092
−14.81
4.87
without LSD , epoch 1499
−11.99
0.091
−14.32
4.87
with LSD , epoch 499
−0.20
0.060
−5.09
4.84
Appendix
Table 7: Stereo image on the MoisesDB mixtures, with and without LSD in pretraining.
MoisesDB 10 s
FMA 30 s
Model
Pkg. defaults
TF32
BF16
Pkg. defaults
TF32
BF16
SAME-L
×3.75 [3.65,3.85]
×2.12 [2.07,2.18]
×1.67 [1.63,1.72]
×4.19 [4.08,4.31]
×2.56 [2.49,2.63]
×2.03 [1.97,2.08]
SAME-S
×0.60 [0.58,0.61]
×0.47 [0.46,0.49]
×0.40 [0.39,0.42]
×0.60 [0.58,0.61]
×0.48 [0.47,0.49]
×0.42 [0.41,0.44]
Stable Audio Open
×1.47 [1.43,1.51]
×2.78 [2.70,2.85]
×2.02 [1.97,2.08]
×1.73 [1.68,1.78]
×3.28 [3.18,3.37]
×3.16 [3.07,3.25]
Appendix
Table 8: Time relative to SAGE at the same precision, each model at its faster of eager and compiled, with 95% intervals from clip sampling and the between-run spread. Below one: faster than SAGE.
Package defaults
TF32
BF16
Model
Eager
Comp.
Eager
Comp.
Eager
Comp.
MoisesDB mixtures, 10 s
SAGE (ours)
34.2
23.3
20.3
12.2
23.6
12.5
SAME-L
123.4
87.4
33.4
25.9∗
27.1†
21.0†
SAME-S
21.1
13.9
8.54
5.77
9.89†
5.07∗
Stable Audio Open
75.1
34.2
74.9
33.9
67.6†
25.4†
Appendix
Table 9: Inference time per clip in ms for every precision, median over 2 – 4 runs. † : quality worse than package defaults in eager mode; ∗ : loss between 0.1% and 1% .
Task
Metric
SAGE
SAME-L ∗
SAME-S ∗
Stable Audio Open
CoDiCodec
Music2- Latent
CLAP- oracle †
FMA test split, in domain
Genre classification
accuracy
0.447
0.285
0.279
0.287
0.305
0.307
0.529
Genre clustering
V-measure
0.215
0.146
0.145
0.171
0.168
0.185
0.385
Artist clustering
V-measure
0.706
0.634
0.634
0.655
0.617
0.632
0.719
Artist retrieval
nDCG@10
0.583
0.442
0.438
0.486
0.448
0.435
0.587
Genre reranking
MAP@1000
0.535
0.491
0.491
0.493
0.493
0.513
0.626
Appendix
Table 10: Semantic probing on the nineteen tasks, best per row among the six autoencoders in bold, block averages in the last row of each block. ∗ SAME-L and SAME-S are probed at their native 256 dimensions, four times the width of every other descriptor. † CLAP-oracle, underlined throughout, probes the frozen 512 -dimensional teacher of Section 2.2 under the same protocol; it cannot be inverted back to audio and is a reference, not a seventh autoencoder, so it is excluded from the bolding.
FAD ↓
CLAP ↑
Discriminator
MERT
CLAP
music
audio
SI-SDR ↑
Type, present from the first step
WavTokenizer-RpGAN
0.203
0.056
0.881
0.879
1.55
EnCodec-RpGAN
0.387
0.122
0.821
0.804
0.55
EnCodec-hinge
1.048
0.265
0.693
0.660
−2.21
none
1.453
0.414
0.645
0.624
2.47
Appendix
Table 11: Discriminator type and placement on FMA-medium; only the first two rows of the lower block distill a teacher.
FAD ↓
CLAP ↑
Configuration
Params
MERT
CLAP
music
audio
SI-SDR ↑
Width, λmel=0.5
C=512 , (2,6,2)
388 M
1.050
0.259
0.700
0.720
1.58
C=256 , (2,6,2)
97 M
0.963
0.347
0.680
0.674
1.68
Depth, λmel=0.3
C=256 , (2,18,2)
180 M
1.298
0.38
0.667
0.650
2.94
Appendix
Table 12: Capacity on FMA-medium, without the adversarial branch; values compare inside a study, not across the two.
Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.
Latent representations are at the heart of the majority of modern generative models. In the audio domain they are typically produced by a neural-audio-codec autoencoder. In this work we introduce SAME (Semantically-Aligned Music autoEncoder), an autoencoder for stereo music and general audio that reaches a 4096× temporal compression ratio while maintaining reconstruction quality and downstream generative performance. We achieve this by combining a tranformer-based backbone with set of semantic regularisation approaches, phase-aware reconstruction losses and improved discriminator designs. The architecture delivers substantial computational cost benefits, through both its high compression ratio and its reliance on well-optimised transformer primitives. Two variants (a large SAME-L and a CPU-deployable SAME-S) are released in open-weights form.
Neural autoencoders underpin generative models. Practical, large-scale use of neural autoencoders for generative modeling necessitates fast encoding, low latent rates, and a single model across representations. Existing approaches are reconstruction-first: they incur high latent rates, slow encoding, and separate architectures for discrete vs. continuous latents and for different audio channel formats, hindering workflows from preprocessing to inference conditioning. We introduce a generative-first architecture for audio autoencoding that increases temporal downsampling from 2048x to 3360x and supports continuous and discrete representations and common audio channel formats in one model. By balancing compression, quality, and speed, it delivers 10x faster encoding, 1.6x lower rates, and eliminates channel-format-specific variants while maintaining competitive reconstruction quality. This enables applications previously constrained by processing costs: a 60-second mono signal compresses to 788 tokens, making generative modeling more tractable.
Stable Audio 3 is a family of fast latent diffusion models (small, medium, large) for variable-length audio generation and editing. Since our models can generate several minutes of audio, variable-length generations are key to avoid the cost of producing full-length generations for short sounds. We also support inpainting, enabling targeted audio editing and the continuation of short recordings. Our latent diffusion models operate on top of a novel semantic-acoustic autoencoder that projects audio into a compact latent space, enabling efficient diffusion-based generation while preserving audio fidelity and encouraging semantic structure in the latent. Finally, we run adversarial post-training to both accelerate inference and improve generation quality, reducing the number of inference steps while improving fidelity and prompt adherence. Stable Audio 3 models are trained on licensed and Creative Commons data to generate music and sounds in less than a 2s on an H200 GPU and less than a few seconds on a MacBook Pro M4. We release the weights of small and medium, that can run on consumer-grade hardware, together with their training and inference pipeline.