Neural audio compression models have recently achieved extreme compression rates, enabling efficient latent generative modeling. Conversely, latent generative models have been applied to compression, pushing the limits of continuous and discrete approaches. However, existing methods remain constrained to low-resolution audio and degrade substantially at very low bitrates, where audible artifacts are prominent. In this paper, we present S-PRESSO, a 48kHz sound effect compression model that produces both continuous and discrete embeddings at ultra-low bitrates, down to 0.096 kbps, via offline quantization. Our model relies on a pretrained latent diffusion model to decode compressed audio embeddings learned by a latent encoder. Leveraging the generative priors of the diffusion decoder, we achieve extremely low frame rates, down to 1Hz (750x compression rate), producing convincing and realistic reconstructions at the cost of exact fidelity. Despite operating at high compression rates, we demonstrate that S-PRESSO outperforms both continuous and discrete baselines in audio quality, acoustic similarity and reconstruction metrics.
Figures & tables
Figure 1: Overview of our method. Step 1 : An audio clip is encoded into latent vectors x0 by a low-compression audio autoencoder. It is then compressed into latents z , which are upsampled by fϕ to condition the decoder Dθ , a DiT pretrained to reconstruct x0 from noised inputs. Dθ is finetuned using LoRA adapters, jointly trained with the latent encoder gψ and fϕ . Step 2 : The features z are then quantized offline into zq . Step 3 : the diffusion decoder Dθ is finetuned on zq , to compensate for quantization-induced degradation.
Figure 2: (a) Overview of the latent encoder architecture (b) Conditioning mechanism within the diffusion decoder.
Method
Variant
D
Framerate
R
FAD ↓
FAD CLAP↓
KAD CLAP↓
CLAP audio↑
Si-SDR ↑
AudioAE
–
128
100 Hz
4
0.008
0.008
0.15
0.90
22.3
StableAudio Open
–
64
21.5 Hz
32
0.78
0.066
1.25
0.78
0.48
S-PRESSO
t=4
64
25 Hz
30
0.48
0.038
0.57
0.76
3.21
Music2Latent
–
64
11 Hz
64
1.28
0.168
3.29
0.69
-10.5
S-PRESSO
t=9
64
11 Hz
68
0.59
0.050
0.77
0.76
-2.40
S-PRESSO
t=20
64
5 Hz
150
0.76
0.059
0.92
0.71
-8.80
Table 1: Performance in the continuous case of S-PRESSO vs. continuous audio compression baselines at equivalent compression rates R .
Method
kbps
Framerate
M
FAD ↓
FAD CLAP↓
KAD CLAP↓
CLAP audio↑
Si-SDR ↑
Low bitrates
DAC
1.7
86 Hz
2
3.24
0.108
1.71
0.63
-4.11
SemantiCodec
1.4
100 Hz
1
1.79
0.136
4.93
0.60
-31.8
S-PRESSO
1.32
11 Hz
12
0.55
0.048
0.728
0.73
-4.48
Ultra low bitrates
SemantiCodec
0.3125
25 Hz
1
1.23
0.271
2.70
0.48
-34.5
S-PRESSO
0.3
1 Hz
25
0.64
0.052
0.78
0.71
-27.8
S-PRESSO
0.096
1 Hz
8
0.68
0.060
0.89
0.67
-30.4
Table 2: Performance in the discrete case of S-PRESSO vs. baseline audio codecs at equivalent bitrates.
Figure 3: MUSHRA scores for S-PRESSO, SemantiCodec and a 3.5kHz low-pass anchor, evaluated at ∼ 1.35 kbps and ∼ 0.3 kbps.
Figure 4: Evaluation of S-PRESSO at varying bitrates and framerates.
Neural autoencoders underpin generative models. Practical, large-scale use of neural autoencoders for generative modeling necessitates fast encoding, low latent rates, and a single model across representations. Existing approaches are reconstruction-first: they incur high latent rates, slow encoding, and separate architectures for discrete vs. continuous latents and for different audio channel formats, hindering workflows from preprocessing to inference conditioning. We introduce a generative-first architecture for audio autoencoding that increases temporal downsampling from 2048x to 3360x and supports continuous and discrete representations and common audio channel formats in one model. By balancing compression, quality, and speed, it delivers 10x faster encoding, 1.6x lower rates, and eliminates channel-format-specific variants while maintaining competitive reconstruction quality. This enables applications previously constrained by processing costs: a 60-second mono signal compresses to 788 tokens, making generative modeling more tractable.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Luca Della Libera, Cem Subakan, Mirco Ravanelli
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs. We benchmark LM-based compression on full-fidelity audio across diverse domains (music, speech, bioacoustics), sampling rates (16kHz-48kHz), and bit depths (8, 16, 24-bit). Standard sample-level tokenization becomes intractable at higher bit depths due to vocabulary size (65K for 16-bit; 16.7M for 24-bit). We propose Trilobyte, a byte-level tokenization schema for full resolution audio, improving vocabulary scaling from O(2b) to O(1) and enabling the first tractable 24-bit LM-based lossless compression. While LMs consistently outperform FLAC and yield state-of-the-art compression at 8-bit and 16-bit, we observe that compression gains become more modest as bit depth increases beyond 8-bit.
Phillip Long, Zachary Novack, Chris Donahue
University of California, San Diego, Computer Science and Engineering Department · Carnegie Mellon University, School of Computer Science