We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.
Figures & tables
Figure 1: AudioCodex’s autoencoder main architecture. Blue denotes learnable components.
a) Autoencoder
Formulation
rFAD
gFAD
CM cSTFT
2.17
4.81
v -MF mSTFT∅CFG
0.83
3.63
x -MF mSTFT∅CFG
0.82
3.61
Bottleneck
tanh
0.82
5.14
Table 1: Architecture ablations for: a) the autoencoder, evaluated using r/gFAD; b) the phase estimator using rFAD.
Model
rFAD
rFAD C
gFAD
STFT
MEL
Codec Reference
DAC [ 9 ]
0.601
0.044
N/A
0.094
0.057
Baselines
M2L [ 2 ]
0.943
0.233
5.241
0.132
0.090
CDC [ 17 ]
0.304
0.088
5.065
0.114
0.079
SAO [ 10 ]
1.151
0.155
9.518
0.106
0.068
Table 2: Comparison to baselines. Metrics are defined in 5 . For all metrics, lower is better.
Component
Time (ms)
RTF
Single latent streaming
Encoding
6.80
12.56x
Combined decoding
41.38
2.06x
10s sample
Encoding
18.91
528.8x
Combined decoding
87.64
114.1x
Table 3: Inference latency and real-time factor (RTF) measured on a NVIDIA A100 GPU with 48 kHz stereo audio.
Neural autoencoders underpin generative models. Practical, large-scale use of neural autoencoders for generative modeling necessitates fast encoding, low latent rates, and a single model across representations. Existing approaches are reconstruction-first: they incur high latent rates, slow encoding, and separate architectures for discrete vs. continuous latents and for different audio channel formats, hindering workflows from preprocessing to inference conditioning. We introduce a generative-first architecture for audio autoencoding that increases temporal downsampling from 2048x to 3360x and supports continuous and discrete representations and common audio channel formats in one model. By balancing compression, quality, and speed, it delivers 10x faster encoding, 1.6x lower rates, and eliminates channel-format-specific variants while maintaining competitive reconstruction quality. This enables applications previously constrained by processing costs: a 60-second mono signal compresses to 788 tokens, making generative modeling more tractable.
Latent audio generative models are typically trained in two stages: an audio codec is learned first, followed by a latent generative model. This decomposition leads to a decoder train-generation mismatch: the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents at inference time. Across diverse datasets and latent generative models, we observe clear reconstruction-generation gaps under both FD and FAD, showing that strong reconstruction quality does not necessarily translate into strong end-to-end generation quality. A natural remedy is to adapt the decoder on generation-produced latents, but generated latents lack correspondence with source audio and therefore cannot directly provide the paired supervision used for decoder fine-tuning. We introduce AudioGAR, which constructs intermediate latents by perturbing encoder latents and denoising them through the frozen latent diffusion model. These latents form a trajectory from reconstruction toward generation, with lower-noise latents retaining source correspondence and supporting paired decoder fine-tuning. We fine-tune only the codec decoder on these latents, while keeping the codec encoder and latent generative model frozen. When applied to AudioX, AudioGAR substantially improves generative performance. It requires only 1.5% of the original training audio hours and 0.26% of the original training cost.
Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets
Dinghao Zhou, Xingchen Song, Di Wu +3
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China