Neural audio codecs (NACs) typically encode the short-term energy (gain) and normalized structure (shape) of speech/audio signals jointly within the same latent space. As a result, they are poorly robust to a global variation of the input signal level in the sense that such a variation has a strong influence on the embedding vectors at the output of the encoder and their quantization. This methodology is inherently inefficient, leading to codebook redundancy and suboptimal bitrate-distortion performance. To address these limitations, we propose to introduce shape-gain decomposition, widely used in classical speech and audio coding, into the NAC framework. The principle of the proposed Equalizer methodology, easily applicable to any NAC, is to decompose the input signal---before the NAC encoder---into gain and normalized shape vector on a short-term basis. The shape vector is processed by the NAC, while the gain is quantized with scalar quantization and transmitted separately. The output (decoded) signal is reconstructed from the normalized output of the NAC and the quantized gain. Our experiments conducted on speech signals with four different prominent codecs show that this general methodology enables a substantial gain in bitrate-distortion performance, as well as a massive reduction in quantizer complexity.
Figures & tables
Fig. 1 : Effect of global input gain variation on BigCodec, SpeechTokenizer and DAC latent representations. Top: Average embedding norm (normalized to 0 dB). Middle: Average cosine similarity with the 0 dB reference embedding. Bottom: Discrete code stability after quantization of the embedding vectors. The vertical dashed line marks 0 dB, the reference gain.
Fig. 2 : Architecture of the proposed Equalizer NAC pipeline. Block 1: The input signal s is decomposed into a temporal gain envelope and an equalized waveform sˉ using short-term analysis, normalization, and OLA synthesis. Block 2: The resulting successive shape vectors are processed by a NAC applying vector quantization on the corresponding embedding vectors. Block 3: In parallel, the gain is quantized with scalar quantization (typically μ -law). Block 4: The decoded equalized output waveform and the quantized gain are used to generate the final output waveform using again short-term analysis-synthesis with OLA.
Codec
Emb. vec.
Num. of RVQ
Codebook
Bitrate
rate ( f in Hz)
stages ( Nq )
size ( C )
( r in kbps)
EnCodec
50
8
128–1024
2.8–4.0
EnCodec*
50
2, 4, 6
1024
1.0–3.0
DAC
50
6, 8, 10, 12
1024
3.0–6.0
BigCodec
80
1
1024–8192
0.8–1.04
WavTokenizer
40
1
4096
0.48
TABLE I : Configuration of the NACs as used in our experiments.
Fig. 3 : Performance of EnCodec-Eq (a) and BigCodec-Eq (b) (solid lines) and the corresponding baseline codecs (dashed lines) as a function of the input gain α , for four different codebook sizes C . Scores are averaged over approximately 5.4 h of test signals from the Librispeech-test dataset. For both codecs, the Equalizer version remains nearly invariant to changes in α , whereas the baseline models degrade as α deviates from 0 dB. Furthermore, the Equalizer versions consistently outperform their corresponding baselines even at 0 dB.
−12 dB
−4 dB
0 dB
+4 dB
+12 dB
Codec
Equalizer
PESQ ↑
STOI ↑
MCD ↓
PESQ ↑
STOI ↑
MCD ↓
PESQ ↑
STOI ↑
MCD ↓
PESQ ↑
STOI ↑
MCD ↓
PESQ ↑
STOI ↑
MCD ↓
WavTok. (0.48 kbps)
✗
1.19
0.782
7.24
1.28
0.812
6.93
1.31
0.818
6.85
1.31
0.819
6.78
1.26
0.808
6.62
✓
1.58
0.860
5.80
1.58
0.860
5.85
1.58
0.860
5.87
1.58
0.860
5.86
1.58
0.860
5.59
BigCodec (1.04 kbps)
✗
2.08
0.904
4.55
2.22
0.913
4.24
2.20
0.912
4.20
2.15
0.909
4.23
1.35
0.776
5.31
✓
2.46
0.930
3.69
2.46
0.930
3.71
2.46
0.930
3.71
2.46
0.930
3.71
2.46
0.930
3.70
EnCodec* (3 kbps)
✗
2.17
0.898
4.78
2.31
0.910
4.54
2.33
0.911
4.48
2.32
0.908
4.56
1.86
0.878
4.87
TABLE II : PESQ, STOI, and MCD scores obtained in our experiments for the four codecs, with and without combination with The Equalizer and for five selected values of the global input gain. All codecs were retrained from scratch on the LibriSpeech-100 dataset. The Equalizer version (✓) demonstrates near-perfect robustness to input gain variations as opposed to the corresponding baseline (✗).
Fig. 4 : Performance of the proposed Equalizer methodology and of the baselines, as a function of the (total) bitrate, and for the four considered codecs (two versions for EnCodec). Each score is averaged over ≈5.4 h of test signals from the Librispeech-test dataset and over an input gain range of ±12 dB. The Equalizer (solid lines) consistently outperforms the corresponding baseline (dashed lines).
Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization is applied only post-training to the fitted parameters. Rather than relying on discrete codebook stages for bitrate control, our representation exposes a fine-grained rate-quality tradeoff: a single trained checkpoint supports post-training bitrate control by varying the number of primitives and the per-parameter bit depth, with no retraining required. GS-Codec matches or exceeds well-established open-source codecs such as EnCodec and DAC on speaker similarity (SIM), intelligibility (STOI), and perceptual quality (UTMOS) at comparable bitrates, while achieving comparable semantic performance (WER). Code and audio samples are available at https://ronaluf.github.io/gs-codec/
Ron Aluf, Alon Canfi, Eliya Nachmani
School of Electrical and Computer Engineering Ben-Gurion University of the Negev
Neural audio codecs (NACs) convert speech into discrete token sequences, and prior work has reported that these sequences follow language-like statistical laws. This paper analyzes the token statistics of 13 NACs spanning multi-codebook residual vector quantization (RVQ), single-codebook VQ, and non-VQ designs, evaluated on three corpora under clean, white-noise, and real-world DEMAND-noise conditions. Zipf and Heaps parameters, unigram entropy, codebook occupancy, and Jensen-Shannon divergence (JSD) are estimated from matched token samples with explicit fit-validity safeguards and family-conditional n-gram orders. Corpus identity explains little variance in any metric, whereas acoustic condition and quantizer meta-category dominate in a metric-dependent way, and unigram entropy is the metric most strongly associated with meta-category. Clean-to-noise JSD computed at a common unigram order is associated with mel-cepstral distortion most clearly under DEMAND noise. The collapse and explosion degradation signatures previously reported for RVQ codecs concentrate in RVQ cells under white and DEMAND noise, respectively; explosion also occurs in non-VQ codecs, and single-codebook VQ codecs shift in occupancy and distribution shape without either signature. These results provide architecture-conditioned conventions for applying language-statistical analysis to NAC tokens.
Joonyong Park, Shinnosuke Takamichi, David M. Chan +3
Grad. School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Faculty of Science and Technology, Keio University, Tokyo, Japan · Berkeley Artificial Intelligence Research Lab (BAIR), University of California, Berkeley, CA, USA
Neural Audio Codecs (NACs) are widely adopted in modern speech systems, yet how they encode linguistic and paralinguistic information remains unclear. Improving the interpretability of NAC representations is critical for understanding and deploying them in sensitive applications. Hence, we employ Sparse Autoencoders (SAEs) to decompose dense NAC representations into sparse, interpretable activations. In this work, we focus on a challenging paralinguistic attribute-accent-and propose a framework to quantify NAC interpretability. We evaluate four NAC models under 16 SAE configurations using a relative performance index. Our results show that DAC and SpeechTokenizer achieve the highest interpretability. We further reveal that acoustic-oriented NACs encode accent information primarily in activation magnitudes of sparse representations, whereas phonetic-oriented NACs rely more on activation positions, and that low-bitrate EnCodec variants show higher interpretability.