Speech compression is increasingly important in scenarios such as real-time and satellite communication. Neural codecs have shown superior Rate-Distortion (RD) performance over traditional signal-processing codecs. Models with long-range dependency encoders/decoders and entropy models outperform convolution-based structures, but they still lack streaming support since they rely on sequence-level correlations. In this paper, we propose StreamLSC, a frame-level streaming learned speech compression model built on three fundamental components of streaming networks, i.e. Boundary-Cached Causal Convolution, Intra-Frame Layer Normalization, and RNN-mode RWKV. Our model incorporates a streamable entropy quantizer module, ensuring efficient context model while preserving causal constraints. Experiments show that the streaming model achieves performance comparable to its offline counterpart, attaining state-of-the-art rate-distortion results, significant gains in subjective evaluations, -51.37% (ViSQOL) and -53.99% (PESQ) BD-RATE savings on average, and improvements of 0.2154 and 0.5156 in BD-ViSQOL and BD-PESQ.
Figures & tables
Fig. 1: Subjective evaluation results using MUSHRA. The center point of each circle represents the average value, and the size reflects variability of listener ratings. Our model achieves superior perceptual quality at ultra-low bitrates.
Fig. 2: Illustration of model architectures. (a) Previous non-streamable method. The whole audio sequence is processed at a time. (b) StreamLSC, frame-level streaming processing mode. (c) Streamable model components: FS-CRM block with explicit state propagation in the BCC and RWKV layers. Red blocks denote causal convolutions or convolution-based modules, while green blocks denote inherently streamable components.
Fig. 3: Objective evaluation results: ViSQOL and PESQ scores of the proposed model compared with baselines. The proposed StreamLSC achieves frame-level streamability while maintaining RD performance consistent with the non-streaming version, significantly outperforming the baseline solution with a performance gain of 0.2154 BD-ViSQOL and 0.5156 BD-PESQ.
Frame length
80 ms
160 ms
320 ms
640 ms
ViSQOL
4.8257
4.8283
4.8315
4.8319
TABLE I: Effect of frame length on quality (ViSQOL).
Training Strategy
Bitrate (bps)
ViSQOL
PESQ
Full GAN Training
3529.00
4.4500
3.1800
Two-stage GAN Training
3377.29
4.5398
3.1780
TABLE II: Comparison of training strategies on test_clean and test_others datasets.
Model
RTF
StreamLSC (Ours)
5.39×10−2
FunCodec
5.45×10−3
EnCodec
4.695×10−3
DAC
6.367×10−3
BigCodec
1.504×10−2
TABLE III: Model parameters and real-time factor (RTF) on RTX 4090.
Learning-based speech compression has achieved promising low-bitrate performance, but many neural speech codecs still describe quantized latents with preset-rate discrete symbols or apply entropy coding only after symbol generation. Such designs decouple representation learning from probability modeling, limiting their ability to exploit the non-uniform usage and temporal dependencies of learned speech latents. In this paper, we benchmark neural speech compression from a rate--distortion perspective and further investigate entropy-constrained coding for low-bitrate speech compression. We first formulate a unified learning-based speech coding pipeline and provide a benchmark-style analysis of recent neural speech codecs, showing that explicit probability modeling remains underexplored in learned speech compression. We then propose ECC, an Entropy-Constrained Codec that combines scalar quantization with a learned entropy model. ECC integrates hyperprior-based side information, channel-wise context modeling, latent residual prediction, and lightweight temporal modeling to estimate latent likelihoods for rate estimation during training and arithmetic coding during inference. To further improve low-bitrate efficiency, ECC introduces entropy skip, which omits highly predictable residual symbols using decoder-available scale estimates without transmitting additional skip masks. Extensive experiments show that ECC achieves a favorable low-bitrate rate--distortion trade-off over conventional and neural codec baselines, reducing BD-rate by 39.9% on ViSQOL and 76.3% on PESQ on average over two widely-used test sets. Ablation and diagnostic studies further validate the effectiveness of entropy modeling. Project Page: https://avery-xu.github.io/ECC-demo/
Jun Xu, Zhengxue Cheng, Fengxi Zhang +3
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Luca Della Libera, Cem Subakan, Mirco Ravanelli
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
Neural speech codecs increasingly serve as tokenizers for speech language models (SLMs). Lowering the frame rate reduces the computational and memory costs of SLMs, but makes it difficult to preserve both linguistic information and acoustic detail. Existing approaches rely on rule-based compression: average pooling can discard linguistic information, whereas similarity-based merging uses a fixed threshold on adjacent-frame similarity and applies the resulting boundaries to the acoustic stream. We propose Q-SPT, a low-frame-rate dual-stream speech tokenizer with separate, context-aware, learnable query-based compressors specialized for semantic and acoustic representations. In particular, queries at a fixed rate independently attend to the semantic and acoustic streams as separate key-value sources, enabling stream-specific, context-aware aggregation through two separately learned compressors. In addition, an autoregressive text loss explicitly supervises the semantic compressor to preserve linguistic information. Experimental results show that Q-SPT achieves the best reconstruction among the evaluated codecs at the same frame rate. In downstream SLMs, it yields the best speech recognition accuracy and text-to-speech perceptual quality with competitive intelligibility.