Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering interoperability. To resolve this, we propose BridgeCodec, a versatile framework that decouples these pairs by translating latent representations between mismatched, frozen endpoints. We formulate this cross-codec translation as a Schrödinger Bridge optimal transport problem, enabling robust mapping between disparate distributions without predefined noise priors. To efficiently capture longrange speech temporal dependencies, we integrate a Mambaenhanced U-Net architecture. We validate BridgeCodec on an extreme deployment scenario: mapping a lightweight 8 kHz source encoder directly to a 48 kHz target decoder at an ultralow bitrate of 1 kbps. Experimental results demonstrate that BridgeCodec achieves superior wideband reconstruction and high perceptual quality, successfully bridging heavily bottlenecked, heterogeneous codecs.
Figures & tables
Figure 1: The BridgeCodec framework consists of Ens(⋅) , Mamba-enhanced backbone, and Det(⋅) . The source codec is marked with straight lines and the target codec with wavy lines, indicating they can use different model architectures and quantization methods.
Model
STOI ↑
STFT ↓
MEL ↓
DNSMOS ↑
WER ↓
SIM ↑
MOS ↑
target codec
0.96
1.63
1.05
3.26
1.70
0.99
4.24 ± 0.10
source codec
0.23
12.24
10.01
3.11
1.92
0.81
3.80 ± 0.11
BridgeCodec UNet
0.86
2.27
1.74
3.09
2.16
0.93
-
BridgeCodec
0.88
2.22
1.65
3.17
2.11
0.96
4.12 ± 0.11
w/o Two stage training
0.87
2.19
1.68
3.15
2.04
0.95
-
Table 1: Main results of BridgeCodec on the testset. All audio is resampled to 48 kHz for STOI, STFT and MEL. And due to evaluation model constraints, all audio is resampled to 16 kHz for DNSMOS, WER and SIM. MOS represents the subjective Mean Opinion Score for naturalness and speech quality with 95% confidence intervals.
NFE
STOI
STFT
MEL
DNSMOS
WER
SIM
10
0.88
2.22
1.65
3.17
2.11
0.96
5
0.88
2.16
1.63
3.17
2.37
0.95
1
0.89
2.13
1.58
3.18
2.39
0.95
Table 2: The results with different number of function evaluations.
Speech Bandwidth Extension improves clarity and intelligibility by restoring/inferring appropriate high-frequency content for low-bandwidth speech. Existing methods often rely on spectrogram or waveform modeling, which can incur higher computational cost and have limited high-frequency fidelity. Neural audio codecs offer compact latent representations that better preserve acoustic detail, yet accurately recovering high-resolution latent information remains challenging due to representation mismatch. We present CodecFlow, a neural codec-based BWE framework that performs efficient speech reconstruction in a compact latent space. CodecFlow employs a voicing-aware conditional flow converter on continuous codec embeddings and a structure-constrained residual vector quantizer to improve latent alignment stability. Optimized end-to-end, CodecFlow achieves strong spectral fidelity and enhanced perceptual quality on 8 kHz to 16 kHz and 44.1 kHz speech BWE tasks.
Bowen Zhang, Junchuan Zhao, Ian McLoughlin +2
Singapore Institute of Technology, Singapore · Nanyang Technological University, Singapore · National University of Singapore, Singapore
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Luca Della Libera, Cem Subakan, Mirco Ravanelli
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results after 300k training steps on a single H200 GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. On LibriSpeech test-clean, JHCodec achieves the best WER and CER among the evaluated codecs while maintaining zero lookahead and low end-to-end latency. We open-source the full implementation, training pipeline, and demo on GitHubh ttps://github.com/jhcodec843/jhcodec.
Junhyeok Lee, Xiluo He, Jihwan Lee +6
Center for Language and Speech Processing, Johns Hopkins University, USA · Signal Analysis and Interpretation Laboratory, University of Southern California, USA