CodecFlow: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Space
Authors: Bowen Zhang, Junchuan Zhao, Ian McLoughlin, Ye Wang, A S Madhukumar
Organizations: Singapore Institute of Technology, Singapore · Nanyang Technological University, Singapore · National University of Singapore, Singapore
Speech Bandwidth Extension improves clarity and intelligibility by restoring/inferring appropriate high-frequency content for low-bandwidth speech. Existing methods often rely on spectrogram or waveform modeling, which can incur higher computational cost and have limited high-frequency fidelity. Neural audio codecs offer compact latent representations that better preserve acoustic detail, yet accurately recovering high-resolution latent information remains challenging due to representation mismatch. We present CodecFlow, a neural codec-based BWE framework that performs efficient speech reconstruction in a compact latent space. CodecFlow employs a voicing-aware conditional flow converter on continuous codec embeddings and a structure-constrained residual vector quantizer to improve latent alignment stability. Optimized end-to-end, CodecFlow achieves strong spectral fidelity and enhanced perceptual quality on 8 kHz to 16 kHz and 44.1 kHz speech BWE tasks.
Figures & tables
Figure 1: An overview of the proposed CodecFlow framework, including (a) the overall model pipeline, (b) the architecture of the voicing extractor, and (c) the architecture of the flow prediction network from the flow embedding converter (FEC).
Figure 2: V/UV segmentation and LR–HR embedding cosine similarity over time. Upper: spectrogram with word boundaries and V/UV labels. Lower: cosine similarity; orange dashed regions mark UV-aligned drops, blue dashed line shows the global mean.
Models
8 kHz → 16kHz
8 kHz → 44.1 kHz
LSD ↓
LSD-LF ↓
LSD-HF ↓
VISQOL ↑
MOS ↑
COL ↑
LSD ↓
LSD-LF ↓
LSD-HF ↓
VISQOL ↑
MOS ↑
COL ↑
Input (LR Speech)
5.36
1.16
7.49
2.83
3.77
3.20
5.50
0.81
6.07
2.16
3.85
3.28
Target (HR Speech)
-
-
-
4.51
4.29
4.06
-
-
-
4.73
4.50
4.32
NU-Wave2 [ 13 ]
1.75
0.51
2.42
2.60
3.19
2.84
1.50
0.72
1.62
2.19
3.49
2.98
AP-BWE [ 11 ]
1.24
0.46
1.69
2.42
3.97
3.79
1.14
0.23
1.25
3.23
4.44
4.25
Fre-Painter [ 12 ]
1.35
0.34
1.87
2.64
3.76
3.69
1.30
0.62
1.40
3.31
4.01
3.99
Table 1: Bandwidth extension evaluation at 16 kHz and 44.1 kHz targets with multiple baseline models. Best results are highlighted in bold , and second-best results are underlined .
Figure 3: Spectrogram comparison between the 8 kHz input, the 44.1 kHz ground truth (GT), and model outputs, including NU-Wave2, AP-BWE, FlowHigh, and the proposed CodecFlow.
Figure 4: Spectrogram comparison between the 8 kHz input, the 16 kHz ground truth (GT), and model outputs, including NU-Wave2, AP-BWE, FlowHigh, and the proposed CodecFlow.
Figure 5: Gender-wise comparison of objective and perceptual metrics under two bandwidth-extension settings (8 kHz → 16 kHz and 8 kHz → 44.1 kHz).
Method
LSD ↓
LSD-LF ↓
LSD-HF ↓
VISQOL ↑
MOS ↑
COL ↑
Task: 8 kHz → 16 kHz
CodecReg (w/o FT)
1.19
0.67
1.53
2.46
3.77
3.63
CFM-Conf (w/o FT)
1.33
0.72
1.73
2.24
3.82
3.70
CFM-UConf (w/o FT)
1.21
0.82
1.48
2.29
4.05
3.90
CodecFlow
1.01
0.64
1.27
2.72
4.25
4.04
Task: 8 kHz → 44.1 kHz
Table 2: Ablation study of converter variants in CodecFlow, where CodecReg denotes direct latent residual regression, CFM-Conf and CFM-UConf denote conditional flow–based converters with Conformer and U-Conformer backbones, “w/o FT” indicates training without fine-tuning, and CodecFlow denotes the full fine-tuned model.
Figure 6: Comparison of continuous codec latent embeddings across LR input, CodecReg, CodecFlow and HR ( 16 kHz ) ground truth.
Figure 7: Comparison of continuous codec latent embeddings across LR input, CodecReg, CodecFlow, and HR ( 44.1 kHz ) ground truth.
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
Bandwidth extension, the task of reconstructing the high-frequency components of an audio signal from its low-passed counterpart, is a long-standing problem in audio processing. In this work, we extend recent advances in neural architectures by framing bandwidth extension as an audio token prediction problem. Specifically, we train a transformer-based language model on the discrete representations produced by a disentangled neural audio codec, where the disentanglement is guided by a Harmonic-Percussive decomposition of the input signals, highlighting spectral structures particularly relevant for bandwidth extension. Our approach introduces a novel codec design that explicitly accounts for the downstream token prediction task, enabling a more effective coupling between codec structure and transformer modeling. This joint design yields high-quality reconstructions of the original signal, as measured by both objective metrics and subjective evaluations. These results highlight the importance of aligning codec disentanglement and representation learning with the generative modeling stage, and demonstrate the potential of global, representation-aware design for advancing bandwidth extension.
Benoît Ginies, Olivier Fercoq, Gaël Richard
LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France
Bandwidth extension, the task of reconstructing the high-frequency components of an audio signal from its low-passed counterpart, is a long-standing problem in audio processing. In this work, we extend recent advances in neural architectures by framing bandwidth extension as an audio token prediction problem. Specifically, we train a transformer-based language model on the discrete representations produced by a disentangled neural audio codec, where the disentanglement is guided by a Harmonic-Percussive decomposition of the input signals, highlighting spectral structures particularly relevant for bandwidth extension. Our approach introduces a novel codec design that explicitly accounts for the downstream token prediction task, enabling a more effective coupling between codec structure and transformer modeling. This joint design yields high-quality reconstructions of the original signal, as measured by both objective metrics and subjective evaluations. These results highlight the importance of aligning codec disentanglement and representation learning with the generative modeling stage, and demonstrate the potential of global, representation-aware design for advancing bandwidth extension.
Benoît Giniès, Xiaoyu Bie, Olivier Fercoq +1
LTCI, Télécom Paris, Institut Polytechnique de Paris Palaiseau, France
Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering interoperability. To resolve this, we propose BridgeCodec, a versatile framework that decouples these pairs by translating latent representations between mismatched, frozen endpoints. We formulate this cross-codec translation as a Schrödinger Bridge optimal transport problem, enabling robust mapping between disparate distributions without predefined noise priors. To efficiently capture longrange speech temporal dependencies, we integrate a Mambaenhanced U-Net architecture. We validate BridgeCodec on an extreme deployment scenario: mapping a lightweight 8 kHz source encoder directly to a 48 kHz target decoder at an ultralow bitrate of 1 kbps. Experimental results demonstrate that BridgeCodec achieves superior wideband reconstruction and high perceptual quality, successfully bridging heavily bottlenecked, heterogeneous codecs.
Zijian Lin, Jing Yang, Jinghao Luo +3
Shenzhen International Graduate School, Tsinghua University, China · Central Media Technology Institute, Huawei