HE-NAC: High-Efficiency Neural Audio Coding via Hierarchical and Residual Multi-Band Quantization
Authors: Darius Pétermann, Wootaek Lim, Seungkwon Beack, Minje Kim
Organizations: Indiana University, Bloomington, IN, USA · Electronics and Telecommunications Research Institute, Daejeon, South Korea · University of Illinois at Urbana-Champaign, Urbana, IL, USA
High-fidelity audio coding requires faithful reconstruction of perceptually important high-frequency components, which are often lost due to the aggressive temporal decimation and quantization of neural codecs. State-of-the-art neural audio coding methods rely on adversarial learning to hallucinate these missing components, yielding perceptually plausible but imprecise reconstructions; a limitation for high-fidelity music applications. As a remedy, we propose high-efficiency neural audio coding (HE-NAC), which provides a multi-band extension of the popular residual vector quantization (RVQ) scheme. HE-NAC introduces an effective training mechanism that harmonizes a bitrate-free blind upsampling process, residual learning to compensate it with minimal bitrate, and a multi-head decoder architecture for band-specific reconstruction. As a result, HE-NAC achieves a balanced trade-off, reliably reconstructing the core-band (≤ 3 kHz) component while preserving overall sound fidelity. Subjective listening tests and objective evaluations demonstrate that HE-NAC outperforms both the DAC baseline and MPEG HE-AAC at equal or lower bitrates.
Figures & tables
Fig. 1: Overview of the proposed HE-NAC system.
Fig. 2: Overall reconstruction flow of the proposed three-stage training paradigm. Gray segments denote components used only during training: they provide blind-upsampled, bitrate-free priors to the next stage, but are omitted (filtered out) at inference.
Model
Band
Stride
CB
FR (Hz)
Ent. kbps
Raw kbps
DAC (31)
Full
400
31
80
23.0
24.8
HE-NAC (16,4,2)
Core
400
16
80
12.0
12.8
Mid
100
4
320
6.0
12.8
High
64
2
500
5.0
10.0
Total
–
–
–
23.0
35.6
TABLE I: DAC vs HE-NAC at the ∼ 23 kbps operating point. Raw rate is pre-entropy (10 bits/token, K=1024 ).
Models
#Params
Mel- distance ↓
STFT -distance ↓
VISQOL ↑
SI-SDR ↑
DAC (31)
85M
0.42
1.68
4.33
15.6
DAC (31)
127M
0.41
1.67
4.36
15.7
DAC (31)
220M
0.41
1.67
4.35
15.7
HE-NAC (16,4,2)
220M
0.52
1.76
4.40
13.7
TABLE II: Objective results across various DAC model with sizes matching our proposed model. Metrics are computed on full-band input/output
Fig. 3: Mean MUSHRA scores for each condition with 95% confidence intervals for the mean. Nominal bitrates (kbps) are indicated under the coded systems.
kbps
Models
Mel distance ↓
STFT distance ↓
VISQOL ↑
SI-SDR ↑
Core-Band (0kHz - 3kHz)
18
DAC (24)
0.49
0.32
–
18.0
HE-NAC (16,4,0)
0.62
0.41
–
15.4
23
DAC (31)
0.37
0.24
–
20.6
HE-NAC (16,4,2)
0.62
0.41
–
15.4
Mid-Band (3kHz - 6kHz)
TABLE III: Objective results across individual subbands and full bands. Lower is better for Mel/STFT distance, higher is better for VISQOL/SI-SDR.
Fig. 4: Rate–distortion (STFT error) curves shown per frequency band (0-3, 3-6, 6-16 kHz, and full-band) over the 13-23 kbps range. DAC (baseline) scales its bitrate by controlling the number of codebooks, ranging from 18 to 31, defined by its single bottleneck. HE-NAC (proposed) keeps 16 codebooks in its CB, but then gradually increases the MB codebooks from 1 to 4, as well as the HB codebooks from 0 to 2. Each DAC datapoint is annotated with its total number of codebooks, J , whereas each HE-NAC datapoint is annotated with the complete branch-specific allocation (JCB,JMB,JHB) . The third entry, JHB , therefore denotes the number of codebooks allocated to the HB branch. When JHB=0 , no dedicated HB residual is transmitted and the 6-16 kHz band is reconstructed solely through blind upsampling from the CB and MB stages. Increasing JHB to 1 and 2 progressively enables dedicated HB residual coding, explaining the sharp reduction in HB distortion.
Fig. 5: Comparisons of spectrograms (3kHz-16kHz) across several audio examples and models. Our model showcase the best reconstruction in that band considering the additional coding artifacts added in HE-AAC.
kbps
13
15
16
18
20
23
HE-NAC CBs (JCB,JMB,JHB)
(16,1,0)
(16,2,0)
(16,3,0)
(16,4,0)
(16,4,1)
(16,4,2)
DAC CBs (J)
18
20
22
24
27
31
Band
Model
SI-SDR (dB)
0-3 kHz
DAC
15.8
16.6
17.3
18.0
19.0
20.6
HE-NAC
15.4
15.4
15.4
15.4
15.4
15.4
TABLE IV: SI-SDR (dB) and GOMPSNR across bands (3, 6, 16 kHz, and full-band) over 13–23 kbps for DAC and HE-NAC. VISQOL scores are included for the full-band evaluation. For HE-NAC, the tuples (JCB,JMB,JHB) denote the number of RVQ codebooks allocated to the core-, mid-, and high-band branches, respectively. Since DAC employs a single RVQ bottleneck, only the total number of codebooks is applicable.
Fig. 6: Comparison in the time-frequency domain between various HE-NAC configurations.
Fig. 7: The distribution of per-band feature vectors. The latent features per bottleneck are aggregated and concatenated over the audio files, and histogram are drawn from them. Their respective Shannon entropy is also shown for each of the distributions.
Model
Active #Params
RTF ↓
GFLOPs / audio s ↓
DAC (31)
85M
0.0066
149.9
DAC (31)
115M
0.0096
337.2
HE-NAC (16,4,0)
127M
0.0095
308.1
DAC (31)
220M
0.0112
426.7
HE-NAC (16,4,2)
220M
0.0165
491.7
TABLE V: Inference complexity of parameter-matched DAC and HE-NAC variants. RTF is the ratio of inference time to audio duration; values below one indicate faster-than-real-time processing. GFLOPs are normalized per second of 32-kHz mono audio using 1 MAC =2 FLOPs. Measurements were obtained with batch size one over 39 ten-second clips (390 seconds total), excluding file I/O and metric computation. All models were evaluated offline and do not inherently support streaming inference.
Georgia Institute of Technology, Atlanta, United States · The Chinese University of Hong Kong, Hong Kong, China · Tencent Music Entertainment, Shenzhen, China