High-Fidelity Binarized Neural Vocoder
Organizations: Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 10190, China · Tencent Hunyuan Multimodal Models Department, Tencent, Shenzhen 518054, China
Abstract
Neural vocoders have achieved remarkable progress in high-fidelity waveform generation in recent years, but most existing methods still rely on full-precision operators, leading to non-negligible storage and computational costs during inference. Binary neural networks (BNNs) provide an alternative direction by quantizing weights and activations to 1 bit. However, directly applying binarization to neural vocoders is rather challenging as it inherently incurs severe information loss. To this end, we propose BiVoC, a BNN-oriented time-frequency neural vocoder for high-fidelity waveform generation. Building on our previous full-precision RNDVoC framework, BiVoC adapts the inverse-problem formulation, range-null decomposition, and fineto-coarse non-uniform subband modeling to the 1-bit setting. Specifically, we develop a simple binarized convolutional unit to provide stable time-frequency feature modeling under binary representations. Furthermore, we analyze the quality-complexity sensitivity of different modules and propose a mixed-precision design to retain quality-sensitive components in full precision while binarizing computation-dominant parts. Extensive experiments are conducted on LibriTTS and multiple out-of-distribution benchmarks. Both objective and subjective results demonstrate that BiVoC substantially outperforms BNN variants obtained by directly binarizing existing vocoders, while achieving competitive performance against several mainstream full-precision vocoder baselines under much lower theoretical computational cost. These results demonstrate the promising potential of BNNs in the audio generation field. Codes and pre-trained weights are available at https://github.com/Andong-Li-speech/BiVoC.
Figures & tables
| Precision | Metric | HSEM | BCM | HSMM+HSPM |
|---|---|---|---|---|
| Full-precision | #Params (M) | 0.28(4.33%) | 2.43(37.96%) | 3.70(57.71%) |
| (Giga/5s) | 0.13(0.44%) | 28.70(93.78%) | 1.77(5.78%) | |
| Binary | #Params (M) | 0.01(1.49%) | 0.20(26.27%) | 0.56(72.24%) |
| (Giga/5s) | 0.006(0.54%) | 0.83(75.87%) | 0.26(23.59%) |
| Weight storage | Peak activation I/O buffer | ||||||
| Component | Main operations | FP32 | Packed | Red. | FP32 | Packed | Red. |
| (MB) | (MB) | (%) | (MB) | (MB) | (%) | ||
| HSEM | Norm. + subband enc. | 1.06 | 0.07 | 93.40 | 0.67 | 0.47 | 29.85 |
| BCM | (MS Conv + MLP) | 9.30 | 0.81 | 91.29 | 22.95 | 11.83 | 48.45 |
| HSMM+HSPM | Mag./phase dec. | 14.11 | 2.19 | 84.46 | 0.92 | 0.47 | 48.91 |
| Ids | Methods | Neural Type | Technique Type | Publish | Domain | Code Link |
|---|---|---|---|---|---|---|
| 1 | HiFiGAN-V1 [ 17 ] | Full-precision | GAN | NeurIPS 2020 | T | https://github.com/jik876/hifi-gan |
| 2 | iSTFTNet-V1 [ 52 ] | Full-precision | GAN | ICASSP 2022 | T | https://github.com/rishikksh20/iSTFTNet-pytorch |
| 3 | Avocodo [ 53 ] | Full-precision | GAN | AAAI 2023 | T | https://github.com/ncsoft/avocodo |
| 4 | BigVGAN-base [ 29 ] | Full-precision | GAN | ICLR 2023 | T | https://github.com/NVIDIA/BigVGAN |
| 5 | BigVGAN [ 29 ] | Full-precision | GAN | ICLR 2023 | T | https://github.com/NVIDIA/BigVGAN |
| 6 | APNet2 [ 54 ] | Full-precision | GAN | NCMMSC 2023 | T-F | https://github.com/redmist328/APNet2 |
| Type | M-STFT | PESQ | MCD | GOMPSNR | VISQOL |
|---|---|---|---|---|---|
| STE | 1.030 | 3.009 | 3.635 | 3.958 | 4.692 |
| HOA | 1.030 | 3.048 | 3.584 | 4.021 | 4.699 |
| AAE | 1.024 | 3.055 | 3.568 | 4.045 | 4.705 |
| Ids | w/i BNN | #Params | OPs | M-STFT | PESQ | MCD | GOMPSNR | VISQOL | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Head | Body | Tail | (in M) | (Giga/5s) | ||||||
| 1 | ✓ | ✓ | ✓ | 0.78 | 1.22 | 1.024 | 3.055 | 3.568 | 4.045 | 4.705 |
| 2 | ✗ | ✓ | ✓ | 1.08 | 1.34 | 0.975 | 3.349 | 3.342 | 4.540 | 4.759 |
| 3 | ✓ | ✗ | ✓ | 3.04 | 29.10 | 0.952 | 3.451 | 3.299 | 4.911 | 4.771 |
| 4 | ✓ | ✓ | ✗ | 3.93 | 2.73 | 0.986 | 3.193 | 3.489 | 4.199 | 4.735 |
| 5 | ✓ | ✗ | ✗ | 6.11 | 30.61 | 0.944 | 3.473 | 3.331 | 4.864 | 4.789 |
| w/i RND | M-STFT | PESQ | MCD | GOMPSNR | VISQOL |
|---|---|---|---|---|---|
| ✗ | 1.512 | 2.676 | 4.968 | 3.464 | 4.536 |
| ✓ | 0.975 | 3.349 | 3.342 | 4.540 | 4.759 |
| w/i Subband | M-STFT | PESQ | MCD | GOMPSNR | VISQOL |
|---|---|---|---|---|---|
| ✗ | 1.049 | 3.117 | 3.670 | 4.531 | 4.687 |
| ✓ | 0.975 | 3.349 | 3.342 | 4.540 | 4.759 |
| Unit Type | M-STFT | PESQ | MCD | GOMPSNR | VISQOL |
|---|---|---|---|---|---|
| ConvNextv2 [ 44 ] | 1.053 | 2.890 | 3.720 | 3.842 | 4.710 |
| LKCAB [ 30 ] | 1.005 | 3.169 | 3.494 | 4.185 | 4.743 |
| BCU w/o Norm | 0.982 | 3.287 | 3.433 | 4.382 | 4.763 |
| BCU (Ours) | 0.975 | 3.349 | 3.342 | 4.540 | 4.759 |
| Models | w/i | #Params. | OPs | Inference Speed | M-STFT | PESQ | MCD | V/UV | Period. | GOMPSNR | VISQOL | WER | |
| BNN | (M) | (Giga/5s) | CPU | GPU | F1 | RMSE | (in %) | ||||||
| GT | - | - | - | - | - | - | - | - | - | - | - | - | 6.21 |
| HiFiGAN-V1 | ✗ | 14.01 | 166.41 | 5.88 | 157 | 1.104 | 3.056 | 4.284 | 0.921 | 0.167 | 3.293 | 4.721 | 7.89 |
| iSTFTNet-V1 | ✗ | 13.33 | 117.30 | 10.66 | 299 | 1.157 | 2.880 | 4.480 | 0.918 | 0.167 | 3.159 | 4.655 | 7.54 |
| Avocodo | ✗ | 14.01 | 166.47 | 5.29 | 148 | 1.113 | 3.217 | 4.314 | 0.913 | 0.161 | 3.541 | 4.762 | 7.25 |
| BigVGAN-base | ✗ | 14.01 | 166.41 | 2.33 | 23 | 0.884 | 3.521 | 3.034 | 0.941 | 0.131 | 4.871 | 4.869 | 6.77 |
| Norm Type | #Params | OPs | PESQ | GOMPSNR | VISQOL |
|---|---|---|---|---|---|
| (in M) | (Giga/5s) | ||||
| LN | 1.08 | 1.34 | 3.349 | 4.540 | 4.759 |
| DyT | 1.08 | 1.12 | 3.347 | 4.512 | 4.759 |
| BN | 1.08 | 1.01 | 3.447 | 4.645 | 4.779 |
| Datasets | Metrics | Full-precision Vocoders | BNN-based Vocoders | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HiFiGAN-V1 | iSTFTNet-V1 | Avocodo | BigVGAN -base | BigVGAN | APNet2 | Vocos ‡ | PriorGrad | RNDVoC | DegVoC | HiFiGAN +BNN | Vocos +BNN | BiVoC (Ours) | ||
| AISHELL3 | M-STFT | 1.059 | 1.132 | 1.040 | 0.858 | 0.769 | 0.975 | 0.845 | 1.054 | 0.702 | 0.747 | 2.599 | 1.355 | 0.897 |
| PESQ | 2.823 | 2.595 | 3.003 | 3.414 | 3.878 | 2.846 | 3.342 | 3.995 | 4.049 | 4.075 | 1.167 | 1.742 | 3.330 | |
| MCD | 2.827 | 3.061 | 2.864 | 1.902 | 1.602 | 2.743 | 2.046 | 1.717 | 1.267 | 1.508 | 6.327 | 3.666 | 1.950 | |
| GOMPSNR | 3.351 | 3.242 | 3.721 | 4.860 | 5.746 | 4.182 | 4.549 | 6.067 | 6.436 | 6.560 | 1.039 | 2.423 | 4.928 | |
| VISQOL | 4.651 | 4.486 | 4.731 | 4.806 | 4.907 | 4.715 | 4.836 | 4.675 | 4.897 | 4.904 | 3.249 | 4.331 | 4.737 | |
| Models | GT | HiFiGAN-V1 | Vocos | APNet2 | HiFiGAN+BNN | Vocos+BNN | BiVoC (Ours) | Friedman -value |
|---|---|---|---|---|---|---|---|---|
| AISHELL3 | 4.32 0.08 | 3.93 0.09 | 4.07 0.08 | 3.69 0.09 | 1.49 0.05 | 2.64 0.08 | 4.00 0.08 | |
| MUSDB18 (Vocals) | 4.22 0.08 | 3.38 0.09 | 3.49 0.08 | 2.84 0.08 | 1.35 0.04 | 2.11 0.07 | 3.61 0.09 |
| Dataset | HiFiGAN | Vocos | HiFiGAN-V1 +BNN | Vocos +BNN |
|---|---|---|---|---|
| AISHELL3 | ( ) | ( ) | ( ) | ( ) |
| MUSDB18 | ( ) | ( ) | ( ) | ( ) |
| Ids | Subband | w/i Even | #Params. | OPs | M-STFT | PESQ | MCD | V/UV | Period. | GOMPSNR | VISQOL |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Number | (M) | (Giga/5s) | F1 | RMSE | |||||||
| 1 | 7 | ✗ | 0.84 | 0.60 | 1.066 | 2.916 | 3.751 | 0.922 | 0.159 | 4.033 | 4.665 |
| 2 | 13 | ✗ | 0.91 | 0.74 | 1.021 | 3.125 | 3.538 | 0.929 | 0.151 | 4.249 | 4.716 |
| 3 | 25 | ✗ | 1.08 | 1.01 | 0.962 | 3.447 | 3.222 | 0.938 | 0.134 | 4.645 | 4.779 |
| 4 | 25 | ✓ | 1.05 | 1.01 | 0.994 | 3.163 | 3.498 | 0.927 | 0.153 | 4.233 | 4.728 |
| 5 | 49 | ✗ | 1.32 | 1.57 | 0.937 | 3.528 | 3.127 | 0.939 | 0.130 | 4.853 | 4.806 |
| Split-Type | Total Number | Subband Width (kHz) | Subband Number |
|---|---|---|---|
| Uneven | 7 | 1 | 3 |
| 2 | 2 | ||
| 4 | 1 | ||
| 1 | 1 | ||
| 13 | 0.5 | 6 | |
| 1 | 4 |
| Models | OPs | M-STFT | PESQ | GOMPSNR |
|---|---|---|---|---|
| (Giga/5s) | ||||
| HiFiGAN-V1 | 0.99 | 1.867 | 1.829 | 1.904 |
| Vocos | 0.98 | 0.987 | 3.196 | 4.004 |
| DegVoC | 1.13 | 0.965 | 3.224 | 4.305 |
| BiVoC (Ours) | 1.01 | 0.962 | 3.447 | 4.645 |