Quantization has become essential for the efficient deployment of speech processing systems. Although widely studied, most existing quantization methods were developed for vision and NLP architectures, while the specific challenges of audio signals remain largely overlooked. In particular, we show that audio activations can exhibit large calibration ranges, leading to significant information loss when standard calibration techniques are applied. To address this, we propose ESC, an Evolution Strategy-based Calibration method that formulates activation scaling as an optimization problem and solves it using a two-step local-global scheme driven by an evolution strategy. ESC enables unaltered performance under full INT8 quantization and is the first calibration method to achieve near-lossless performance for full INT4 quantization across multiple speech tasks. Integrating ESC with PTQ methods further reduces performance loss, achieving a 1% relative accuracy degradation on the AST model.
Figures & tables
Figure 1: Illustration of quantization behavior across audio (Conformer [ 3 ] ), vision (ResNet [ 16 ] ), and NLP (BERT [ 17 ] ) models. Left : Cumulative distribution of normalized activation values, showing an approximately uniform distribution for ResNet, a rapidly saturating distribution for BERT, and a highly compressed distribution for Conformer. Right : Relative performance under weight and activation quantization using max calibration. While all models maintain good performance with 4-bit weight quantization, 4-bit activation quantization severely degrades performance for Conformer, unlike ResNet and BERT.
Figure 2: Overview of the proposed ESC method . First, each layer-wise activation scaling factor is locally optimized by minimizing the MSE between the FP32 and quantized layer outputs. Then, all scaling factors are jointly refined using the CMA-ES algorithm to minimizes the task-specific error between the quantized model output y^ and the target y .
Method
Bits
Conformer
ECAPA
MP-SENet
FastSpeech 2
AST
WER ↓
CER ↓
EER ↓
minDCF ↓
PESQ ↑
STOI ↑
Mel ↓
PostNet ↓
Acc ↑
mAP ↑
Full precision
32
15.94
4.84
0.97
7.17
2.12
93.67
40.18
40.09
98.13
99.98
Max [ 18 ]
8
16.54
5.04
1.11
7.99
2.16
93.31
40.88
40.72
98.12
99.98
Percentile [ 19 ]
8
16.07
4.85
0.99
7.06
2.11
92.88
40.55
40.58
98.06
99.95
Entropy [ 20 ]
8
16.27
4.93
18.70
78.33
2.11
93.55
161.48
161.59
90.09
99.65
MSE [ 21 ]
8
16.09
4.87
0.99
7.53
2.09
92.72
40.45
40.49
98.07
99.96
Table 1: Comparison of calibration methods across multiple models and speech tasks for fully 4-bit and 8-bit quantized models. The symbols ↓ and ↑ indicate that lower and higher values are better, respectively. Best results are shown in bold.
Method
Bits
Conformer
ECAPA
MP-SENet
FastSpeech 2
AST
WER ↓
CER ↓
EER ↓
minDCF ↓
PESQ ↑
STOI ↑
Mel ↓
PostNet ↓
Acc ↑
mAP ↑
Full precision
32
15.94
4.84
0.97
7.17
2.12
93.67
40.18
40.09
98.13
99.98
ESC
4
38.49
13.50
11.28
63.05
2.51
93.79
98.34
98.27
96.41
99.94
Adaround [ 6 ]
4
106.79
75.44
21.76
89.39
2.45
92.67
83.53
83.46
88.53
99.70
NoisyQuant [ 24 ]
4
39.89
13.96
11.28
63.05
2.50
93.61
95.80
95.73
96.26
99.94
DiTAS [ 26 ]
4
99.24
84.58
11.68
66.00
2.51
94.51
98.34
98.27
96.40
99.94
Table 2: Evaluation of PTQ techniques from the NLP and vision domains applied on top of ESC calibration.
Model
Latency (ms)
Speedup
Size (MB)
FP32
INT8
FP32
INT8
Conformer
7.27
5.42
1.34 ×
112.63
43.64
ECAPA
2.19
1.03
2.13 ×
63.58
59.69
MP-SENet
55.95
38.35
1.46 ×
23.86
9.15
FastSpeech 2
23.86
15.45
1.54 ×
398.70
216.12
AST
25.11
4.95
5.07 ×
331.69
113.45
Table 3: Comparison of inference latency and model size between FP32 and INT8 versions of several speech models.
Post-training quantization (PTQ) methods such as GPTQ and AWQ compress large language models effectively, but applying them to automatic speech recognition (ASR) at ultra-low bit-widths (2-3 bits) causes severe transcription degradation. Unlike discrete text tokens, speech activations are highly correlated in steady-state and zero-padded regions yet change rapidly at phonetic boundaries; when all frames contribute equally to the Hessian, static frames dominate calibration and obscure informative transitions. We propose DiffAQ, which computes frame-to-frame activation differences to measure the rate of acoustic change and assigns Hessian importance proportionally, concentrating quantization precision on phonetically critical frames. As a training-free modification to GPTQ, DiffAQ consistently reduces WER across various Whisper sizes and standard benchmarks, with the largest gains at 2-bit precision where baseline methods frequently produce degenerate outputs.
Post-training quantization (PTQ) reduces the cost of on-device text-to-speech (TTS), but published evaluations cover one system or method. We evaluate PTQ across TTS architectures under one protocol with three core models, weight and activation ablations of eight more, and two held-out models quantized blind. Four-bit per-channel weights reduce UTMOS, a predicted mean opinion score, by 2.8 on Supertonic and 0.07 on Kokoro, and per-tensor scaling can cause severe degradation even at 8 bits. The same bit width yields different outcomes, because the sensitive component is model-specific and not reliably predicted from the model class. A staged ablation procedure identifies it, and per-layer GPTQ can restore it to within 0.1 UTMOS. Real int8 and int4 kernels reproduce the simulated ordering at hardware-dependent cost. On a Mac mini, a 4-bit weight kernel runs Supertonic at 0.60x the fp32 latency while int8 is slower, so each configuration requires validation on the target runtime.
Although low-bit quantization provides practical means to deploy speaker verification on resource-constrained devices, its effects on speaker verification performance remain poorly understood. In this paper, we study uniform K-means quantization-aware training of ResNet-36 and ResNet-200 through joint layer-wise and score-level analyses. Our layer-wise analysis highlights fragile components and shows that score degradation is not fully explained by weight distortion alone. We identify a clear knee point at 2 bits, with larger score drift and harmful decision flips concentrated near the FP32 threshold. Our score-level analysis reveals where and how score errors emerge under extreme quantization. Building on these findings, we propose a calibrated multi-precision cascade that resolves most trials at 2 bits and escalates only ambiguous cases, achieving performance close to FP32 while preserving the efficiency benefits of low-bit inference with substantially lower compute and memory costs.
Hugo Leguillier, Driss Matrouf, Guillaume Lechien +1
Avignon University, LIA, UPR 4128, France · Aday, France