Not All Frames Are Equal:Difference-Aware Quantization for Ultra-Low-Bit ASR

Woori Jeon; Jungmin So

cs.SDOct 14, 2026

Not All Frames Are Equal:Difference-Aware Quantization for Ultra-Low-Bit ASR

Authors: Woori Jeon, Jungmin So

Organizations: Sogang University, Seoul, Korea

Abstract

Post-training quantization (PTQ) methods such as GPTQ and AWQ compress large language models effectively, but applying them to automatic speech recognition (ASR) at ultra-low bit-widths (2-3 bits) causes severe transcription degradation. Unlike discrete text tokens, speech activations are highly correlated in steady-state and zero-padded regions yet change rapidly at phonetic boundaries; when all frames contribute equally to the Hessian, static frames dominate calibration and obscure informative transitions. We propose DiffAQ, which computes frame-to-frame activation differences to measure the rate of acoustic change and assigns Hessian importance proportionally, concentrating quantization precision on phonetically critical frames. As a training-free modification to GPTQ, DiffAQ consistently reduces WER across various Whisper sizes and standard benchmarks, with the largest gains at 2-bit precision where baseline methods frequently produce degenerate outputs.

Figures & tables

Figure 1: Waveform (top), Mel-spectrogram (middle), and DiffAQ importance weight $w_{t}$ (bottom) for a LibriSpeech utterance. $w_{t}$ drops toward the floor $\alpha{=}0.2$ during silence and rises above the mean ( $w_{t}{=}1.0$ ) at phonetic transitions, giving acoustically dynamic frames greater importance during quantization calibration.

Figure 2: Overview of the DiffAQ encoder quantization pipeline. Input speech is converted to a log-Mel spectrogram and processed by Conv1D layers. DiffAQ then computes frame-to-frame activation differences, derives the importance weight $w_{t}$ , and applies it to scale the Hessian before quantization.

Method	Precision	Whisper Medium		Whisper Small		Whisper Base
Method	Precision	clean	other	clean	other	clean	other
FP16 Baseline	16-bit	2.73	5.69	3.42	7.54	5.17	12.33
RTN	2-bit	40.38 $\pm$ 0.13	66.41 $\pm$ 0.18	99.32 $\pm$ 0.03	$>100$	$>100$	$>100$
AWQ	2-bit	40.39 $\pm$ 0.13	66.27 $\pm$ 0.18	99.61 $\pm$ 0.02	99.60 $\pm$ 0.02	$>100$	$>100$
Standard GPTQ	2-bit	13.73 $\pm$ 1.67	17.53 $\pm$ 1.36	36.64 $\pm$ 2.10	51.30 $\pm$ 2.24	$>100$	$>100$
DiffAQ (Ours)	2-bit	7.93 $\pm$ 1.70	12.93 $\pm$ 1.34	28.02 $\pm$ 2.35	39.26 $\pm$ 2.70	99.50 $\pm$ 0.02	$>100$

Table 1: Word Error Rate (WER %) on LibriSpeech across different quantization methods for Whisper Medium, Small, and Base. Results are reported as mean (

\pm

95% confidence interval) across calibration seeds. (

>100\%

WER indicates degenerate output).

Method	Precision	Whisper Medium	Whisper Small	Whisper Base
FP16 Baseline	16-bit	4.58	6.33	9.85
RTN	2-bit	76.65 $\pm$ 0.49	$>100$	$>100$
AWQ	2-bit	75.80 $\pm$ 0.39	99.80 $\pm$ 0.00	$>100$
Standard GPTQ	2-bit	18.05 $\pm$ 3.76	66.55 $\pm$ 4.99	$>100$
DiffAQ (Ours)	2-bit	12.07 $\pm$ 0.86	49.10 $\pm$ 4.29	$>100$
RTN	3-bit	5.01 $\pm$ 0.33	9.23 $\pm$ 0.01	31.51 $\pm$ 0.17

Table 2: Word Error Rate (WER %) on FLEURS across different quantization methods for Whisper Medium, Small, and Base. Results are reported as mean (

\pm

95% confidence interval) across calibration seeds. (

>100\%

WER indicates degenerate output).

Low-Bit Quantization

Low-bit quantization reduces neural-network precision to save memory and computation while seeking to preserve task performance.

See all 222 papers

TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers
Aug 11, 2026Xiaoqin Feng, Yuqi Li, Weilun Feng +9Diffusion Model Quantization Post-Training Quantization
Wyze Inc., USA · The City College of New York, City University of New York, USA · Institute of Computing Technology, Chinese Academy of Sciences, China +2
TARQ: Tail-Aware Reconstruction Quantization for Rare-Word Robust Automatic Speech Recognition
May 27, 2026Xinyu Wang, Ziyu Zhao, Ke Bai +4Automatic Speech Recognition Post-Training Quantization
McGill University · Boson AI · Arizona State University
Same Bit Width, Different Outcomes: Post-Training Quantization of Text-to-Speech Across Architectures
Sep 24, 2026Se Un Park, Yutae Kim, Junyoung ParkTTS Evaluation Weight-Only Quantization
UX Factory, Inc.

Aug 11, 2026J/K move · Enter open · S save

TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers

Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin+8

Wyze Inc., USA · The City College of New York, City University of New York, USA · Institute of Computing Technology, Chinese Academy of Sciences, China+2

Diffusion Model Quantization Post-Training Quantization Mixed-Precision Quantization Zero-Shot TTS Diffusion Transformer

Open paper AskPDF arXiv

Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.

Abstract

Low-Bit Quantization

Explore similar work

TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers

TARQ: Tail-Aware Reconstruction Quantization for Rare-Word Robust Automatic Speech Recognition

Same Bit Width, Different Outcomes: Post-Training Quantization of Text-to-Speech Across Architectures