TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers
Authors: Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen
Organizations: Wyze Inc., USA · The City College of New York, City University of New York, USA · Institute of Computing Technology, Chinese Academy of Sciences, China · National Taiwan University, Taiwan · Tsinghua University, China
Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.
Figures & tables
Figure 1: Overview of TTS-Q. Coverage-guided calibration sampling selects conditions in a reference-speaker embedding space. After initial local reconstruction, speaker identity distillation compares teacher and student speech under matched text, voice prompt, and initial noise. Local reconstruction remains active on cached teacher states. The dashed purple path carries identity gradients through the frozen decoder and speaker encoder. The lower inset shows a detached prefix (student for LongCat, teacher for F5) and differentiable sampling tail. Only quantization parameters are updated. Waveforms and embedding geometry are schematic.
Figure 2: One selection step in a schematic Euclidean space. Blue points form the current set S ; orange marks the next candidate j⋆ selected from P∖S . Solid links show nearest assignments after the addition; the dashed segment shows its distance to S before the addition. The shared starting set is illustrative; initialization follows the rules in the text.
Seed English
Seed Mandarin
Seed Mandarin hard
Model
Method
WER ↓
SIM ↑
UTMOS ↑
CER ↓
SIM ↑
UTMOS ↑
CER ↓
SIM ↑
UTMOS ↑
LongCat-1B
Full Prec
2.403
0.7623
3.789
1.263
0.8118
3.161
7.058
0.7891
2.997
RTN
125.333
0.0420
1.287
100.243
0.0145
1.310
100.692
-0.0063
1.286
Q-DiT † ( Chen et al., 2025a )
3.753
0.7235
3.566
1.438
0.7865
2.869
8.701
0.7182
2.607
QuaRot † ( Ashkboos et al., 2024 )
2.917
0.7168
3.488
1.353
0.7835
2.764
7.192
0.7615
2.536
GPTQ † ( Frantar et al., 2023 )
3.769
0.7235
3.543
1.601
0.7881
2.885
9.432
0.7227
2.636
Table 1: Speech quality on Seed-TTS under W4A4 quantization of DiT linear layers. WER/CER are percentages. Full Prec uses each model’s native precision; bold marks the best quantized value before rounding. Calibration-based methods share the selected conditions. † denotes a task adaptation (Appendix B ). Reconstruction controls are reported in Tables 2 and 3 .
EN
ZH
Model
Selection
WER ↓
CER ↓
SIM ↑
UTMOS ↑
1B
Random §
2.592
1.271
0.8002
2.909
Farthest-point ∗
2.344
1.186
0.8006
2.953
Average-distance
2.598
1.262
0.7996
2.931
3.5B
Random §
2.089
1.152
0.8088
2.995
Farthest-point
2.140
1.214
0.8097
2.988
Table 2: Calibration-set selection under the same reconstruction procedure and budget. \lx@sectionsign Mean of three random calibration sets. ∗ Strategy selected on the development set for TTS-Q. Bold marks the best value per model before rounding. Full results appear in Appendix E .
Seed English
Seed Mandarin
Seed Mandarin hard
Model
Variant
WER ↓
SIM ↑
UTMOS ↑
CER ↓
SIM ↑
UTMOS ↑
CER ↓
SIM ↑
UTMOS ↑
1B
Random §
2.592
0.7466
3.673
1.271
0.8002
2.909
7.158
0.7813
2.727
Selected coverage
2.344
0.7435
3.674
1.186
0.8006
2.953
7.018
0.7816
2.757
Reconstruction continuation
2.518
0.7468
3.679
1.299
0.8001
2.940
7.516
0.7814
2.737
TTS-Q
2.475
0.7475
3.730
1.244
0.8023
2.990
7.158
0.7826
2.794
3.5B
Random §
2.089
0.7715
3.652
1.152
0.8088
2.995
6.382
0.7906
2.814
Table 3: Complete component comparison on Seed-TTS. WER/CER are percentages. Selected coverage is the first-stage checkpoint, using farthest-point selection for 1B and average-distance selection for 3.5B and F5. TTS-Q applies the second stage with identity supervision; Reconstruction continuation ablates this supervision ( λ=0 ). Both branches start from the same checkpoint with identical learning rates and update schedules. \lx@sectionsign Mean of three random calibration sets. Bold marks the best value per model and metric before rounding.
Figure 3: Coverage of the calibration pool in the original speaker embedding space. Cumulative distributions of cosine distance to the nearest selected condition, using 64 of 256 candidates. Curves farther left indicate closer coverage. Gray dashed curves show the three random draws; the initial 25% at zero comprises selected conditions.
Diffusion model quantization compresses model weights and activations to reduce inference memory and latency while preserving generated image, video, and speech quality.
Post-training quantization (PTQ) reduces the cost of on-device text-to-speech (TTS), but published evaluations cover one system or method. We evaluate PTQ across TTS architectures under one protocol with three core models, weight and activation ablations of eight more, and two held-out models quantized blind. Four-bit per-channel weights reduce UTMOS, a predicted mean opinion score, by 2.8 on Supertonic and 0.07 on Kokoro, and per-tensor scaling can cause severe degradation even at 8 bits. The same bit width yields different outcomes, because the sensitive component is model-specific and not reliably predicted from the model class. A staged ablation procedure identifies it, and per-layer GPTQ can restore it to within 0.1 UTMOS. Real int8 and int4 kernels reproduce the simulated ordering at hardware-dependent cost. On a Mac mini, a 4-bit weight kernel runs Supertonic at 0.60x the fp32 latency while int8 is slower, so each configuration requires validation on the target runtime.
Post-training quantization (PTQ) methods such as GPTQ and AWQ compress large language models effectively, but applying them to automatic speech recognition (ASR) at ultra-low bit-widths (2-3 bits) causes severe transcription degradation. Unlike discrete text tokens, speech activations are highly correlated in steady-state and zero-padded regions yet change rapidly at phonetic boundaries; when all frames contribute equally to the Hessian, static frames dominate calibration and obscure informative transitions. We propose DiffAQ, which computes frame-to-frame activation differences to measure the rate of acoustic change and assigns Hessian importance proportionally, concentrating quantization precision on phonetically critical frames. As a training-free modification to GPTQ, DiffAQ consistently reduces WER across various Whisper sizes and standard benchmarks, with the largest gains at 2-bit precision where baseline methods frequently produce degenerate outputs.
Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness, since local errors propagate to later positions and can escalate into hallucination. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, since the model receives the full input text before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a 1/t-weighted training objective and a time-shifted inference schedule that together defer low-confidence positions to later steps. Trained on only 585 hours of LibriTTS, DELTA-TTS achieves a 1.75% WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens 3.3 times faster. Further analysis shows that DELTA-TTS produces sharper text-speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation. Audio samples, inference code, and model weights are available at https://slpr-lab.github.io/DELTA-TTS.