Emotion-Aware Quantization for Discrete Speech Representations: An Analysis of Emotion Preservation

Haoguang Zhou; Siyi Wang; Jingyao Wu; James Bailey; Ting Dang

cs.SDMar 22, 2026

Emotion-Aware Quantization for Discrete Speech Representations: An Analysis of Emotion Preservation

Authors: Haoguang Zhou, Siyi Wang, Jingyao Wu, James Bailey, Ting Dang

Organizations: The University of Melbourne, Australia · Massachusetts Institute of Technology, USA · Monash University, Australia

Abstract

Modern speech systems increasingly use discretized self-supervised speech representations for compression and integration with token-based models, yet their impact on emotional information remains unclear. We study how residual vector quantization (RVQ) reshapes emotional information in discrete speech representations from both representation- and task-level perspectives. Our analysis shows that aggressive compression disproportionately degrades emotion, with uneven loss across emotion classes and model architectures. To address this, we introduce emotion-aware quantization using emotion-specific and emotion-biased codebooks, improving the preservation of both hard and soft emotion perception. We further propose Emo-Q, a lightweight routed quantization method that selects emotion-specialized codebooks, improving emotion recognition performance at lower bitrates. These results highlight the importance of emotion-aware discretization for robust affective speech processing.

Figures & tables

Figure 1: Quantization of discrete speech representation

Figure 2: Overview of the pipeline . (a) Balanced and emotion-specific codebooks. (b) Representation-level evaluation: layer-wise degradation analysis (RQ1), primary emotion retention (RQ2), and soft distribution fidelity (RQ3). (c) Task-level evaluation: downstream SER via routed quantization (RQ4).

Panel of Figure 3 — Figure 3: Reconstruction fidelity (cosine similarity, top) and primary emotion recall (bottom) versus quantization depth under RVQ for three different SSL frontends.

Panel of Figure 4 — Figure 4: RQ2: affective retention (left) and codebook utilization (right) for balanced and emotion-specific quantization.

Panel of Figure 5 — Figure 5: RQ3 evaluation: emotion distribution matching (left) and top2 emotion recall (right) for varying codebook training strategies.

Config.	$8{\times}32$	$8{\times}64$	$8{\times}128$	$32{\times}2$	$32{\times}4$	$64{\times}2$	$128{\times}2$
Bal	$-0.09$	$+0.02$	$+0.02$	$-0.98$	$-0.53$	$-1.00$	$+0.17$
Emo-Q (100)	$+0.95$	$+0.97$	$+0.66$	$+0.51$	$\mathbf{+1.18}$	$\mathbf{+0.67}$	$\mathbf{+0.77}$
Emo-Q (99)	$\mathbf{+1.57}$	$\mathbf{+1.45}$	$\mathbf{+1.20}$	$\mathbf{+0.57}$	$+0.25$	$-0.18$	$-0.18$

Table 1: Primary emotion Macro-

F_{1}

improvement (

\Delta

) over the continuous baseline for different codebook configurations (Codebooks

\times

Entries).

Explore similar work

AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ
May 22, 2026Zhaoyang Meng, Zhengyao Ma, Kecan Mao +2Neural Audio Codecs Neural Encoding
Beijing University of Posts and Telecommunications
AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling
May 11, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Audio Representation Learning Neural Audio Codecs
College of William & Mary · Emory University · George Mason University
Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detection
Mar 10, 2026Jinyang Wu, Zihan Pan, Qiquan Zhang +2Audio Deepfake Detection Discrete Speech Representation Learning
Agency for Science, Technology and Research (A*STAR), Singapore · The University of New South Wales, Australia

May 22, 2026J/K move · Enter open · S save

AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ

Zhaoyang Meng, Zhengyao Ma, Kecan Mao, Yingming Gao+1

Beijing University of Posts and Telecommunications

Neural Audio Codecs Neural Encoding Neural Network Quantization Affective Computing

Open paper AskPDF arXiv

Neural speech codecs have become the discrete interface between raw audio and speech language models, yet they remain optimized primarily for acoustic reconstruction fidelity, which leaves emotion-relevant cues vulnerable to being discarded during quantization, limiting the affective capacity of downstream models. We trace this degradation to two mechanisms: reconstruction-driven bit allocation under limited bitrate and cross-stream leakage in concatenation-based codecs, where acoustic gradients can overwrite nominally emotion-reserved dimensions. We propose AffectCodec, an emotion-preserving neural speech codec built on Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ). By imposing block-diagonal input and output projections over emotion and acoustic subspaces, BD-RFSQ transforms bit allocation from implicit and loss-driven to explicit and structurally guaranteed, while still preserving a flat token interface for downstream speech language models. AffectCodec further combines this structurally constrained quantizer with multi-granularity emotion conditioning and multi-rate training, enabling robust affect preservation at low bitrates. Experiments across multiple emotional speech benchmarks show that AffectCodec substantially improves emotion preservation, especially in the low-bitrate regime, while maintaining competitive acoustic quality and intelligibility. These results suggest that structurally protected quantization is an effective principle for preserving emotion-relevant information and may provide a general route toward attribute-aware neural speech compression.

Abstract

Explore similar work

AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ

AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling

Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detection