Modern speech systems increasingly use discretized self-supervised speech representations for compression and integration with token-based models, yet their impact on emotional information remains unclear. We study how residual vector quantization (RVQ) reshapes emotional information in discrete speech representations from both representation- and task-level perspectives. Our analysis shows that aggressive compression disproportionately degrades emotion, with uneven loss across emotion classes and model architectures. To address this, we introduce emotion-aware quantization using emotion-specific and emotion-biased codebooks, improving the preservation of both hard and soft emotion perception. We further propose Emo-Q, a lightweight routed quantization method that selects emotion-specialized codebooks, improving emotion recognition performance at lower bitrates. These results highlight the importance of emotion-aware discretization for robust affective speech processing.
Figures & tables
Figure 1: Quantization of discrete speech representation
Figure 2: Overview of the pipeline . (a) Balanced and emotion-specific codebooks. (b) Representation-level evaluation: layer-wise degradation analysis (RQ1), primary emotion retention (RQ2), and soft distribution fidelity (RQ3). (c) Task-level evaluation: downstream SER via routed quantization (RQ4).
Figure 3: Reconstruction fidelity (cosine similarity, top) and primary emotion recall (bottom) versus quantization depth under RVQ for three different SSL frontends.
Figure 4: RQ2: affective retention (left) and codebook utilization (right) for balanced and emotion-specific quantization.
Figure 5: RQ3 evaluation: emotion distribution matching (left) and top2 emotion recall (right) for varying codebook training strategies.
Config.
8×32
8×64
8×128
32×2
32×4
64×2
128×2
Bal
−0.09
+0.02
+0.02
−0.98
−0.53
−1.00
+0.17
Emo-Q (100)
+0.95
+0.97
+0.66
+0.51
+1.18
+0.67
+0.77
Emo-Q (99)
+1.57
+1.45
+1.20
+0.57
+0.25
−0.18
−0.18
Table 1: Primary emotion Macro- F1 improvement ( Δ ) over the continuous baseline for different codebook configurations (Codebooks × Entries).
Neural speech codecs have become the discrete interface between raw audio and speech language models, yet they remain optimized primarily for acoustic reconstruction fidelity, which leaves emotion-relevant cues vulnerable to being discarded during quantization, limiting the affective capacity of downstream models. We trace this degradation to two mechanisms: reconstruction-driven bit allocation under limited bitrate and cross-stream leakage in concatenation-based codecs, where acoustic gradients can overwrite nominally emotion-reserved dimensions. We propose AffectCodec, an emotion-preserving neural speech codec built on Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ). By imposing block-diagonal input and output projections over emotion and acoustic subspaces, BD-RFSQ transforms bit allocation from implicit and loss-driven to explicit and structurally guaranteed, while still preserving a flat token interface for downstream speech language models. AffectCodec further combines this structurally constrained quantizer with multi-granularity emotion conditioning and multi-rate training, enabling robust affect preservation at low bitrates. Experiments across multiple emotional speech benchmarks show that AffectCodec substantially improves emotion preservation, especially in the low-bitrate regime, while maintaining competitive acoustic quality and intelligibility. These results suggest that structurally protected quantization is an effective principle for preserving emotion-relevant information and may provide a general route toward attribute-aware neural speech compression.
Zhaoyang Meng, Zhengyao Ma, Kecan Mao +2
Beijing University of Posts and Telecommunications
Neural speech codecs provide discrete representations for speech language models, but emotional cues are often degraded during quantization. Existing codecs mainly optimize acoustic reconstruction, leaving emotion expressiveness insufficiently modeled at the representation level. We propose an emotion-guided neural speech codec that explicitly preserves emotional information while maintaining semantic fidelity and prosodic naturalness. Our framework combines emotion-semantic guided latent modulation, relation-preserving emotional-semantic distillation, and emotion-weighted semantic alignment to retain emotionally salient cues under compression. Extensive evaluations across speech reconstruction, emotion recognition, and downstream text-to-speech generation demonstrate improved emotion consistency and perceptual quality without sacrificing content accuracy.
Jiacheng Shi, Hongfei Du, Xinyuan Song +3
College of William & Mary · Emory University · George Mason University
Neural audio codecs discretize speech via residual vector quantization (RVQ), forming a coarse-to-fine hierarchy across quantizers. While codec models have been explored for representation learning, their discrete structure remains underutilized in speech deepfake detection. In particular, different quantization levels capture complementary acoustic cues, where early quantizers encode coarse structure and later quantizers refine residual details that reveal synthesis artifacts. Existing systems either rely on continuous encoder features or ignore this quantizer-level hierarchy. We propose a hierarchy-aware representation learning framework that models quantizer-level contributions through learnable global weighting, enabling structured codec representations aligned with forensic cues. Keeping the speech encoder backbone frozen and updating only 4.4% additional parameters, our method achieves relative EER reductions of 46.2% on ASVspoof 2019 and 13.9% on ASVspoof5 over strong baselines.
Jinyang Wu, Zihan Pan, Qiquan Zhang +2
Agency for Science, Technology and Research (A*STAR), Singapore · The University of New South Wales, Australia