Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering

Shih-Heng Wang; Tiantian Feng; Aditya Kommineni; Huang-Cheng Chou; Bowen Yi; Xuan Shi; Shrikanth Narayanan

cs.SDSep 28, 2026

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering

Authors: Shih-Heng Wang, Tiantian Feng, Aditya Kommineni, Huang-Cheng Chou, Bowen Yi, Xuan Shi, Shrikanth Narayanan

Organizations: University of Southern California, USA

Abstract

Neural audio codecs (NACs) are widely used in speech generation and audio-language modeling, yet how they encode speaker-trait information remains poorly understood. Prior work applied sparse autoencoders (SAEs) to investigate accent information in NACs through task-level analysis. Here, we extend this analysis to the waveform level and to age, gender, and accent, using SAE steering to probe trait-related information in sparse activations. We identify trait-associated dimensions, modify their activations, and evaluate the resulting reconstructed speech. Across five NACs, steering the selected dimensions induces target-directed shifts in speaker-trait predictions. A random-dimension baseline on Mimi produces smaller shifts, supporting the relevance of the selected dimensions. However, responses vary across codecs, traits, and steering directions, and increasing steering strength does not consistently amplify the intended shifts. Steering also generally increases word error rates and lowers predicted perceptual quality. These findings suggest that SAEs capture speaker-trait information in steerable activations, while the accompanying quality degradation highlights the need to better separate trait-related information from other information.

Figures & tables

Figure 1: Overview of the SAE steering framework. (A) Illustration of NAC encoding speech utterance into quantized frame-level representations. (B) A SAE decomposes each frame into sparse activations and reconstructs it. (C) Trait-associated activation dimensions are identified and modified to steer reconstructed speech from a source trait $g_{1}$ toward a target trait $g_{2}$ .

Figure 2: Steering results across five NACs for (a) gender, (b) age, (c) US–UK accent, and (d) US–SA accent. US, UK, and SA denote North American, British Isles, and Southern Asian accents, respectively. Diamonds indicate unsteered SAE reconstruction baselines; circles indicate steering results at steering strength $\alpha$ . Mimi Random uses randomly selected SAE dimensions from Mimi’s representation.

Waveform	Gender	Age	US–UK	US–SA
	Female / Male	Young / Senior	US / UK	US / SA
Original	95.9 / 7.6	27.8 / 63.1	89.6 / 3.2	57.3 / 1.7
EnCodec 6 kbps	94.9 / 7.8	36.7 / 65.0	89.3 / 3.8	67.4 / 2.4
DAC	96.0 / 7.2	28.7 / 63.7	88.2 / 5.0	73.7 / 4.4
Mimi	95.3 / 8.0	31.5 / 62.0	88.9 / 3.6	60.6 / 1.8
SpeechTokenizer	93.9 / 7.9	33.1 / 62.7	86.8 / 4.8	57.8 / 3.4

Table 1: Speaker-trait predictions before steering. Gender and accent scores are mean female and pairwise US probabilities (%), respectively; age is reported in years.

Figure 3: Speech quality after steering across NACs. The top row reports SHEET quality scores ( $\uparrow$ ), and the bottom row reports WER ( $\downarrow$ ).

Explore similar work

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: A Case Study on Accent Information
Mar 18, 2026Shih-Heng Wang, Tiantian Feng, Aditya Kommineni +4Neural Audio Codecs Paralinguistic Cues
University of Southern California, USA
Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions
Aug 31, 2026Joonyong Park, Shinnosuke Takamichi, David M. Chan +3Neural Audio Codecs Residual Vector Quantization
Grad. School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Faculty of Science and Technology, Keio University, Tokyo, Japan · Berkeley Artificial Intelligence Research Lab (BAIR), University of California, Berkeley, CA, USA
Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens
Jul 28, 2026Daigo Takizawa, Tomohiko Nakamura, Samuele Cornell +3Self-Supervised Speech Models Neural Audio Codecs
National Institute of Advanced Industrial Science and Technology (AIST), Japan · Carnegie Mellon University, USA

Mar 18, 2026, cs.SDJ/K move · Enter open · S save

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: A Case Study on Accent Information

Shih-Heng Wang, Tiantian Feng, Aditya Kommineni, Thanathai Lertpetchpun+3

University of Southern California, USA

Neural Audio Codecs Paralinguistic Cues Case Study

Open paper AskPDF arXiv

Neural Audio Codecs (NACs) are widely adopted in modern speech systems, yet how they encode linguistic and paralinguistic information remains unclear. Improving the interpretability of NAC representations is critical for understanding and deploying them in sensitive applications. Hence, we employ Sparse Autoencoders (SAEs) to decompose dense NAC representations into sparse, interpretable activations. In this work, we focus on a challenging paralinguistic attribute-accent-and propose a framework to quantify NAC interpretability. We evaluate four NAC models under 16 SAE configurations using a relative performance index. Our results show that DAC and SpeechTokenizer achieve the highest interpretability. We further reveal that acoustic-oriented NACs encode accent information primarily in activation magnitudes of sparse representations, whereas phonetic-oriented NACs rely more on activation positions, and that low-bitrate EnCodec variants show higher interpretability.

Abstract

Explore similar work

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: A Case Study on Accent Information

Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions

Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens