Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics

Jihwan Lee; Tiantian Feng; Minu Kim; Junhyeok Lee; Sean Foley; Thanathai Lertpetchpun; Najim Dehak; Dani Byrd

eess.ASOct 14, 2026

Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics

Authors: Jihwan Lee, Tiantian Feng, Minu Kim, Junhyeok Lee, Sean Foley, Thanathai Lertpetchpun, Najim Dehak, Dani Byrd,

Organizations: Signal Analysis and Interpretation Lab (SAIL), University of Southern California · Center for Language and Speech Processing, Johns Hopkins University

Abstract

Neural audio codecs are increasingly used for low-bitrate speech coding and serve as tokenizers for speech language models, yet it remains unclear how their codebook hierarchies encode speaker traits and speech characteristics. We systematically study 17 multi-codebook neural audio codecs across seven traits: sex, age, accent, voice quality, emotion, speech flow, and non-verbal sounds. For each codec, we reconstruct speech using partial codebooks and use machine listeners to compare trait predictions on each reconstruction. We measure prediction agreement with reweighted Cohen's \k{appa}, and define trait depth as a proxy for the codebook depth by which each trait is encoded and recoverable. Overall, sex has the shallowest trait depth, followed by speech flow, whereas voice quality and age group require deeper stacks; emotion, accent, and non-verbal sounds fall in between. Within each trait, speech-flow classes are at relatively similar trait depths, whereas emotion, accent, and non-verbal sounds differ more. Additionally, we explore how certain codec design choices relate to trait depths. These findings show that codebook hierarchies are trait-dependent and may not exhibit uniform coarse-to-fine structures, motivating trait-specific codec evaluation and design for future applications.

Figures & tables

Figure 1: Overview of the protocol. (1) A speech utterance

x

with different speaker and speech traits. (2) A neural audio codec encodes

x

into

K

codebooks and decodes with partial codebooks

\hat{x}_{k}=D(c_{1:k})

. (3) Machine listeners predict each trait on

x

and on every

\hat{x}_{k}

. (4) The reweighted agreement (Cohen’s

\kappa

) between

\hat{x}_{k}

and

x

is measured; and trait depth

d_{t}

is defined as the relative codebook depth where

\kappa

first reaches 90% of its full-depth value.

Codec	Framerate	Sampling Rate	Number of	Bitrate	Distillation	Streaming
	(Hz)	(kHz)	Codebooks	(kbps)
EnCodec ( Défossez et al., 2023 )	75	24	32	24	✗	✓
DAC ( Kumar et al., 2023 )	50	16	8 †	4	✗	✗
FunCodec 50 Hz ( Du et al., 2024 )	50	16	32	16	✗	✗
FunCodec 25 Hz	25	16	32	8	✗	✗
SpeechTokenizer ( Zhang et al., 2024 )	50	16	8	4	✓	✗

Table 1: The 17 neural audio codecs with multiple codebooks and their codec design.

Trait	Labels	Machine Listeners	Dataset
Sex Age	female, male under 30, 30–60, over 60	VoxProfile; audeering	TIMIT, VoxCeleb, Common Voice
Accent	North America, (England) English, Scottish, Irish, Northern Irish, Welsh, Oceania, South African, Germanic, Romance, Slavic, Semitic, East Asia, Southeast Asia, South Asia, other	Voxlect; GenAID; CommonAccent ECAPA	Voxlect English test set (Section 3.2 )
Voice quality	Pitch : shrill, nasal, deep Rhythm : singsong, pitchy, flowing, monotone, staccato, punctuated, enunciated, hesitant Clarity : crisp, slurred, lisp, stammering Texture : silky, husky, raspy, guttural, vocal fry Volume : booming, authoritative, loud, hushed, soft	VoxProfile	ParaSpeechCaps
Emotion	anger, contempt, disgust, fear, happiness, neutral, sadness, surprise, other	VoxProfile; emotion2vec+	IEMOCAP
Speech flow	fluent, block, prolongation, sound repetition, word repetition, interjection	VoxProfile	FluencyBank, SEP-28k
Non-verbal	laughter, sigh, cough, throat clearing, sneeze, sniff	SSLAM; AST; SenseVoice	VocalSound

Table 2: Speaker traits and speech characteristics used for analysis and their labels, adopted from VoxProfile ( Feng et al., 2025 ) , Voxlect ( Feng et al., 2026 ) , and VocalSound ( Gong et al., 2022 ) ; machine listeners used to predict each trait; and the corresponding datasets.

Figure 2: Trait depths

d_{t}

for each codec for each trait, the relative codebook depth at which the reweighted

\kappa

first reaches 90% of its full-depth value. Each dot is a codec and the gray bars indicate the mean values. Cool and warm colors indicate distilled versus non-distilled codecs, respectively. Different framerates and streamability have different markers.

Figure 3: Per-class reweighted $\kappa$ by relative codebook depth, averaged over the 17 codecs. Darker cells indicate higher agreement, and a tick marks trait depth , i.e., the first depth where $\kappa$ reaches 90% of its full-depth value.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Codec	$K$	Decoded prefixes $k$
EnCodec	32	1–8, 12, 16, 24, 32
DAC	8 †	1–8
FunCodec 50 Hz	32	1–8, 12, 16, 24, 32
FunCodec 25 Hz	32	1–8, 12, 16, 24, 32
SpeechTokenizer	8	1–8
Mimi	32	1–8, 12, 16, 24, 32

Appendix

Table 3: Codebook prefixes decoded for each codec, in the order of Table 1 . † The first eight of DAC’s 12 codebooks.

		$\rho$ of $\kappa$
Trait	Auxiliary Listener	$k=1$	All
Sex	audeering	0.98	0.94
Age group	audeering	0.95	0.96
Accent	GenAID	0.79	0.97
	CommonAccent ECAPA	0.82	0.98
Emotion	emotion2vec+	0.82	0.87

Appendix

Table 4: Spearman’s

\rho

between the

\kappa

of each trait’s representative and auxiliary machine listeners.

Trait	Clips	Speakers	Hours
Accent	39,911	173	82.18
Sex, age group	12,619	1,295	23.45
Voice quality	13,076	63	27.16
Non-verbal sound	3,591	574	4.14
Emotion	600	10	1.00
Speech flow	1,622	162	1.35

Appendix

Table 5: The number of clips, speakers, and total hours of the audio.

Codec	Checkpoint	Code	License
EnCodec	huggingface.co/facebook/encodec_24khz	github.com/huggingface/transformers	Apache-2.0
DAC	huggingface.co/descript/dac_16khz	github.com/huggingface/transformers	MIT, Apache-2.0
FunCodec (50, 25 Hz)	50 Hz: huggingface.co/alibaba-damo/audio_codec-encodec-en-libritts-16k-nq32ds320-pytorch ; 25 Hz: huggingface.co/alibaba-damo/audio_codec-encodec-en-libritts-16k-nq32ds640-pytorch	github.com/modelscope/FunCodec	MIT
SpeechTokenizer	huggingface.co/fnlp/SpeechTokenizer ( speechtokenizer_hubert_avg )	github.com/ZhangXInFD/SpeechTokenizer	Apache-2.0
Mimi	huggingface.co/kyutai/mimi	github.com/huggingface/transformers	CC BY 4.0, Apache-2.0
HILCodec	github.com/aask1357/hilcodec/tree/master/onnx ( hil_speech )	github.com/aask1357/hilcodec	MIT

Appendix

Table 6: Checkpoint, code, and license of the codecs.

Listener	Checkpoint	Code	License
VoxProfile	huggingface.co/tiantiaf	github.com/tiantiaf0627/vox-profile-release	OpenRAIL, RAIL-S 1.1
Voxlect	huggingface.co/tiantiaf/voxlect-english-dialect-whisper-large-v3	github.com/tiantiaf0627/voxlect	OpenRAIL, RAIL-S 1.1
SSLAM	huggingface.co/ta012/SSLAM_AS2M_Finetuned (rev. 4500c5d )	github.com/ta012/SSLAM	MIT
audeering	huggingface.co/audeering/wav2vec2-large-robust-24-ft-age-gender	github.com/huggingface/transformers	CC BY-NC-SA 4.0, Apache-2.0
GenAID	drive.google.com/file/d/1slGrpZSu5g-nF7R-QMCmtGcjN3kw7lQj/view ( GenAID_v6 )	github.com/jzmzhong/GenAID	Apache-2.0
CommonAccent ECAPA	huggingface.co/Jzuluaga/accent-id-commonaccent_ecapa	github.com/speechbrain/speechbrain	MIT, Apache-2.0

Appendix

Table 7: Checkpoint, code, and license of machine listeners.

Figure 4: An example screenshot of the interactive webpage.

Neural Audio Codecs

Neural audio codecs learn compact representations for compressing and reconstructing speech, music, and other sounds.

See all 87 papers

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering
Sep 28, 2026Shih-Heng Wang, Tiantian Feng, Aditya Kommineni +4Neural Audio Codecs Sparse Autoencoders
University of Southern California, USA
How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality
Sep 10, 2026Chibuzor Okocha, Christan Earl GrantNeural Audio Codecs Speech Processing
Department of Computer & Information Science & Engineering University of Florida, Gainesville, FL, USA
AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling
May 11, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Audio Representation Learning Neural Audio Codecs
College of William & Mary · Emory University · George Mason University

Sep 28, 2026J/K move · Enter open · S save

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering

Shih-Heng Wang, Tiantian Feng, Aditya Kommineni, Huang-Cheng Chou+3

University of Southern California, USA

Neural Audio Codecs Sparse Autoencoders Controllable Speech Generation

Open paper AskPDF arXiv

Neural audio codecs (NACs) are widely used in speech generation and audio-language modeling, yet how they encode speaker-trait information remains poorly understood. Prior work applied sparse autoencoders (SAEs) to investigate accent information in NACs through task-level analysis. Here, we extend this analysis to the waveform level and to age, gender, and accent, using SAE steering to probe trait-related information in sparse activations. We identify trait-associated dimensions, modify their activations, and evaluate the resulting reconstructed speech. Across five NACs, steering the selected dimensions induces target-directed shifts in speaker-trait predictions. A random-dimension baseline on Mimi produces smaller shifts, supporting the relevance of the selected dimensions. However, responses vary across codecs, traits, and steering directions, and increasing steering strength does not consistently amplify the intended shifts. Steering also generally increases word error rates and lowers predicted perceptual quality. These findings suggest that SAEs capture speaker-trait information in steerable activations, while the accompanying quality degradation highlights the need to better separate trait-related information from other information.

Abstract

Neural Audio Codecs

Explore similar work

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering

How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality

AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling