How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality

Chibuzor Okocha; Christan Earl Grant

cs.SDSep 10, 2026

How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality

Authors: Chibuzor Okocha, Christan Earl Grant

Organizations: Department of Computer & Information Science & Engineering University of Florida, Gainesville, FL, USA

Abstract

Neural audio codecs enable low-bitrate speech compression and tokenization, yet their robustness on accented and multilingual speech remains under-evaluated. We benchmark seven open-source neural codecs (DAC, EnCodec, FocalCodec, LanguageCodec, SemantiCodec, UniCodec, WavTokenizer) on three African speech datasets afrinames, afrispeech dialog, afrispeech multilingual, reporting signal-level quality (NISQA, UTMOS, ViSQOL, STOI, F0-RMSE) alongside two downstream tasks: automatic speech recognition (ASR) and speaker verification (ASV). Our analysis yields four findings. First, signal metrics differ sharply in downstream validity: reference-based structural/intelligibility measures (ViSQOL, STOI) and prosodic error (F0-RMSE) track ASR and ASV degradation far more reliably than neural mean-opinion-score predictors (NISQA, UTMOS). Second, intelligibility and speaker-identity preservation diverge sharply across architectures, and the apparent identity ranking itself depends on the ASV backend. Third, degradation is strongly domain-dependent and largest for conversational dialog. Fourth, the resulting degradation is partly recoverable: parameter-efficient codec adaptation (LoRA, $\sim$ 1--3% of parameters) on roughly 35 hours of African speech reduces the compression-induced word-error-rate gap, recovering recognition toward the uncompressed baseline (developed in companion work). These results motivate task-aware, domain-representative, and adaptation-aware evaluation of speech codecs as a prerequisite for inclusive deployment.

Figures & tables

Dataset	Manifest	Eval.	ASV trials	Coverage
	clips	files
afrinames	280	115–150	13.2k–22.5k	8 accents, 24 kHz
afrispeech_dialog	210	49	2.4k	6 accents, 24 kHz
afrispeech_multilingual	–	8–100	64–10.0k	Common Voice, 24 kHz mono

TABLE I: Dataset and evaluation protocol summary. “Eval.” counts are the files scored in the primary downstream runs; ASV counts are exhaustive target/impostor trials.

Codec	Primary Metric	Configs
DAC [ 2 ]	Bitrate (kbps)	8, 16, 24
EnCodec [ 1 ]	Bitrate (kbps)	3, 6, 12, 24
FocalCodec [ 13 ]	Bitrate (kbps)	0.16–0.65
LanguageCodec [ 14 ]	Bitrate (kbps)	0.3 (target)
SemantiCodec [ 12 ]	Token Rate (Hz)	25, 50, 100
UniCodec [ 17 ]	Bitrate (kbps)	3.4, 6.6

TABLE II: Neural audio codec configurations. RVQ models are defined by bitrate; semantic and tokenization models by token rate (Hz).

Dataset		NISQA	UTMOS	ViSQOL	STOI
afrinames	ref.	2.92	2.42	5.00	1.00
afrinames	codecs	3.25	2.32	4.59	0.85
afrispeech_dialog	ref.	3.25	2.27	5.00	1.00
afrispeech_dialog	codecs	3.13	2.06	4.87	0.71
afrispeech_multilingual	ref.	2.35	1.84	5.00	1.00
afrispeech_multilingual	codecs	2.74	1.78	4.88	0.81

TABLE III: Signal-level quality per dataset (mean over codec variants) vs. the uncompressed reference. ViSQOL/STOI: higher is better; STOI

\in[0,1]

Dataset	Down.	NISQA	UTMOS	ViSQOL	STOI	F0
dialog	WER	$-0.43$	$-0.26$	$\mathbf{-0.74}$	$-0.72$	$\mathbf{0.85}$
dialog	CER	$-0.43$	$-0.27$	$\mathbf{-0.74}$	$-0.72$	$\mathbf{0.85}$
dialog	EER	$-0.46$	$0.15$	$-0.57$	$-0.54$	$\mathbf{0.64}$
dialog	minDCF	$-0.56$	$0.10$	$-0.65$	$-0.63$	$\mathbf{0.72}$
afrinames	EER	$0.49$	$0.32$	$-0.93$	$-0.89$	$\mathbf{0.95}$
afrinames	minDCF	$0.59$	$0.45$	$-0.83$	$-0.79$	$\mathbf{0.86}$

TABLE IV: Signal-vs.-downstream Spearman correlations after recomputing ASV rows with ECAPA-TDNN EER/minDCF. Negative

\rho

means higher signal quality corresponds to lower downstream error; positive F0-RMSE correlations mean larger pitch error corresponds to larger downstream error.

Domain	Codec/variant	Signal	Down.	Ranks	MMG
multiling.	Focal 12.5 Hz	UTMOS	EER	1 / 31	30
afrinames	Focal 12.5 Hz	UTMOS	EER	1 / 31	30
dialog	WavTok S-600	NISQA	WER	1 / 28	27
dialog	WavTok S-600	NISQA	CER	1 / 28	27

TABLE V: Representative metric-masking cases. Low signal rank is good, but high downstream rank is bad; MMG is the rank gap.

Codec/Var.	EER $\downarrow$	minDCF $\downarrow$	WER $\downarrow$	CER $\downarrow$
DAC
16kbps	0.00	0.00	28.26	25.54
24kbps	0.00	0.00	29.23	26.25
8kbps	0.00	0.00	29.74	26.86
EnCodec
12kbps	0.40	0.18	30.34	27.29

TABLE VI: Downstream behavior on afrispeech_dialog : speaker verification (ECAPA-TDNN) vs. recognition (Whisper-large-v3; uncompressed baseline 17.3 WER). EER and WER/CER in %. Lower is better (

\downarrow

). Several ASR-usable codecs still fail verification.

Fig. 1: Per-variant ASR error on afrispeech_dialog (Whisper-large-v3), sorted by WER. Mid-bitrate RVQ codecs (DAC, EnCodec, LanguageCodec) cluster near the uncompressed baseline (dashed), whereas aggressive semantic and low-token-rate settings degrade severely; CER tracks WER closely.

Codec / Variant	afrinames	dialog	multiling.
DAC 8kbps	0.96	0.00	0.00
DAC 16kbps	0.75	0.00	0.00
DAC 24kbps	0.95	0.00	0.00
EnCodec 3kbps	3.18	2.04	2.00
EnCodec 6kbps	1.93	1.81	1.00
EnCodec 12kbps	1.39	0.40	1.00

TABLE VII: Speaker verification EER (%, ECAPA-TDNN) across the three domains. Lower is better. “–” = condition not run.

Recognizer	Pretrained	Enc-LoRA	Full-LoRA	Full FT
Whisper-lg-v3	21.21	17.81	18.94	19.35
parakeet-tdt	12.48	11.48	11.42	11.54

TABLE VIII: Codec adaptation on a held-out afrispeech_dialog subset (49 files), DAC 8 kbps. WER (%): pretrained codec vs. parameter-efficient (LoRA) and full fine-tuning, with the recognizer held fixed. Lower is better (

\downarrow

Neural Audio Codecs

Neural audio codecs learn compact representations for compressing and reconstructing speech, music, and other sounds.

See all 87 papers

Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics
Oct 14, 2026Jihwan Lee, Tiantian Feng, Minu Kim +6Neural Audio Codecs Speech Processing
Signal Analysis and Interpretation Lab (SAIL), University of Southern California · Center for Language and Speech Processing, Johns Hopkins University
Attack-Dependent Robustness of Neural Audio Codecs for Adversarial ASR
Mar 10, 2026Jordan Prescott, Thanathai Lertpetchpun, Shrikanth NarayananASR Evaluation Neural Audio Codecs
Signal Analysis and Interpretation Laboratory, University of Southern California, USA
AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ
May 22, 2026Zhaoyang Meng, Zhengyao Ma, Kecan Mao +2Neural Audio Codecs Neural Encoding
Beijing University of Posts and Telecommunications

Oct 14, 2026J/K move · Enter open · S save

Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics

Jihwan Lee, Tiantian Feng, Minu Kim, Junhyeok Lee+5

Signal Analysis and Interpretation Lab (SAIL), University of Southern California · Center for Language and Speech Processing, Johns Hopkins University

Neural Audio Codecs Speech Processing Audio Representation Learning Discrete Speech Representation Learning

Open paper AskPDF arXiv

Neural audio codecs are increasingly used for low-bitrate speech coding and serve as tokenizers for speech language models, yet it remains unclear how their codebook hierarchies encode speaker traits and speech characteristics. We systematically study 17 multi-codebook neural audio codecs across seven traits: sex, age, accent, voice quality, emotion, speech flow, and non-verbal sounds. For each codec, we reconstruct speech using partial codebooks and use machine listeners to compare trait predictions on each reconstruction. We measure prediction agreement with reweighted Cohen's \k{appa}, and define trait depth as a proxy for the codebook depth by which each trait is encoded and recoverable. Overall, sex has the shallowest trait depth, followed by speech flow, whereas voice quality and age group require deeper stacks; emotion, accent, and non-verbal sounds fall in between. Within each trait, speech-flow classes are at relatively similar trait depths, whereas emotion, accent, and non-verbal sounds differ more. Additionally, we explore how certain codec design choices relate to trait depths. These findings show that codebook hierarchies are trait-dependent and may not exhibit uniform coarse-to-fine structures, motivating trait-specific codec evaluation and design for future applications.

Abstract

Neural Audio Codecs

Explore similar work

Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics

Attack-Dependent Robustness of Neural Audio Codecs for Adversarial ASR

AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ