ASR ensembling for phoneme intelligibility evaluation of speech anonymizers

Victor Ménestrel; Sebastian Möller; Slim Ouni; Dorothea Kolossa

cs.SDSep 23, 2026

ASR ensembling for phoneme intelligibility evaluation of speech anonymizers

Authors: Victor Ménestrel, Sebastian Möller, Slim Ouni, Dorothea Kolossa

Organizations: Technische Universität Berlin, Faculty IV, Berlin, Germany · Université de Lorraine, CNRS, Inria, Loria, F-54000, Nancy, France

Abstract

We present the first phoneme-level intelligibility evaluation of speech anonymizers, assessing the performance of ASR-ensemble-based metrics against measured intelligibility from a crowdsourced listening test. Our results show that simple hard-voting ASR metric reaches correlations above 0.9 with human ratings when aggregated by feature, test-type, or condition, provided that multiple ASR models are combined; evaluating stimuli with and without a carrier sentence further improves the correlation at the stimulus level. However, posterior-probability-based confidence metrics bring no gain, which can be traced back to the insufficient calibration of the state-of-the-art open ASR models that were utilized here. All data, code, and evaluation tools are released as open source.

Figures & tables

Condition	Intelligibility (%) $\uparrow$	WER (%) $\downarrow$
B2 (McAdams)	72.2	9.96
B3 (STTTS)	70.4	4.31
B4 (Neural Codec)	85.2	5.90
B5 (ASR-BN)	78.2	4.44
Qwen-TTS	90.2	N/A
Clean (reference)	91.8	1.84

Table 1: Intelligibility (human ratings) and reported WER by the VPC utility evaluation protocol, per condition. Note: Since Qwen-TTS is not an anonymizer, the WER from the VPC evaluation does not apply.

Aggregation	Carrier	${I_{\mathrm{soft}}}\uparrow$	${I_{\mathrm{hard}}}\uparrow$	${I_{\mathrm{LLR}}}\uparrow$
stimuli ( $N=2006$ )	w/o	0.4039	0.4100	0.3164
	w/	0.3371	0.3484	0.3224
	w/ + w/o	0.4424	0.4513	0.3852
$C$ ( $N=36$ )	w/o	0.8159	0 .8208	0.7461
	w/	0.7284	0.7251	0.7198
	w/ + w/o	0.8085	0.8079	0.7708

Table 2:

r^{2}

between human intelligibility & ASR-based scores (

I_{\mathrm{soft}}

I_{\mathrm{hard}}

I_{\mathrm{LLR}}

), carrier condition (without / with / both), and aggregation level (

N

= number of data points). A Steiger’s

Z

test confirms that

r_{I_{\mathrm{soft}}}

and

r_{I_{\mathrm{hard}}}

differ significantly at the stimulus level (

p<0.05

), but we cannot reject the null hypothesis at higher aggregation levels.

r_{w/o}

and

r_{w/+w/o}

differ significantly at the stimulus level (

p<0.05

Model	w/o $\uparrow$	w/ $\uparrow$	w/ + w/o $\uparrow$
Cohere Transcribe	0.2392	0.1922	0.2874
wav2vec2-lv-60-espeak	0.1739	0.2163	0.2419
wav2vec2-xlsr-53-espeak	0.1714	0.1440	0.2041
Whisper-large-v3	0.1379	0.2069	0.2690
Whisper-large-v3-turbo	0.1185	0.1849	0.2424
Distil-Whisper-large-v3.5	0.1888	0.1943	0.2826

Table 3:

r^{2}

between listening-test intelligibility according to Eq. ( 1 ) and

I_{\mathrm{soft}}

, per ASR model and carrier condition at the stimulus aggregation level. Model names are abbreviated. The last row reports the ensemble; the fourth column gives the score when stimuli with and without carrier are combined.

Explore similar work

PHONOS: PHOnetic Neutralization for Online Streaming Applications
Mar 27, 2026Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah +1Grapheme-To-Phoneme Anonymization
Department of Computer Science & Engineering, Texas A&M University, College Station, US
Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs
Jun 28, 2026Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov +4Automatic Speech Recognition Evaluation
NVIDIA, USA
A Large-Scale Per-Speaker Analysis of Re-identification Risk in Speech Anonymization
Jun 5, 2026Orane Dufour, Paul Magron, Mickael Rouvier +1Anonymization Speaker
Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France · LIA, Avignon University, F-84911 Avignon, France

Mar 27, 2026, eess.ASJ/K move · Enter open · S save

PHONOS: PHOnetic Neutralization for Online Streaming Applications

Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah, Ricardo Gutierrez-Osuna

Department of Computer Science & Engineering, Texas A&M University, College Station, US

Grapheme-To-Phoneme Anonymization Neutrality Device

Open paper AskPDF arXiv

Speaker anonymization (SA) systems modify timbre while leaving regional or non-native accent cues intact, which is problematic because such cues can reveal a speaker's first-language or geographic background and narrow the anonymity set. To address this issue, we present PHONOS, a streaming module for real-time SA that performs accent neutralization in a privacy sense: reducing accent-origin cues by converting non-native segmental realizations toward a chosen target accent domain. Our approach pre-generates golden speaker utterances that preserve source timbre and rhythm but replace foreign segmentals with native ones using silence-aware DTW alignment and zero-shot voice conversion. These utterances supervise a causal accent translator that maps non-native content tokens to native equivalents with at most 40ms look-ahead, trained using joint cross-entropy and CTC losses. Our evaluations show an 81% reduction in non-native accent confidence, with listening-test accentedness ratings consistent with this shift. PHONOS also moves outputs away from the original speaker in embedding space, suggesting lower linkability under an embedding-based proxy, while running with

\leq241\,\mathrm{ms}

end-to-end latency on a single GPU.

Abstract

Explore similar work

PHONOS: PHOnetic Neutralization for Online Streaming Applications

Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

A Large-Scale Per-Speaker Analysis of Re-identification Risk in Speech Anonymization