Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography

Ting-Hui Cheng; Line H. Clemmensen; Sneha Das

cs.LGMar 5, 2026

Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography

Authors: Ting-Hui Cheng, Line H. Clemmensen, Sneha Das

Organizations: Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark · Department of Mathematical Sciences, University of Copenhagen, Denmark

Abstract

Automatic speech recognition (ASR) systems are predominantly evaluated using the Word Error Rate (WER). However, raw token-level metrics fail to capture semantic fidelity and routinely obscures the `diversity tax', the disproportionate burden on marginalized and atypical speaker due to systematic recognition failures. In this paper, we explore the limitations of relying solely on lexical counts by systematically evaluating a broader class of non-linear and semantic metrics. To enable rigorous model auditing, we introduce the sample difficulty index (SDI), a novel metric that quantifies how intrinsic demographic and acoustic factors drive model failure. By mapping SDI on data cartography, we demonstrate that metrics EmbER and SemDist expose hidden systemic biases and inter-model disagreements that WER ignores. Finally, our findings are the first steps towards a robust audit framework for prospective safety analysis, empowering developers to audit and mitigate ASR disparities prior to deployment.

Figures & tables

Example	Metric	WER	CER	MER	WIL	EmbER	SemDist
	Level	word	character	word	word	word	sentence
	Range	$[0,\infty)$	$[0,\infty)$	$[0,100]$	$[0,100]$	$[0,\infty)$	$[0,\infty]$
	Usage%	86.56%	35.41%	0.66%	0.33%	0.33%	1.31%
Ref: ..go meet..		0.014	0.009	0.014	0.014	1.429	0.073
Pred: ..go to meet..
Ref: ..Ask her..		0.014	0.006	0.014	0.014	1.429	0.188

Table 1: Overview of ASR evaluation metrics, including unit level, value range, and usage in Interspeech papers over the past three years, alongside examples of reference and predicted transcriptions from the Speech Accent Archive. Bold indicates the worst (highest) score in each column. Italics highlight values that are identical to at least one other value in the same column, showing where metrics fail to distinguish between corpora.

Figure 1: Characteristics of datasets as averages or ratios.

Figure 2: A) Mean and std error of $\beta,\alpha$ coefficients from the fixed effect (FE) model for the demographic and acoustic characteristics. B) Mean Coefficients from FE of models and datasets; C) Summary of the FE model fit statistics across the six metrics.

Figure 3: Latent space mapping of ASR performance using Principal Component Analysis. Axes are truncated to highlight the primary variance clusters.

Figure 4: Cartography plots mapping mean error ( $\mu$ ) against inter-model disagreement ( $\sigma$ ), colored by SDI decile (1 = Easiest, 10 = Hardest). SDI Deciles divide the dataset’s speech samples into ten equal tiers based on their calculated intrinsic difficulty, ranging from 1 (the easiest samples for models to transcribe) to 10 (the hardest).

Figure 5: Cartography plots of mean EmbER error ( $\mu$ ) against inter-model disagreement ( $\sigma$ ) across ASR systems using the EmbER metric. The $\mu$ reflects the overall recognition difficulty of each sample, while $\sigma$ captures the extent of ambiguity, indicating how consistently different ASR models perform on the same utterance.

ASR Evaluation

Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.

See all 144 papers

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
Sep 18, 2026Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto +4ASR Evaluation Semantic Textual Similarity
Faculty of Computer Science and Technology, Algoma University · Idiap Research Institute · School of Psychology, Algoma University +1
ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition
Feb 28, 2026Swapnil ParekhDemographic Bias in ASR Automatic Speech Recognition
Intuit
OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics
Jun 19, 2026Korbinian Kuhn, Gottfried ZimmermannASR Evaluation Low-Resource Language Processing
Stuttgart Media University, Germany · University of Tübingen, Germany

Sep 18, 2026J/K move · Enter open · S save

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto, Petr Motlicek+3

Faculty of Computer Science and Technology, Algoma University · Idiap Research Institute · School of Psychology, Algoma University+1

ASR Evaluation Semantic Textual Similarity

Open paper AskPDF arXiv

Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.

Abstract

ASR Evaluation

Explore similar work

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition

OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics