mu-bench: A Multilingual Utterance Transcription Benchmark

Andrea Li; Soham Ray

cs.CLSep 25, 2026

mu-bench: A Multilingual Utterance Transcription Benchmark

Authors: Andrea Li, Soham Ray

Organizations: UC Berkeley · Sierra AI

Abstract

Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin, centered on form-field inputs such as names, email addresses, and confirmation codes. We release Utterance Error Rate (UER), an LLM judge of whether a transcript preserves meaning, calibrated against human raters, together with an LLM normalizer that makes WER comparable across providers' output formats. On 1,847 human-rated transcripts, UER agrees with annotators at $κ$ = 0.78, versus 0.53 for exact-match WER on normalized text. We rank six commercial providers on a public leaderboard; the best reaches 11.9% UER, and Mandarin is hardest for all six.

Figures & tables

Benchmark	Multi.	Source
LibriSpeech [ 14 ]		Read
TED-LIUM 3 [ 10 ]		Prepared
GigaSpeech [ 4 ]		Web audio
SPGISpeech [ 13 ]		Earnings calls
People’s Speech [ 7 ]		Found
MLS [ 16 ]	✓	Read

Table 1: Coverage of ASR benchmarks on the three criteria in § 2.1

Locale	Utt.	Conv.	Audio (hr)	Length (s)
en-US	817	50	0.93	4.1
es-MX	792	50	1.01	4.6
tr-TR	846	50	1.18	5.0
vi-VN	975	50	1.21	4.5
zh-CN	840	50	0.76	3.3
Total	4,270	250	5.08	4.3

Table 2: mu-bench composition by locale. Length is the mean utterance length in seconds.

Norm.	Metric	$\boldsymbol{\kappa}\,\uparrow$	95% CI	Prec $\uparrow$	Rec $\uparrow$
mu-bench	UER	0.779	[0.717, 0.835]	0.859	0.764
none	UER	0.764	[0.699, 0.821]	0.831	0.764
Whisper	UER	0.681	[0.606, 0.748]	0.705	0.753
blind LLM	UER	0.646	[0.567, 0.716]	0.690	0.703
mu-bench	LASER	0.556	[0.475, 0.634]	0.514	0.825
mu-bench	WER >0	0.533	[0.453, 0.613]	0.462	0.954

Table 3: Cohen’s

\kappa

against human annotators on the binary label significant error vs. not, sorted by

\kappa

. Norm = normalization; blind LLM is the same gpt-6-astra normalizer given one transcript at a time (§ 5.5 ). WER >0 flags any difference after normalization. LASER is the published rubric of [ 15 ] run verbatim with the same gpt-6-astra judge, flagging any major-penalty error.

Normalizer varied
		UER $\boldsymbol{\kappa}\,\uparrow$
Model	$/1k	Overall	en-US	es-MX	tr-TR	vi-VN	zh-CN	Prec	Rec	Flip (%) $\downarrow$
gpt-5.6-luna	0.22	0.755	0.745	0.687	0.698	0.840	0.758	0.881	0.710	1.0
gpt-5.6-terra	1.62	0.704	0.699	0.647	0.598	0.830	0.701	0.835	0.666	1.4
gpt-5.6-sol	3.08	0.751	0.743	0.667	0.709	0.840	0.754	0.851	0.726	0.5
gpt-6-astra	7.39	0.775	0.721	0.654	0.717	0.840	0.839	0.861	0.757	0.0

Table 4: Effect of model size on the normalizer and the judge, varied one at a time; mean over five seeds (sd across seeds at most 0.011 for Overall and 0.048 for any locale cell). Flip is the share of rows whose UER verdict differs between seeds of the varied component (the other is fixed at seed 7, hence two different gpt-6-astra rows). Prec = precision, Rec = recall, $/1k = list price per 1,000 rows.

Provider	Mode	en-US	es-MX	tr-TR	vi-VN	zh-CN	all	IC	p50	p95
		UER (%) $\downarrow$							Latency (ms) $\downarrow$
Google Chirp-3	batch	3.2	7.3	9.5	6.7	32.9	11.9	17.7	734	1136
Google Chirp-3	stream	5.8	11.6	14.2	12.0	38.0	16.3	24.0	646	887
Microsoft Azure Speech	batch	2.8	11.2	17.5	22.1	31.2	17.0	20.1	315	1030
ElevenLabs Scribe v2	batch	4.8	10.5	12.3	24.2	34.8	17.3	16.6	415	847
Microsoft Azure Speech	stream	3.1	15.7	16.3	18.8	41.3	19.0	28.9	242	431

Table 5: Results on mu-bench, sorted by UER (%). IC = UER on the 1,706 input-collection utterances (§ 5.4 ), macro-averaged over locales like all .

		Raw		Whisper-normalized
Class	Count	Reference	Prediction	Reference	Prediction
Digits vs. words	1,496	C N 6 5 3 。	CN 六五三	c n 6 5 3	cn 六五三
Contact format	1,141	Ashley dot Brown at email dot com.	[email protected]	ashley dot brown at email dot com	ashley brown email com
Grouping	1,058	55 12 34 56 7 8.	55 12 34 56 78	55 12 34 56 7 8	55 12 34 56 78
Fillers (non-en)	841	Sí.	Este, sí.	sí	este sí
Word variant	543	Okay.	OK.	okay	ok

Table 6: Failure classes of rule-based normalization on semantically equivalent outputs: the pair as transcribed (Raw) and after Whisper normalization, which still leaves it distinct; mu-bench maps the prediction onto the raw reference. Count is the number of (utterance, provider) pairs only mu-bench reconciles.

Provider	1s	3s	1s	3s	1s	3s
	Changed (%)		$\boldsymbol{\Delta}$ UER (pp)		$\boldsymbol{\Delta}$ Latency (ms)
Deepgram Nova-3	20.6	21.5	$-0.1$	$-0.8$	$+1$	$+3$
Azure Speech	29.4	32.4	$+1.5$	$+2.1$	$+7$	$+21$
ElevenLabs Scribe	28.5	28.2	$+0.5$	$-0.3$	$+6$	$+12$
OpenAI 4o Mini	36.6	37.5	$-0.6$	$+0.7$	$+7$	$+30$
Google Chirp-3	18.4	20.1	$+0.2$	$+0.2$	$+18$	$+57$

Table 7: Effects of prepending silence. “Changed” is the share of predictions that differ from the unpadded request. Grok was not run: we had no API access at the time.

Conversational Speech Recognition

See all 9 papers

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages
Sep 21, 2026Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Sierra · Princeton University · Mercor
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
Sep 21, 2026Chenxu Xiong, Dongming Shen, Yuzhi Tang +3Audio-Language Model Evaluation Voice Agent Evaluation
Boson AI Santa Clara, CA 95054

Sep 21, 2026J/K move · Enter open · S save

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages

Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela, Karthik Narasimhan+2

Sierra · Princeton University · Mercor

Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems Speech Generation Evaluation

Open paper AskPDF arXiv

English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce

τ

-Multilingual, extending

τ

-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.

Abstract

Conversational Speech Recognition

Explore similar work

τττ-Multilingual: Benchmarking Voice Agents Across Languages

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages