Back to Basics: Revisiting ASR in the Age of Voice Agents

Geeyang Tay; Wentao Ma; Jaewon Lee; Yuzhi Tang; Daniel Lee; Weisu Yin; Dongming Shen; Silin Meng

cs.AIMar 26, 2026

Back to Basics: Revisiting ASR in the Age of Voice Agents

Authors: Geeyang Tay, Wentao Ma, Jaewon Lee, Yuzhi Tang, Daniel Lee, Weisu Yin, Dongming Shen, Silin Meng,

Organizations: Boson AI

Abstract

Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.

Figures & tables

Figure 1: Multilingual ASR robustness under real-world distribution shifts in WildASR. We evaluate seven ASR systems across four languages and aggregate performance over three OOD dimensions. The horizontal line denotes the in-distribution clean-set model-average reference (5.7%), defined as the average error rate on the FLEURS test set across all models and languages. The sharp and uneven degradation across OOD conditions shows that human-parity performance on in-distribution data does not reliably transfer to real-world settings.

Categories	Languages	#Samples	Avg Duration (s)	Curation Steps
Environmental degradation
Reverberation	EN/ZH/JA/KO	2841/3735/2850/2046	10.0/12.9/12.5/10.4	DC $\to$ QF $\to$ NR $\to$ AA $\to$ MV
Far-field	EN/ZH/JA/KO	2841/3735/2850/2046	10.0/13.0/12.5/10.4	DC $\to$ QF $\to$ NR $\to$ AA $\to$ MV
Phone codec	EN/ZH/JA/KO	1894/2490/1900/1364	7.5/10.4/10.0/7.9	DC $\to$ QF $\to$ NR $\to$ AA $\to$ MV
Noise gap	EN/ZH/JA/KO	3788/4980/3800/2728	8.6/11.6/11.2/9.1	DC $\to$ QF $\to$ NR $\to$ AA $\to$ MV
Clipping	EN/ZH/JA/KO	947/1245/950/682	7.5/10.4/10.0/7.9	DC $\to$ QF $\to$ NR $\to$ AA $\to$ MV

Table 1: Overview of the proposed WildASR. Each OOD dimension is decomposed into explicitly defined subcategories. For each subcategory, we report the covered languages, the number of samples per language, the average utterance duration, and the curation steps applied (defined in § 3.1 ). Detailed data sources are listed in Appendix D .

Figure 2: Error heatmap for seven ASR models on WildASR. Each cell visualizes error rate (WER for EN and CER for CJK), with lighter colors indicating lower error. This patchy landscape reveals that ASR systems still exhibit large performance degradation and uneven robustness gaps.

Perturbations	EN		ZH		JA		KO
Perturbations	WER (%)	$\Delta$	CER (%)	$\Delta$	CER (%)	$\Delta$	CER (%)	$\Delta$
Original	19.9/4.1	–/–	14.6/7.8	–/–	19.7/5.1	–/–	19.5/5.9	–/–
Reverberation	31.9/9.5	+12.0/+5.3	25.7/13.0	+11.1/+5.2	45.3/15.5	+25.5/ +10.4	46.6/15.5	+27.0/+9.6
Far-field	26.0/15.8	+6.1/ +11.7	23.1/12.3	+8.5/+4.5	33.7/13.5	+13.9/+8.4	40.1/19.1	+20.6/ +13.2
Phone (G.711)	20.5/10.4	+0.6/+6.3	16.9/8.6	+2.3/+0.8	29.1/6.7	+9.4/+1.6	24.8/8.8	+5.3/+2.9
Phone (GSM)	22.9/5.2	+3.0/+1.1	25.0/9.4	+10.4/+1.6	33.8/10.5	+14.1/+5.4	47.6/7.9	+28.0/+2.0

Table 2: Impact of environmental degradations on multilingual ASR performance. Average error rates across seven ASR models under controlled acoustic perturbations. Results are reported as MagicData / FLEURS.

\Delta

denotes the absolute increase in error rates relative to the clean condition. Bold highlights the largest degradation magnitude per language and dataset.

Panel of Figure 3 — Figure 3: ASR error dynamics under increasing reverberation for Qwen2-Audio on FLEURS (top: English, bottom: Chinese).

	Accent		Children		Older
Model	ZH	EN	ZH	EN	ZH	EN
Nova 2	59.2	6.6	54.4	27.4	51.6	2.9
GPT-4o Transcribe	40.7	2.6	39.9	29.4	36.0	1.1
Gemini 2.5 Pro	49.9	5.0	58.6	25.1	52.6	1.8
Gemini 3 Pro	62.5	3.0	55.3	18.2	41.4	0.7
Qwen2-Audio	7.5	6.8	23.4	26.7	18.6	1.5

Table 3: ASR performance under demographic shift. English remains relatively robust, while Chinese and child speech exhibit substantially higher error rates.

Figure 4: Prompt sensitivity of Gemini 2.5 Pro on demographic subsets across ten paraphrased prompts (EN/ZH).

		WER/CER/MER (%)				HER (%)
Model	Category	ZH	EN	JA	KO	ZH	EN	JA	KO
Nova 2	code-switch	33.7	-	32.0	56.4	68.4	-	58.1	71.9
	short	57.6	43.2	56.8	65.3	52.6	36.3	46.9	59.6
	incomplete	35.0	13.5	37.2	61.0	38.1	7.8	37.4	56.8
Gemini 2.5 Pro	code-switch	20.7	-	9.8	18.2	7.0	-	9.4	19.1
	short	40.6	64.4	48.6	55.6	30.5	35.4	28.1	34.1

Table 4: ASR performance and hallucination behavior under linguistic diversity. We can see that short and truncated inputs induce high error and frequent hallucinations, revealing semantic failures not captured by lexical metrics alone (EN not applicable for code-switching).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Inference setting	Value
Temperature	0.2
Top- $p$	0.9
Max new tokens	2048
Language conditioning	Prompt ( {language_name} ) and/or SDK language code

Appendix

Table 5: Unified inference setting for ASR benchmarking. Audio inputs are resampled to 16kHz. The default instruction is: ‘Please transcribe the audio in {language_name}. Do not add any additional text that is not in the speech content.’

Panel of Figure 5 — Figure 5: Accent Distribution in WildASR. Left: English. Right: Chinese.

Subcategory	Sources
Environmental degradation	FLEURS, MagicData
Children	Zenodo, Child_Speech, ChildMandarin
Older adults	GLOBE_V3, seniortalk
Accent	GLOBE_V2, KeSpeech
Short utterances	YODAS
Incomplete audio	YODAS

Appendix

Table 6: Source datasets for each WildASR subcategory.

ID	English prompt variant
p01	English transcription only. Return the spoken words as text—nothing else.
p02	Task: produce a verbatim english transcript of the audio. Output transcript text only; omit all other content.
p03	Perform speech-to-text for this audio in english. Respond with plain transcript text only (no labels or commentary).
p04	Please write the english transcript of what you hear. Reply with only the transcription.
p05	Create the most faithful english transcript possible. Output ONLY the transcript (no summaries, notes, or formatting).
p06	In english, transcribe the audio. Provide only the transcript.

Appendix

Table 7: English prompt variants used for prompt-sensitivity evaluation. All prompts share the same intent— transcribe the audio into { english } and output only the transcript —but differ in wording and style.

Environmental degradation
Subset	Ground Truth	Model Prediction
Noise gap	“a car bomb detonated at police headquarters in gaziantep turkey yesterday morning killed two police officers and injured more than twenty other people”	“a car bomb detonated at police headquarters in geyve tepe turkey yesterday morning killing two police officers and injuring more than a hundred other people”
phone codec	“the center of tibetan meditation is the deity yoga through the visualization of various deities the energy channels are cleaned the chakras are activated and the enlightenment consciousness is created”	“ [noise] [sigh] [sigh] [sigh] [sigh] [sigh] [sigh] [sigh] …”
clipping	“ah yeah.”	“ I’m with breast .”
Demographic shift
child	“he was searching for him everywhere”	“ they were shouting for him everywhere”

Appendix

Table 8: Representative English failure cases from WildASR. Rows are grouped by OOD dimension. We highlight the erroneous or safety-relevant portion in bold

Augmentation		ZH		EN		JA		KO
Perturbation	Method	CER (%)	$\Delta$	WER (%)	$\Delta$	CER (%)	$\Delta$	CER (%)	$\Delta$
Original	–	10.1	–	6.0	–	7.0	–	6.5	–
Clipping	–	42.6	+32.5	60.4	+54.4	32.0	+25.0	51.9	+45.4
Far-field	–	25.6	+15.5	71.0	+65.0	19.5	+12.5	45.9	+39.4
Phone	G.711	12.6	+2.5	6.6	+0.6	8.8	+1.8	6.6	+0.1
Phone	GSM	14.5	+4.4	7.9	+1.9	11.6	+4.6	7.7	+1.2

Appendix

Table 9: Nova 2 — Environmental degradation on FLEURS.

\Delta