BlasBench: An Open Benchmark for Irish Speech Recognition

Jyoutir Raj; John Conway

cs.CLApr 12, 2026

BlasBench: An Open Benchmark for Irish Speech Recognition

Authors: Jyoutir Raj, John Conway

Organizations: Independent Researcher

Abstract

Existing multilingual benchmarks include Irish among dozens of languages but apply no Irish-aware text normalisation, leaving reliable and reproducible ASR comparison impossible. We introduce BlasBench, an open evaluation harness that provides a standalone Irish-aware normaliser preserving fadas, lenition, and eclipsis; a reproducible scoring harness and per-utterance predictions released for all evaluated runs. We pilot this by benchmarking 12 systems across four architecture families on Common Voice ga-IE and FLEURS ga-IE. All Whisper variants exceed 100% WER through insertion-driven hallucination. Microsoft Azure reaches 22.2% WER on Common Voice and 57.5% on FLEURS; the best open model, Omnilingual ASR 7B, reaches 30.65% and 39.09% respectively. Models fine-tuned on Common Voice degrade 33-43 points moving to FLEURS, while massively multilingual models degrade only 7-10 - a generalisation gap that single-dataset evaluation misses.

Figures & tables

Panel of Figure 1 — Figure 1: BlasBench evaluation pipeline. The Irish normaliser (highlighted) is the language-specific component.

#	model	type	arch	wer $\downarrow$	sub	ins	del	cer $\downarrow$
ref	ABAIR / Fotheidil	closed	TDNN-HMM	19.6*	—	—	—	—
1	azure / speech-ga-IE	API	proprietary	22.2	15.8	1.7	4.8	11.4
2	omniASR LLM 7B	open-w	w2v2+Trans	30.6	25.0	2.5	3.2	14.6
3	Aditya3107 / xls-r-1b	open	w2v2 CTC	32.4	26.4	1.7	4.3	12.8
4	omniASR LLM 300M	open-w	w2v2+Trans	37.6	29.3	3.4	4.9	19.2
5	kingabzpro / xls-r-1b	open	w2v2 CTC	45.8	38.2	3.6	4.0	18.9

Table 1: Common Voice ga-IE (874 utterances). ABAIR is self-reported ( Lonergan et al., 2025 ) using a different normaliser. Whisper results are zero-shot. Type column: open = weights and training data public; open-w = weights public, training data not fully disclosed; API = commercial cloud API; closed = weights unavailable.

#	model	type	arch	wer $\downarrow$	sub	ins	del	cer $\downarrow$
ref	ABAIR / Fotheidil	closed	TDNN-HMM	44.5*	—	—	—	—
1	omniASR LLM 7B	open-w	w2v2+Trans	39.1	32.2	3.4	3.5	18.6
2	omniASR LLM 300M	open-w	w2v2+Trans	47.7	38.4	4.9	4.4	24.1
3	azure / speech-ga-IE	API	proprietary	57.5	21.5	3.5	32.5	43.8
4	mms-1b-all	open	w2v2 CTC	61.6	51.9	3.2	6.5	26.0
5	Aditya3107 / xls-r-1b	open	w2v2 CTC	75.8	62.1	6.2	7.5	36.0

Table 2: FLEURS ga-IE (842 utterances). ABAIR’s 44.5% is on FLEURS-R ( Ma et al., 2024 ) , not this test set.

model	cv	fleurs	$\Delta$
mms-1b-all	54.2	61.6	+7.3
omniASR 7B	30.6	39.1	+8.4
omniASR 300M	37.6	47.7	+10.1
azure	22.2	57.5	+35.2
cpierse	49.4	83.2	+33.8
Aditya3107	32.4	75.8	+43.4

Table 3: Cross-corpus generalisation gap.

\Delta

= FLEURS

-

CV WER. Multilingual models generalise; CV-trained models do not.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Common Voice
model	wer	s	i	d
azure ga-IE	22.3	15.8	1.7	4.8
omniASR 7B	30.7	25.0	2.5	3.2
Aditya3107 xls-r-1b	32.4	26.4	1.7	4.3
omniASR 300M	37.6	29.3	3.4	4.9
kingabzpro xls-r-1b	45.8	38.2	3.6	4.0

Appendix

Table 4: Error-type breakdown. S, I, D reported as percentages of reference words, aggregated globally. Sorted by WER within each dataset.

ASR Evaluation

Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.

See all 144 papers

Benchmarking Automatic Speech Recognition Tools for Iberian Languages
Sep 29, 2026Fernando López, Pablo Gómez, David Solans +2ASR Evaluation Automatic Speech Recognition
Telefónica Innovación Digital, Spain · Universidad Autónoma de Madrid, Spain
OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics
Jun 19, 2026Korbinian Kuhn, Gottfried ZimmermannASR Evaluation Low-Resource Language Processing
Stuttgart Media University, Germany · University of Tübingen, Germany
Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
Mar 1, 2026Kaushal Santosh Bhogale, Tahir Javed, Greeshma Susan John +4ASR Evaluation Automatic Speech Recognition
AI4Bharat, WSAI, Indian Institute of Technology Madras, India · Sarvam AI, India

Sep 29, 2026J/K move · Enter open · S save

Benchmarking Automatic Speech Recognition Tools for Iberian Languages

Fernando López, Pablo Gómez, David Solans, Paulo Villegas+1

Telefónica Innovación Digital, Spain · Universidad Autónoma de Madrid, Spain

ASR Evaluation Automatic Speech Recognition Low-Resource Speech Recognition Multilingual ASR

Open paper AskPDF arXiv

Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficiency trade-offs are underexplored. We benchmark eleven systems, ten open-weight models and one commercial API, across five Iberian languages (Basque, Catalan, Galician, Portuguese, Spanish), with German and Turkish as controls. Evaluation uses an 85-hour dataset covering read speech, broadcast media, and audiobooks, assessing accuracy and efficiency via word error rate (WER) and real-time factors (RTF/RTFx). Results show no single model dominates: accuracy, efficiency, and language coverage present clear trade-offs. Low-resource languages, especially Basque, degrade significantly, highlighting the role of training coverage. We observe consistent sex disparities across most systems, highlighting fairness challenges in multilingual ASR. Overall, the benchmark provides practical guidance for real-world model selection.

Abstract

ASR Evaluation

Explore similar work

Benchmarking Automatic Speech Recognition Tools for Iberian Languages

OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages