cs.CLOct 19, 2026

TeleASR-Amharic: A Benchmark for Robust Amharic ASR in Realistic Telecommunication Conditions

Authors: Dagnachew Mekonnen Marilign, Enrique Molero, Abrham Haile, Louis Powell, Israel Abebe Azime

Organizations: HiLCoE School of Computer Science and Technology · GSMA · gheero · Saarland University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The real-world usability of ASR models remains challenging, particularly for low-resource languages, where existing speech resources are often collected in clean studio environments. In this work, we introduce TeleASR-Amharic, a benchmark for evaluating Amharic ASR under various acoustic and telecommunication conditions. We analyze the major degradations observed in real telecommunications recordings and develop a benchmark that covers multiple channel conditions, environmental disturbances, and telecom-domain variations. Our results show that telecommunication degradation can substantially reduce recognition performance, making even high-performing ASR models impractical under severe conditions. For example, the WER of the best-performing model increases from 0.23 to 0.79 under severe telecommunication degradation, corresponding to a relative increase of 243% in recognition errors. We further investigate the effects of different noise and channel distortions and show that scaling training data and applying carefully designed telecommunication-aware augmentation substantially improve robustness. In particular, our best training configuration achieves a 23.6% relative WER reduction under extreme noise conditions. Together with the release of our telecommunications-domain data, these findings provide a reproducible benchmark for measuring progress in Amharic ASR while highlighting open challenges in developing robust speech technologies for real-world deployment.

Figures & tables

Figure 1: Examples of problematic ASR transcriptions in the telecommunications domain. The analysis shows that models frequently miss numeric information and that transcription performance varies substantially across different audio-quality conditions.
Figure 2: The process of creating the ASR robustness benchmark for the telecommunications domain. Each step is discussed in detail in Section 3 .
GenderUtterancesDuration (h)Share %
Female9,96718.6360.8
Male5,99811.9939.2
All15,96530.63 
Table 1: Speech duration statistics by gender for the TeleASR-Amharic dataset.
SourceUtterancesHours
Training data
WaxalNLP (amh)38,022189.0
Waxal auto-labeled (amh)80,321398.1
BDU Amharic ASR32,901101.0
Common Voice (amh)5630.9
Leyu – Gonder dialect11,339112.8
Table 2: Composition of the combined Amharic ASR corpus for our model training. The training set is expanded by combining multiple publicly available Amharic ASR datasets. The validation set is limited to the WAXAL validation split, to which we apply our 0.4% augmentations to provide a compact yet representative evaluation of the conditions covered by the test set.
Figure 3: ASR performance across acoustic and telecommunication conditions. WER (left) and CER (right) are reported for all evaluated models across the average test set and individual degradation conditions, including weak signal, clean audio, ambient noise, distant/low-volume microphone, harsh environments, wideband phone (VoLTE), randomized phone channels, and narrowband 2G with packet loss. Lower values indicate better performance. The results highlight substantial variation in model robustness across realistic telecommunication conditions.
Figure 4: Word Error Rate (WER, lower is better) of our fine-tuned Amharic ASR models compared with the strongest publicly available baseline (Ethio-ASR-amharic) across eight evaluation conditions, ordered from least to most acoustically degraded.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Error analysis platform developed to identify failure patterns in the dataset and inform evaluation gaps.
Figure 6: Evaluation of Generated Augmentations. We consulted telecommunication experts to assess whether the generated augmentations accurately reflected real-world telecommunication noise and channel conditions.
More in this topic

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 459 papers

Explore similar work

CardsList
Sep 28, 2026

Almieyar: A Culturally Grounded Benchmark for Multi-Dialect Arabic Speech Recognition

Arabic speech technology has largely focused on Modern Standard Arabic, leaving the living dialects spoken by hundreds of millions under-served. We introduce ALMIEYAR, a culturally grounded ASR benchmark covering 17 Arabic dialects across six families, built entirely from newly recorded speech unseen by existing models. Dialect-community coordinators selected culturally relevant images across 10 topics, and native speakers described them through five structured scenarios, yielding approximately 50 minutes per dialect (13.7 hours total). We benchmark 12 state-of-the-art ASR systems zero-shot, including GPT-4o-transcribe, Voxtral-Mini-4B, Fanar-STT-LF, Whisper, SeamlessM4T-v2, and wav2vec2-based models. GPT-4o-transcribe achieves the lowest overall WER at 35.0%, followed by Voxtral-Mini-4B, Fanar-STT-LF, and Whisper-Large-v3 at 41.1%, 45.9%, and 49.5%, respectively, indicating substantial remaining errors across Arabic dialect communities. Performance varies considerably across dialect groups, with no model performing uniformly best across all groups. WER alone also obscures dialectal ASR behaviour: wav2vec2-based models show large WER/CER gaps, where character-level agreement remains much higher than word-level accuracy, motivating joint WER/CER reporting. ALMIEYAR provides a unified benchmark for culturally grounded Arabic ASR evaluation, including the first published benchmark for Ahwazi Arabic.

Omid Ghahroodi, Anas Madkoor, Dima Faris Al Saudi +37

QCRI, HBKU · Qatar University · UDST +15

ASR EvaluationArabic NLPAutomatic Speech RecognitionMultilingual ASRSpeech Recognition BenchmarksArabic Dialects
AskPDF
Mar 26, 2026

Back to Basics: Revisiting ASR in the Age of Voice Agents

Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.

Geeyang Tay, Wentao Ma, Jaewon Lee +8

Boson AI

Robust ASRAutomatic Speech RecognitionASR EvaluationMultilingual ASRDemographic Bias in ASR
AskPDF
Jun 17, 2026

Responsible ASR: Overcoming Challenges of Foundational Models in Narrow-Band and Low-Resource Settings

Telephony conversations worldwide are conducted over narrow-band channels and are often spontaneous and colloquial in nature. This paper evaluates the performance of widely used foundational automatic speech recognition (ASR) models -- both open-source and commercial -- on narrow-band conversations in Hindi, a low-resource language, and Indian-accented English, a low-resource accent. We first assess these models in a zero-shot setting and find that their performance remains suboptimal across the board. Highlighting the challenges faced by ASR models in narrow-band and low-resource language scenarios, we further investigate the impact of fine-tuning open-source models using a limited set of real-life annotated recordings. Our findings indicate that while fine-tuning provides some improvements, its effectiveness varies across languages and accents, largely influenced by the amount of data encountered during pretraining

Tejas Godambe, Nutan Choudhary, Sanket Shah +2

Applied AI, Krutrim, India

ASR EvaluationAutomatic Speech RecognitionLow-Resource Speech Recognition
AskPDF