From Statistical Methods to Pre-Trained Models; A Survey on Automatic Speech Recognition for Resource Scarce Urdu Language
Authors: Muhammad Sharif, Zeeshan Abbas, Jiangyan Yi, Chenglin Liu
Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation Chinese Academy of Sciences, Beijing 100190, P.R. China · University of Chinese Academy of Sciences, Beijing China · School of Electronics and Information Engineering, Shenzhen University · School of Artificial Intelligence, CAS Institute of Automation
Automatic Speech Recognition (ASR) technology has witnessed significant advancements in recent years, revolutionizing human-computer interactions. While major languages have benefited from these developments, lesser-resourced languages like Urdu face unique challenges. This paper provides an extensive exploration of the dynamic landscape of ASR research, focusing particularly on the resource-constrained Urdu language, which is widely spoken across South Asian nations. It outlines current research trends, technological advancements, and potential directions for future studies in Urdu ASR, aiming to pave the way for forthcoming researchers interested in this domain. By leveraging contemporary technologies, analyzing existing datasets, and evaluating effective algorithms and tools, the paper seeks to shed light on the unique challenges and opportunities associated with Urdu language processing and its integration into the broader field of speech research.
Figures & tables
Figure 1: Main modules and information flow of a conventional ASR system
S. No
Tool(s)
Technology
Accuracy WER
Ref.
1
Scikit, Keras
SVM, CNN
0.97
Chandio et al. (2021)
2
libSVM
SVM, RF, LDA
73%
Ali et al. (2015b)
3
Sphinx, CLTK
HMM
74%
Shaukat et al. (2016)
4
Sphinx
SVM, GMM
92.4%
Qasim et al. (2016b)
5
Kaldi, SRILM
SGMM, MMI
24.2
Raza et al. (2018)
6
Sphinx4
HMM
10.66
Ashraf et al. (2010)
Table 1: Software tools utilized for Urdu ASR
Figure 2: Structured process used to search, screen, classify, and synthesize Urdu ASR studies.
Figure 3: Flow of the Study
S. No
Dataset
Size (utts./hrs)
Vocab
Speech Type
Public
Ref.
1
Urdu Spoken Words
5200 utts.
52
Isolated
No
Ashraf et al. (2010)
2
AUDD
25,518 utts.
-
isolated
No
Chandio et al. (2021)
3
Isolated Urdu Corpus
250 Words
2500
Isolated
Yes
Ali et al. (2016)
4
PRUS Corpus
70 Minutes
42,289
Read Speech
Yes
Raza et al. (2010)
5
District Names DS
9.5 hours
41293
Spontaneous, isolated
Yes
Qasim et al. (2016a)
6
Urdu Mixed Datasets
126.8 hours
-
Read Speech
Partially
Naeem et al. (2020)
Table 2: Urdu speech recognition systems and datasets
Figure 4: Gender and Age of Participants
S. No
Technique(s)
%WER/ Accuracy
Ref.
1
SGMMI
16.7
Naeem et al. (2020)
2
SVM, CNN
0.97
Chandio et al. (2021)
3
GMM, CMLLR
32.6
Shaik et al. (2015)
4
SVM, RF, LDA
73%
Ali et al. (2015a)
5
HMM
74%
Shaukat et al. (2016)
6
SVM, GMM
92.4%
Qasim et al. (2016b)
Table 3: Statistical Techniques and Results Summary
Figure 5: General architecture of end-to-end ASR system
S. No
A.M Technique(s)
Language Model LM
WER (%) Urdu
Avg WER (%) (Multi Lingual)
Ref.
1
SSL, IndicWav2Vec2, XLS-R
KenLM
12.6
12.4
Javed et al. (2022)
2
Wav2Vec, Conformer, CTC, OpenSLR
KenLM
-
15.3
Bhogale et al. (2023b)
3
IndicWav2Vec
-
25.1
21.0
Bhogale et al. (2023a)
IndicWhisper, etc.
19.4
13.6
Google SST
23.3
23.9
Azure SST
-
20.0
Table 4: Multilingual ASR Systems with the Urdu Language
Figure 6: Wav2Vec2.0 paradigm adopted for Urdu ASR Mohiuddin et al. (2023)
Figure 7: DNN based approaches for Urdu ASR Zia and Zahid (2019) ; Khan et al. (2021) ; Javed et al. (2022) ; Ali Humayun et al. (2019) ; Farooq et al. (2020) ; Farooq et al. (2019)
Ref.
Method
Objective
Accomplishments
Limitation
Javed et al. (2022)
IndicSUPERB Wav2Vec2-based multimodal ASR
• Larger dataset, 1684 hours, for 12 Indic languages including Urdu. • Benchmarks for speech tasks including ASR. • Train and evaluate self-supervised models along with FBANK.
• Able to reduce average WER for average and specific language. • Able to boost the accuracy with a Jointly fine-tuned multilingual model
• Limited to 12 Indic languages. • Test results with unknown speakers are different from those with known speakers. • Not robust in noisy environment
Bhogale et al. (2023b)
Shrutilipi dataset Wav2vec2-based multimodal ASR
• Proposed mining text and audio pairs of low-resource Indic languages, Shrutilipi 6400 hours, as an alternative to creating labeled datasets. • Adopt CTC with the Needleman-Wunsch algorithm.
• Able to reduce WER by 5.8% for 7 languages, can further with the Conformer model. • High quality and diverse dataset effective for ASR down streaming. • Able to Train models which are more robust to noisy input.
• Limited to 12 Indic languages • Smallest proportion of Urdu. • Unable to show language-specific accuracy
Bhogale et al. (2023a)
Vistaar IndicWhisper
• A set of 59 benchmarks collated from ASR systems named Vistaar. • trained for 10.7K hours long 12 languages.
• Diversity in every language. • Reduce an average of 4% WER. • High volume Urdu speech data 320 hours with 13.6% WER.
• More datasets are possible to compete with high-resource languages and multi-lingual benchmarks.
Radford et al. (2023)
Whisper: Wav2Vec and Seq-to-seq architecture
• Multilingual and multi-task 680,000 hours labeled datasets. • Whisper Model: Robust speech processing with pre-trained Models based on Seq-to-seq transformer architecture.
• Capable of getting an Average 29.3% WER on wav2vec without the language model, and 12.8% WER on the Whisper model. • Whisper model robust enough for long-form (few minutes to few hours) transcriptions.
• Low accuracy for low-resource languages. • Highest proportion of English • Lacks fine-tuning • Strong decoder but no CTC or wav2vec together with a language model.
Conneau et al. (2023)
FLEURS Few-shot learning, mSLAM, and wav2vec-Bert Models
• Consists of 102 languages with a 12-hour dataset per language including Urdu. • Attempt to catalyze resource-scarce languages in speech tech technology.
• Simple, baselines for the ASR tasks on pre-trained mSLAM and wav2vec-Bert models. • Considerable %CER specifically in Urdu with %CER reductions of 82.9 and 83.1 respectively.
• Limited datasets • No Language model
Table 5: Comparison of multilingual ASR system with Urdu language
The rapid advancement of Artificial Intelligence (AI) and Natural Language Processing (NLP) has revolutionized the way humans interact with machines. Among the most impactful developments is Automatic Speech Recognition (ASR), which enables computers to convert spoken language into text. Systems such as those built on deep neural networks, transformer architectures, and self-supervised learning have achieved near-human performance for well-resourced languages such as English and French. Yet, these advances have disproportionately benefited a small fraction of the world's languages.
Sophie Gertrude Ngo Mock, Charles Moudina Varmantchaonala, Paul Dayang +2
Department of Mathematics and Computer Science, Faculty of Science, University of Ngaoundere, Ngaoundere, 454, Cameroon · Institute for Physics, Faculty V, Carl von Ossietzky University of Oldenburg, 26129 Oldenburg, Germany
Automatic speech recognition (ASR) for low-resource languages remains a major challenge. Sinhala, the primary language of Sri Lanka with about 16 million speakers, illustrates the difficulty: agglutinative morphology, a 54-phoneme inventory, subject-object-verb (SOV) syntax and scarce annotated speech data limit both conventional and modern ASR systems. This paper presents the first critical review of Sinhala ASR research, tracing its development from Hidden Markov Models (HMMs) through deep neural networks to self-supervised pre-trained models such as wav2vec 2.0, XLS-R, Whisper and Massively Multilingual Speech (MMS). We compare existing Sinhala systems with related low-resource ASR work on Tamil, Malayalam and Hindi in terms of architecture, training data, word error rate (WER) and robustness to real-world acoustic conditions, and we assess self-supervised and transfer learning as responses to scarce labeled data. We show that most reported WERs are not directly comparable because they differ in corpus, data split and scoring, and that the only controlled comparison in the literature attributes an 18.1% relative WER reduction to corpus correction alone. We also discuss context-aware ASR that draws on phonological, syntactic and semantic knowledge. We identify six research gaps: (1) the lack of large annotated corpora covering multiple dialects and acoustic conditions; (2) weak contextual modeling of Sinhala morphosyntax; (3) high WER in real-world conditions; (4) the absence of standardized benchmarks; (5) the lack of parameter-efficient fine-tuning studies; and (6) the absence of annotated code-switched Sinhala-English speech resources. We outline a research agenda to address these gaps, intended as a roadmap for researchers working on Sinhala and other morphologically rich languages.
MSc in Data Science and Artificial Intelligence Department of Computer Science & Engineering University of Moratuwa, Sri Lanka · University of Moratuwa, Sri Lanka
Telephony conversations worldwide are conducted over narrow-band channels and are often spontaneous and colloquial in nature. This paper evaluates the performance of widely used foundational automatic speech recognition (ASR) models -- both open-source and commercial -- on narrow-band conversations in Hindi, a low-resource language, and Indian-accented English, a low-resource accent. We first assess these models in a zero-shot setting and find that their performance remains suboptimal across the board. Highlighting the challenges faced by ASR models in narrow-band and low-resource language scenarios, we further investigate the impact of fine-tuning open-source models using a limited set of real-life annotated recordings. Our findings indicate that while fine-tuning provides some improvements, its effectiveness varies across languages and accents, largely influenced by the amount of data encountered during pretraining