Despite recent advancements in ASR, identifying effective architectures, fine-tuning approaches, and data adaptation strategies for low-resource languages remains challenging. We present a systematic comparison across five low-resource African languages and Swahili as a mid-resource contrast, fine-tuning three models from distinct architectures: Phi-4-multimodal-instruct, Whisper-large-v3-turbo, and MMS-1B-all. We analyze data sufficiency, monolingual versus multilingual adaptation, domain fine-tuning, and language-adapter transfer. Phi-4 achieves the best overall accuracy, MMS shows comparable CER, while Whisper exhibits high CER, consistent with known hallucinations and repetition loops. Performance saturates at different data scales, offering practical data-efficiency targets, while typological complexity consistently shapes error rates. Domain fine-tuning benefits Phi-4 and Whisper but degrades MMS, revealing limitations of adapter-based adaptation. Proxy language adapters outperform training MMS adapters from scratch.
Figures & tables
Language Family
Lang Code
Hours
Nilo-Saharan
luo
513.2
kln
386.7
mas
235.5
Niger-Congo
kik
967.8
swa
280.0
Afro-Asiatic
som
341.9
Table 1: Language families, ISO 639-3 codes, and available speech hours from the listed corpora
Model
Speech Params
Languages Supported
Architecture
Phi-4
920 M 5 5 5 For Phi-4, the speech-specific parameters comprise the audio encoder and modality projector (460M) together with the speech LoRA (460M) ( Agrawal and others, 2025 ) , while the underlying 5.6B language model is shared across modalities.
8
LLM multimodal
Whisper
809 M
99
Encoder–decoder
MMS
1.0 B
1,162
Encoder-only
Table 2: Overview of the three ASR models evaluated in this study.
Figure 1: CER across six languages for Phi-4, MMS, and Whisper fine-tuned on 200 h of monolingual data. Corresponding WER is reported in Figure 8 (Appendix A ).
Figure 2: CER as a function of training data (hours) across six languages. Performance largely plateaus after 150 h.
Figure 3: Monolingual vs. multilingual CER at 200 h per language for Phi-4 (left) and Whisper (right).
Figure 4: Domain robustness (CER) for Kikuyu with and without in-domain agricultural fine-tuning.
Figure 5: MMS adapter strategies (CER): training new language-specific adapters versus reusing related-language proxy adapters for Kalenjin and Maasai. Corresponding WER is reported in Figure 7 (Appendix A ).
Figure 6: CER by language grouped by family (Niger-Congo, Nilo-Saharan, Afro-Asiatic) across all three models at 200 h, showing the correlation between typological complexity and error rate. Corresponding WER is reported in Figure 9 (Appendix A ).
Model
VRAM
Train Time (200 h)
(GB)
Multi
Mono
Whisper
29.1
79
26
Phi-4
71.2
53
6
MMS
26.6
N/A
15
Table 3: Peak GPU VRAM usage and training time per model for baseline fine-tuning (hours)
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: MMS adapter strategies (WER): training new language-specific adapters versus reusing related-language proxy adapters for Kalenjin and Maasai (companion to Figure 5 ).
Figure 8: WER across six languages for Phi-4, MMS, and Whisper fine-tuned on 200 h of monolingual data (companion to Figure 1 ).
Figure 9: WER by language grouped by family across all three models at 200 h (companion to Figure 6 ).
Model
Lang.
CER 150h
CER 450h
Δ CER
MMS
kik
0.18
0.17
− 0.01
kln
0.31
0.30
− 0.01
luo
0.19
0.18
− 0.01
mas
0.36
0.36
− 0.00
som
0.37
0.36
− 0.01
swh
0.26
0.25
− 0.01
Appendix
Table 4: Data sufficiency analysis comparing CER at 150 h and the final checkpoint (450 h) for MMS and Phi-4 across six languages. Paired significance tests are conducted per model ( n=6 languages). Negative Δ denotes improvement and positive Δ denotes degradation.
Model
Effective Batch Size
Learning Rate
Weight Decay
Whisper
256
1×10−5
0.01
Phi-4
96
1×10−4
0.005
MMS
128
1×10−3
0.01
Appendix
Table 5: Fine-tuning hyperparameters used in all experiments. All experiments use the AdamW optimizer.
Extending large-scale multilingual automatic speech recognition (ASR) models to low-resource languages remains challenging. Model performance is skewed toward high-resource languages and degrades sharply for languages with limited labeled data and pre-training exposure. To address this, we investigate parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper. Alongside warm initialization and attention-based fusion, we propose Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter. Under controlled experiments, these approaches are evaluated on three target languages unsupported by Whisper -- Asturian, Assamese, and Xhosa -- using source languages with varying degrees of relatedness. Sequential Adapter Stacking with the closest related source consistently and significantly outperforms full fine-tuning across the three targets, with 5--8% relative WER reductions. These gains largely persist with only one hour of target training data.
Thai Thi Thanh Thao Dang, Mengjie Qian, Kate Knill
Department of Engineering, University of Cambridge, UK
We present a phoneme-level analysis of automatic speech recognition (ASR) for two low-resourced and phonologically complex East Caucasian languages, Archi and Rutul, based on curated and standardized speech-transcript resources totaling approximately 50 minutes and 1 hour 20 minutes of audio, respectively. Existing recordings and transcriptions are consolidated and processed into a form suitable for ASR training and evaluation. We evaluate several state-of-the-art audio and audio-language models, including wav2vec2, Whisper, and Qwen2-Audio. For wav2vec2, we introduce a language-specific phoneme vocabulary with heuristic output-layer initialization, which yields consistent improvements and achieves performance comparable to or exceeding Whisper in these extremely low-resource settings. Beyond standard word and character error rates, we conduct a detailed phoneme-level error analysis. We find that phoneme recognition accuracy strongly correlates with training frequency, exhibiting a characteristic sigmoid-shaped learning curve. For Archi, this relationship partially breaks for Whisper, pointing to model-specific generalization effects beyond what is predicted by training frequency. Overall, our results indicate that many errors attributed to phonological complexity are better explained by data scarcity. These findings demonstrate the value of phoneme-level evaluation for understanding ASR behavior in low-resource, typologically complex languages.
V. S. D. S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger
Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA across both model architectures, with statistically significant gains in several evaluation settings, while retaining the same number of trainable parameters. These results show that task-informed subspace construction can be effective for low-resource speech adaptation, and that adaptive rank allocation provides a robust way to improve parameter efficiency without increasing model capacity.
Asmee Mishra, Mengjie Qian, Brechtje Post +1
Department of Engineering, University of Cambridge, United Kingdom · Phonetics Laboratory, University of Cambridge, United Kingdom