Despite recent advancements in ASR, identifying effective architectures, fine-tuning approaches, and data adaptation strategies for low-resource languages remains challenging. We present a systematic comparison across five low-resource African languages and Swahili as a mid-resource contrast, fine-tuning three models from distinct architectures: Phi-4-multimodal-instruct, Whisper-large-v3-turbo, and MMS-1B-all. We analyze data sufficiency, monolingual versus multilingual adaptation, domain fine-tuning, and language-adapter transfer. Phi-4 achieves the best overall accuracy, MMS shows comparable CER, while Whisper exhibits high CER, consistent with known hallucinations and repetition loops. Performance saturates at different data scales, offering practical data-efficiency targets, while typological complexity consistently shapes error rates. Domain fine-tuning benefits Phi-4 and Whisper but degrades MMS, revealing limitations of adapter-based adaptation. Proxy language adapters outperform training MMS adapters from scratch.
Figures & tables
Language Family
Lang Code
Hours
Nilo-Saharan
luo
513.2
kln
386.7
mas
235.5
Niger-Congo
kik
967.8
swa
280.0
Afro-Asiatic
som
341.9
Table 1: Language families, ISO 639-3 codes, and available speech hours from the listed corpora
Model
Speech Params
Languages Supported
Architecture
Phi-4
920 M 5 5 5 For Phi-4, the speech-specific parameters comprise the audio encoder and modality projector (460M) together with the speech LoRA (460M) ( Agrawal and others, 2025 ) , while the underlying 5.6B language model is shared across modalities.
8
LLM multimodal
Whisper
809 M
99
Encoder–decoder
MMS
1.0 B
1,162
Encoder-only
Table 2: Overview of the three ASR models evaluated in this study.
Figure 1: CER across six languages for Phi-4, MMS, and Whisper fine-tuned on 200 h of monolingual data. Corresponding WER is reported in Figure 8 (Appendix A ).
Figure 2: CER as a function of training data (hours) across six languages. Performance largely plateaus after 150 h.
Figure 3: Monolingual vs. multilingual CER at 200 h per language for Phi-4 (left) and Whisper (right).
Figure 4: Domain robustness (CER) for Kikuyu with and without in-domain agricultural fine-tuning.
Figure 5: MMS adapter strategies (CER): training new language-specific adapters versus reusing related-language proxy adapters for Kalenjin and Maasai. Corresponding WER is reported in Figure 7 (Appendix A ).
Figure 6: CER by language grouped by family (Niger-Congo, Nilo-Saharan, Afro-Asiatic) across all three models at 200 h, showing the correlation between typological complexity and error rate. Corresponding WER is reported in Figure 9 (Appendix A ).
Model
VRAM
Train Time (200 h)
(GB)
Multi
Mono
Whisper
29.1
79
26
Phi-4
71.2
53
6
MMS
26.6
N/A
15
Table 3: Peak GPU VRAM usage and training time per model for baseline fine-tuning (hours)
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: MMS adapter strategies (WER): training new language-specific adapters versus reusing related-language proxy adapters for Kalenjin and Maasai (companion to Figure 5 ).
Figure 8: WER across six languages for Phi-4, MMS, and Whisper fine-tuned on 200 h of monolingual data (companion to Figure 1 ).
Figure 9: WER by language grouped by family across all three models at 200 h (companion to Figure 6 ).
Model
Lang.
CER 150h
CER 450h
Δ CER
MMS
kik
0.18
0.17
− 0.01
kln
0.31
0.30
− 0.01
luo
0.19
0.18
− 0.01
mas
0.36
0.36
− 0.00
som
0.37
0.36
− 0.01
swh
0.26
0.25
− 0.01
Appendix
Table 4: Data sufficiency analysis comparing CER at 150 h and the final checkpoint (450 h) for MMS and Phi-4 across six languages. Paired significance tests are conducted per model ( n=6 languages). Negative Δ denotes improvement and positive Δ denotes degradation.
Model
Effective Batch Size
Learning Rate
Weight Decay
Whisper
256
1×10−5
0.01
Phi-4
96
1×10−4
0.005
MMS
128
1×10−3
0.01
Appendix
Table 5: Fine-tuning hyperparameters used in all experiments. All experiments use the AdamW optimizer.