A Speech Corpus for Mizo Automatic Speech Recognition: Whisper and SraVaani 1.0 Fine-Tuning with Morphology-Aware Evaluation

Priyankoo Sarmah; Sanasam Ranbir Singh; Lalhmingmawia

cs.CLAug 19, 2026

A Speech Corpus for Mizo Automatic Speech Recognition: Whisper and SraVaani 1.0 Fine-Tuning with Morphology-Aware Evaluation

Authors: Priyankoo Sarmah, Sanasam Ranbir Singh, Lalhmingmawia

Organizations: Center for Linguistic Science and Technology Indian Institute of Technology Guwahati, India

Abstract

This study reports the development of an Automatic Speech Recognition (ASR) system in Mizo, a low-resource language. The development included collecting 17.62 hours of speech data, curating it, and fine-tuning the Mizo ASR system with three Whisper multilingual models and with the SraVaani 1.0 Indic multilingual model. Whisper-large-v3 achieved the lowest conventional WER (18.08%), while morphology-aware evaluation yielded a WER of 7.22%. Zero-shot evaluation of the SraVaani 1.0 Indic multilingual model yielded a WER of 58.27%, while Mizo-specific fine-tuning reduced the conventional WER to 29.45% and the morphology-aware WER to 17.93%. The results demonstrate that the Whisper model can achieve a substantially low WER, even when adapted to an unseen language. In contrast, SraVaani 1.0 supports the Mizo language in its multilingual model; however, fine-tuning with carefully curated Mizo speech data substantially improves its performance.

Figures & tables

Figure 1: Histogram of duration of speech files in the corpus.

	Speakers	Sentences	Hours
Training	184	7656	16.18
Validation	11	426	1.02
Testing	05	192	0.42

Table 1: Data split for training, validation and testing.

Sentences	8274
Total duration	17.62 hours
Minimum duration	0.63 seconds
Maximum duration	41.22 seconds
Mean duration	7.67 seconds
Median duration	6.94 seconds

Table 2: Durational characteristics of the speech database

Figure 2: Schematic diagram showing the flow of the experimental protocol.

Model	Architecture	Parameters	Pretraining data	Mel bins
Whisper-small	Encoder–Decoder Transformer	244 M	680k h	80
Whisper-medium	Encoder–Decoder Transformer	769 M	680k h	80
Whisper-large-v3	Encoder–Decoder Transformer	1,550 M	$\sim$ 5M h	128
SraVaani 1.0	FastConformer Hybrid RNNT/CTC	$\sim$ 430 M	$\sim$ 31k h	–

Table 3: Characteristics of the multilingual ASR models evaluated in this study.

Model	Optimizer	LR	Effective batch size	Epochs	Selection
Whisper-small	Adafactor	$5\times 10^{-6}$	16	20	Best val. WER
Whisper-medium	Adafactor	$5\times 10^{-6}$	16	20	Best val. WER
Whisper-large-v3	Adafactor	$5\times 10^{-6}$	16	20	Best val. WER
SraVaani 1.0	AdamW	$1\times 10^{-4}$	16	20 + extended	Best val. WER

Table 4: Fine-tuning configurations used for the ASR models.

Model	Best epoch	Validation WER
Whisper-small Mizo–FT	$15$	$28.99$
Whisper-medium Mizo–FT	$13$	$26.51$
Whisper-large-v3 Mizo–FT	$13$	$23.00$
SraVaani 1.0 Mizo–FT	$18$	$33.81$

Table 5: Best epochs and corresponding validation WER of four fine-tuned models.

Model	CER (%)	WER (%)	MA-WER (%)
Whisper-small Mizo–FT	04.83	24.00	11.49
Whisper-medium Mizo–FT	04.02	21.69	08.87
Whisper-large-v3 Mizo–FT	03.26	18.08	07.22
SraVaani 1.0	17.71	58.27	36.27
SraVaani 1.0 Mizo-FT	06.90	29.45	17.93

Table 6: Results of evaluation on five models.

Model	Foreign script	Names	Glottal stops	Numeral transcripts	Code-mix error	$<$ \textsubdot t $>$
Whisper-small Mizo–FT	NIL	17	7	2	9	0
Whisper-medium Mizo–FT	NIL	18	6	5	3	0
Whisper-large-v3 Mizo–FT	4 sentences	8	4	3	2	0
SraVaani 1.0	21 sentences	44	9	19	49	31
SraVaani 1.0 Mizo–FT	NIL	24	6	2	19	4

Table 7: Error distribution by model. Foreign-script outputs are reported as the number of utterances; all other entries indicate the number of error occurrences.

Speech Corpus Construction

See all 24 papers

Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models
Jul 19, 2026Ganapati Das, Dwipen Laskar, Hasin Afzal Ahmed +4Automatic Speech Recognition Low-Resource Speech Recognition
Department of Computer Science, Gauhati University, Guwahati, 781014, Assam, India · Department of Computer Science, Pandit Deendayal Upadhyaya Adarsha Mahavidyalaya, Goalpara, 783101, Assam, India · Department of Computer Science and Technology, Bodoland University, Kokrajhar, 783370, Assam, India +1
Fine-Tuning Whisper for Automatic Speech Recognition in Baniwa: A Preliminary Study
Aug 26, 2026Leonardo Duart, Tiago Fonseca, Thiago ChacónAutomatic Speech Recognition Low-Resource Speech Recognition
Department of Statistics, University of Brasilia, Brasilia, Brazil.
Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition
May 13, 2026Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon +2ASR Evaluation Speech Recognition Benchmarks
Adalat AI, India

Jul 19, 2026J/K move · Enter open · S save

Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models

Ganapati Das, Dwipen Laskar, Hasin Afzal Ahmed, Sanjib Kr Kalita+3

Department of Computer Science, Gauhati University, Guwahati, 781014, Assam, India · Department of Computer Science, Pandit Deendayal Upadhyaya Adarsha Mahavidyalaya, Goalpara, 783101, Assam, India · Department of Computer Science and Technology, Bodoland University, Kokrajhar, 783370, Assam, India+1

Automatic Speech Recognition Low-Resource Speech Recognition

Open paper AskPDF arXiv

Developing Automatic Speech Recognition (ASR) for morphologically rich, low-resource languages such as Assamese is challenging due to insufficient annotated speech data. The pretrained Whisper model performs poorly on Assamese speech recognition tasks. This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus. A hardware-aware optimized training pipeline is implemented for resource-constrained environments, employing mixed-precision training and gradient accumulation on Tesla 4 Graphics Processing Units (T4 GPUs). The proposed fine-tuned model significantly outperformed the Zero-shot baseline, yielding Word Error Rate (WER), Character Error Rate (CER), Match Error Rate (MER), and Word Infomation Loss (WIL) of 43.17%, 13.18%, 43%, and 64.81%, respectively, achieving significant relative improvements of 78.26%, 93.10%, 57.0%, and 35.19% over the baseline. Semantic evaluation of the fine-tuned model also demonstrates notable improvement over a zero baseline, attaining Bilingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit ORdering (METEOR) scores of 30.81 and 0.5262, respectively. Additionally, the predicted hallucination rate and Real-Time Factor (RTF) are substantially improved by 96.70% and 32.38%, compared to the zero-shot baseline.

Abstract

Speech Corpus Construction

Explore similar work

Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models

Fine-Tuning Whisper for Automatic Speech Recognition in Baniwa: A Preliminary Study

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition