Nepali speakers regularly mix English words into conversation, but multilingual speech recognisers often struggle with language switches and the writing script used for English words. We study how to adapt SraVaani, a 449M-parameter hybrid TDT-CTC model, using a 172.8-hour corpus covering six Nepali speech domains. Its pretrained tokenizer does not support digits and several common symbols. Rather than replace it or discard these recordings, we generate Nepali and English readings for numbers and use zero-shot ASR output to select a plausible form. Full fine-tuning reduces test word error rate (WER) from 48.6% to 17.1%; a 12-epoch run reaches 16.3%, compared with 16.4% for fine-tuned Whisper large-v3-turbo. Freezing the encoder increases WER to 25.5%, while excluding numeric clips raises WER on numeric test speech from 14.3% to 21.3%. We also evaluate on FLEURS using both its original references and mixed-script alternatives. The comparison shows that transcription conventions explain part, but not all, of the out-of-domain performance loss.
Figures & tables
Figure 1: YouTube speech collection: video selection, audio processing, segmentation and transcript verification.
Subset
Clips
Unenc. %
Digits %
Punct. %
OpenSLR Nepali
72,283
7.5
7.4
0.1
Lectures
20,138
86.3
38.9
47.3
Podcast
7,165
62.7
24.1
38.6
News & talk
14,820
42.2
14.4
27.8
Conversation
997
22.6
8.1
14.4
Interviews
772
93.5
9.2
84.3
Table 1: Source clips whose transcript contains a character the SraVaani tokenizer cannot encode, by cause.
Figure 2: Spelling out numbers, on a real lecture clip. Rules propose a Nepali and an English reading for each number. We romanise the candidates and the base model’s output, which writes English in Devanagari, and keep the closest candidate.
Original
Base-model hypothesis
Final label
1947 \nepalifont मा India \nepalifont मा
\nepalifont उन्नाइस सय सय चालिसमा भारतमा
\nepalifont उन्नाइस सय सतचालीस मा india \nepalifont मा
\nepalifont त्यसलाई 1 \nepalifont सय 50 degree
\nepalifont त्यसलाई एक सय पचास डिग्री
\nepalifont त्यसलाई एक सय पचास degree
\nepalifont अब जस्तै 32 27 \nepalifont होइन 24 24
\nepalifont अब जस्तै बत्तीस सत्ताइस होइन चौबीस चौबीस
\nepalifont अब जस्तै बत्तीस सत्ताइस होइन चौबीस चौबीस
6 into 10 \nepalifont को power
\nepalifont सिक्स इन्टो टेनको पावर
six into ten \nepalifont को power
FXK+1
\nepalifont एफ एक्स के प्लस वान
f x k plus one
T.remove teacher
\nepalifont टी डट रिमुभ टिचर
t dot remove teacher
Table 2: Spelling out numbers with the base model’s output as evidence (excerpts). The last three examples are from engineering lectures.
Train
Validation
Test
Subset
Domain
Clips
h
Clips
h
Clips
h
CM %
EN %
OpenSLR Nepali
read speech
69,133
57.62
1,472
1.22
1,500
1.25
15.1
6.6
Lectures
engineering/CS lectures
15,867
44.53
2,277
6.35
1,574
4.25
98.4
35.9
Podcast
podcast / talk show
6,305
28.47
363
1.63
358
1.63
85.1
13.1
News & talk
news and conversation
12,990
19.61
816
1.29
730
1.12
85.9
26.3
Conversation
code-mixed conversation
570
0.99
48
0.08
373
0.62
95.5
36.2
Table 3: The final corpus by subset and split. CM: share of clips with at least one English word; EN: share of English words among all words.
Train
Val
Test
Source clips
106,347
5,138
4,690
Ambiguous number reading (tie)
519
66
59
Hypothesis too far (CER >0.5 )
287
15
13
Label too long for CTC
93
8
8
Final clips
105,448
5,049
4,610
Final hours
152.9
10.8
9.1
Table 4: Number of clips removed at each preparation step.
Component
Structure
Params
Front end
log-mel, 128 bins (fixed)
–
Subsampling
3 strided conv. stages, 256 ch. + linear
4.3M
Conformer blocks
17× ( d=1024 , 8 heads, kernel 9)
428.6M
Prediction network
embedding + LSTM, 640 units
6.5M
Joint network
1024+640→640→5006
4.3M
CTC head
1024→5001
5.1M
Table 5: SraVaani components and parameter counts, from the released checkpoint. All parameters are fine-tuned in our main configuration.
Figure 3: One fine-tuning step of SraVaani. The two decoders share the encoder and the 5,000-piece vocabulary. All 449M parameters are trained, and the encoder gets half the learning rate of the decoders.
Setting
Official script
Ours
Encoder
frozen
trained, LR ×0.5
Peak LR (decoders)
10−4
10−4
Warm-up
10 updates
5% (494 updates)
Schedule
cosine to 10−6
cosine to 10−6
Clips per update
4×8=32
8×8=64
Epochs
2
6 (and 12)
Table 6: Our fine-tuning settings compared with the official SraVaani fine-tuning script.
Original reference
Mixed-script reference
\nepalifont टेलिभिजन रिपोर्टहरूले प्लान्टबाट सेतो धुवाँ आउँदै गरेको देखाउँछन्
television report \nepalifont हरूले plant \nepalifont बाट सेतो धुवाँ आउँदै गरेको देखाउँछन्
\nepalifont गिजामा रहेको ग्रेट पिरामिड सातवटा आश्चर्यमध्ये एउटा मात्रै हो जुन अहिले पनि उभिरहेको छ
giza \nepalifont मा रहेको great pyramid \nepalifont सातवटा आश्चर्यमध्ये एउटा मात्रै हो जुन अहिले पनि उभिरहेको छ
\nepalifont हट चकलेट बेल्जियमको मानक स्तरमा छ फलका रस महङ्गो भए पनि उत्कृष्ट छ
hot chocolate \nepalifont बेल्जियमको मानक स्तरमा छ फलका रस महङ्गो भए पनि उत्कृष्ट छ
Table 7: FLEURS reference adaptation to the mixed-script convention.
Model
Params
WER
CER
EN-WER
NE-WER
Rom-WER
OpenSLR
Lect.
Podc.
News
Conv.
Interv.
FLEURS
Zero-shot
SraVaani
449M
48.6
33.3
80.4
36.1
41.3
62.5
43.7
47.1
55.7
54.9
62.6
22.6
Whisper large-v3-turbo
809M
121.5
81.6
86.0
135.4
115.4
106.1
113.2
126.9
156.7
110.0
98.3
88.4
IndicConformer
120M
49.1
37.2
100.0
29.3
41.6
16.3
51.5
48.0
53.0
54.4
61.5
22.2
Fine-tuned on our training data
Whisper large-v3-turbo
809M
16.4
7.5
14.8
17.0
14.8
16.8
14.0
20.3
20.9
12.8
11.6
36.9
Table 8: Test results of all models (%). Rom-WER ignores script. FLEURS is out of domain. All fine-tuned models use the same training data; bold marks the best fine-tuned result in each column.
System
CM
Mono
EN-WER
NE-WER
EN exact ↑
Wrong script
Rom-WER
Switch
Int.
Zero-shot
49.6
42.1
80.4
36.1
21.2
11.6
41.3
65.9
35.5
Fine-tuned, 6 epochs
17.4
15.2
20.1
15.9
82.3
0.3
15.5
14.3
15.2
Fine-tuned, 12 epochs
16.6
14.4
18.8
15.3
83.6
0.3
14.7
13.6
14.4
Table 9: Errors by language on the test set (%). CM / Mono: WER on clips with and without English words. Wrong script: English words written in Devanagari. Switch / Int.: error rate next to a language switch and elsewhere. Section 5.4 defines the metrics.
Figure 4: Six-epoch validation WER for the main configuration and ablations. Validation is performed twice per epoch.
Figure 5: Validation WER for six- and twelve-epoch full fine-tuning. Each run uses a cosine schedule matched to its training duration. Validation is performed twice per epoch.
System
WER
Δ [95% CI]
EN-WER
Num.
Other
FLEURS
Main (full fine-tune)
17.1
–
20.1
14.3
18.7
31.1
seed 2
17.0
−0.1 [ −0.3,+0.1 ]
19.7
14.4
18.5
31.7
12 epochs
16.3
−0.8 [ −1.0,−0.6 ]
18.8
13.6
17.8
31.4
CTC decoding
19.1
+2.0 [ +1.8,+2.2 ]
23.8
16.1
20.8
32.4
Frozen encoder
25.5
+8.4 [ +8.0,+8.8 ]
30.4
22.0
27.5
31.8
No numeric clips
19.8
+2.7 [ +2.4,+3.0 ]
22.3
21.3
19.0
31.8
Table 10: Ablations on the test set (WER, %). Δ : change from the main configuration, with its 95% confidence interval. Num./Other: clips with and without numbers in the source transcript.
System
English (1,617 words)
Nepali (3,744 words)
Zero-shot
88.0
32.6
No numeric clips
16.6
29.8
Fine-tuned, 6 epochs
4.9
6.4
Table 11: How often number words in the test references are recognised wrongly (substitutions and deletions, %), by the language they were spoken in.
System
WER
WER CM
CER
Rom-WER
Rom-CER ¬sp
Latin
SraVaani zero-shot
22.6
30.0
6.8
17.8
5.1
0.0
SraVaani fine-tune, 6 epochs
31.1
23.4
15.6
26.9
7.6
10.1
SraVaani fine-tune, 12 epochs
31.4
24.0
15.4
27.1
7.6
9.8
Whisper zero-shot
88.4
87.9
28.0
74.9
18.0
0.4
Whisper fine-tune
36.9
29.0
16.8
30.1
8.4
9.1
IndicConformer zero-shot
22.2
29.1
6.8
17.5
5.1
0.0
Table 12: Results on FLEURS Nepali (%). WER CM : WER against references rewritten in our code-mixed convention (Sec. 6.5 ). Rom-CER ¬sp ignores script and spaces; Latin: share of output words in Latin script.
Domain
Reference
Zero-shot
Fine-tuned, 6 epochs
Lectures
a one a one dot length a one \nepalifont को length \nepalifont सम्म
a one a one dot length a one \nepalifont को length \nepalifont सम्म
Conversation
responsibility \nepalifont दिँदा project after \nepalifont पनि how they will continue \nepalifont भन्ने कुरा छ
\nepalifont रेस्पोन्सिबिलिटी दिँदा प्रोजेक्ट आफ्टर पनि हाउ दे विल कन्टिन्यू भन्ने कुरा छ
responsibility \nepalifont दिँदा project after \nepalifont पनि how they will continue \nepalifont भन्ने कुरा छ
Lectures
\nepalifont यो भनेको zero point three five x \nepalifont भनेको thirty five \nepalifont भयो
\nepalifont यो भनेको जिरो पोइन्ट थ्री फाइभ एक्स भनेको थर्टी फाइभ भयो
\nepalifont यो भनेको zero point three five x \nepalifont भनेको thirty five \nepalifont भयो
Conversation
certificate \nepalifont पायो भने we are very very lucky
\nepalifont सर्टिफिकेट पाय भने वी आर वेरी वेरी लकी
certificate \nepalifont पायो भने we are very very lucky
Lectures
\nepalifont कति अरू पढियो girls fast student teacher doctor master
golf fast student teacher doctor master
\nepalifont केटीहरुले बाँकी गर्नुहोस् gulf fast student teacher doctor master \nepalifont ए
Table 13: Representative transcript examples. The fine-tuned model usually retains English in Latin script. The final example illustrates a remaining error.