Synthetic-to-Real Code-Switched ASR: A Spontaneous Nepali-English Podcast Benchmark

Firoj Paudel

cs.CLOct 22, 2026

Synthetic-to-Real Code-Switched ASR: A Spontaneous Nepali-English Podcast Benchmark

Authors: Firoj Paudel

Organizations: Wiseyak Inc.

Abstract

Nepali-English code-switched conversational speech presents severe challenges for automatic speech recognition (ASR), chiefly through script confusion (inappropriate mixing of Devanagari and Latin orthography) and elevated error rates at language switch points. Supervised fine-tuning (SFT) can adapt multilingual foundation models, but human code-switched audio is expensive to collect and annotate. We curate a new benchmark of 13,044 spontaneous podcast utterances (15.45 h total; 10,214 test, 2,830 dev), audited and verified to eliminate machine translation leakages and acoustic loops. We investigate a two-stage synthetic-to-real adaptation framework: Stage 1 applies SFT on 0.95 h of speech synthesized via a multi-speaker, dual-engine TTS pipeline; Stage 2 explores Group Relative Policy Optimization (GRPO) with transcription and script rewards. We find that synthetic SFT provides the primary cross-domain transfer to spontaneous human speech, reducing Mixed Error Rate (MER) from 23.40% to 21.52% (-1.88 pp) and switch-point error rate (SPER) from 35.89% to 28.77% (-7.12 pp). Subsequent GRPO yields small, statistically significant adjustments on 173 test utterances (1.69%), driving English script confusion (SCR_en) from 6.45% to 6.36% (p <= 0.001) and SPER to 28.70% (p = 0.022 clustered). Crucially, ablation reveals that in this SFT-converged encoder-decoder setting, an auxiliary script reward term yields performance identical to MER-only optimization because rollout candidate groups exhibit near-zero script variance on synthetic data. On monolingual benchmarks, SFT improves Nepali MER by 2.71 pp (stabilizing at 18.42% after GRPO, a 2.38 pp gain over baseline) while preserving English lexical WER (3.38% vs. 3.33%). Verified transcripts, timestamps, episode metadata, and reconstruction scripts are publicly released.

Figures & tables

Table 1

Corpus	System	MER	WER	CER	SCR
Nepali (BHASAANUVAD)	$S_{0}$ Base	20.80	30.73	22.52	8.71
(1,000 utt, 1.95 h)	$S_{1}$ SFT	18.09	29.06	19.22	6.95
	$S_{4}$ GRPO	18.42	29.52	19.56	6.95
English (LibriSpeech)	$S_{0}$ Base	4.21	3.33	1.59	0.59
(1,000 utt, 1.50 h)	$S_{1}$ SFT	5.38	3.42	1.49	1.07
	$S_{4}$ GRPO	5.27	3.38	1.47	1.03

Table 3: Monolingual retention evaluation on external benchmarks. All metrics in percentage (%).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus Partition	Utterances	Duration (h)	Episodes	Speakers	Nepali Words	English Words	Total Words	Switch Points
Raw Scraped Pool	13,277	15.74	28	72	—	—	—	—
Quarantined (Audio/Music)	233	0.29	—	—	—	—	—	—
Verified Clean Total	13,044	15.45	28	72	123,174 (67.3%)	59,839 (32.7%)	183,013	29,033
Real Dev Split	2,830	3.55	16	20	28,041 (66.5%)	14,121 (33.5%)	42,162	7,504
Frozen Real Test Split	10,214	11.90	25	52	95,133 (67.5%)	45,718 (32.5%)	140,851	21,529

Appendix

Table 4: Comprehensive breakdown of the curated Nepali–English code-switched conversational benchmark across development, evaluation, and quarantined subsets.

Configuration / Hyperparameter	Stage 1: SFT	Stage 2: GRPO
Model Architecture & Adaptation Strategy
Base Architecture	IndicTranscribeFlex (Canary-1B-v2, 1.22B parameters, FastConformer)
Acoustic Encoder	Frozen (32 Conformer blocks, 8 attention heads, $d_{\text{model}}=1024$ )
Trainable Parameters	410.46M (33.60% of 1.22B base model: 24-layer Transformer decoder + head)
Adapted Modules	transf_decoder.* (all 24 layers) and log_softmax.* projection head
Optimization & Training Schedule

Appendix

Table 5: Training configurations and hyperparameters for Stage 1 Supervised Fine-Tuning (SFT) and Stage 2 Group Relative Policy Optimization (GRPO).

Algorithm: Deterministic Script Confusion Rate (SCR) Calculation
Input: Reference sequence $Y^{}=(w_{1}^{},\dots,w_{M}^{*})$ , Hypothesis sequence $Y=(w_{1},\dots,w_{N})$
Output: Overall Script Confusion Rate $\text{SCR}_{\text{all}}$ , English Script Confusion Rate $\text{SCR}_{\text{en}}$
1: Tokenize sequences into whitespace-delimited word tokens: $W_{\text{ref}}\leftarrow\text{words}(Y^{*}),\;W_{\text{hyp}}\leftarrow\text{words}(Y)$
2: Assign script labels: $S_{\text{ref}}\leftarrow[\text{Script}(w)\text{ for }w\in W_{\text{ref}}]$ , where $\text{Script}(w)\in\{\text{Devanagari},\text{Latin},\text{Mixed},\text{Other}\}$
3: Compute Levenshtein distance alignment matrix $\mathcal{A}\leftarrow\text{Align}(W_{\text{ref}},W_{\text{hyp}})$ using unit edit cost ( $\text{cost}=1$ )
Deterministic backtrace tie-breaking: $(\text{diagonal match/substitution})\succ\text{deletion}\succ\text{insertion}$

Appendix

Table 6: Algorithmic specification for Script Confusion Rate (SCR) alignment and computation.

Utterance ID & Context	System	Transcription
HFtjKhqjJdM_0634	Ground Truth	… \devfont चार पाँच हजार dollar \devfont कमाउनुहुन्थ्यो भने youtube \devfont बाट
“If you earned $4–5k from YouTube…”	$S_{0}$ Base	…4-5 \devfont हजार कमाउनु हुन्थ्यो भने youtube \devfont -बाट
	$S_{1}$ SFT	… \devfont चार पाँच हजार डलर कमाउनुहुन्थ्यो भने युट्युबबाट
	$S_{4}$ GRPO	… \devfont चार पाँच हजार dollar \devfont कमाउनुहुन्थ्यो भने youtube \devfont बाट
kM6-yopuJWU_0673	Ground Truth	\devfont बर्गर context \devfont मा chicken \devfont नै हो हैन त्यसपछि kfc \devfont भयो
“In burger context it’s chicken, then KFC…”	$S_{0}$ Base	… bargar oven x \devfont -मा चिकनै हो होइन त्यसपछि kfc \devfont भयो

Appendix

Table 7: Qualitative comparison of code-mixed transcriptions across adaptation stages on illustrative utterances selected from the 173 diffs between

S_{1}

and

S_{4}

in the Frozen Real Test Split, alongside a non-diff control example ( -IE5q6_CMnU_0082 ) demonstrating Nepali script stability. Utterance IDs link to public Hugging Face manifests.

Thematic Domain	Episodes	Speakers	Duration (h)	Nepali Words	English Words	Representative Embedded Terms
Entrepreneurship & Business	9	24	5.12	41,208 (65.8%)	21,428 (34.2%)	startup, equity, revenue, brand, outlet
Technology & AI	7	18	3.86	30,119 (64.1%)	16,870 (35.9%)	software, model, feature, backend, system
Higher Education & Career	6	15	3.31	27,243 (68.7%)	12,411 (31.3%)	university, semester, career, visa, degree
Public Health & Society	6	15	3.16	24,604 (72.9%)	9,130 (27.1%)	hospital, mental health, community, case
Total Benchmark	28	72	15.45	123,174 (67.3%)	59,839 (32.7%)	29,033 switch points

Appendix

Table 8: Thematic domain distribution and linguistic composition across the 28 podcast episodes in the benchmark.

Code-Switching Speech Recognition

Code-switching speech recognition transcribes speech that shifts between languages, preserving words, language boundaries, and writing systems.

See all 22 papers

Mar 8, 2026

Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR

Nepal Bhasha (Newari), an endangered language of the Kathmandu Valley, remains digitally marginalized due to the severe scarcity of annotated speech resources. In this work, we introduce Nwāchā Munā, a newly curated 5.39-hour manually transcribed Devanagari speech corpus for Nepal Bhasha, and establish the first benchmark using script-preserving acoustic modeling. We investigate whether proximal cross-lingual transfer from a geographically and linguistically adjacent language (Nepali) can rival large-scale multilingual pretraining in an ultra-low-resource Automatic Speech Recognition (ASR) setting. Fine-tuning a Nepali Conformer model reduces the Character Error Rate (CER) from a 52.54% zero-shot baseline to 17.59% with data augmentation, effectively matching the performance of the multilingual Whisper-Small model despite utilizing significantly fewer parameters. Our findings demonstrate that proximal transfer from Nepali language serves as a computationally efficient alternative to massive multilingual models. We openly release the dataset and benchmarks to digitally enable the Newari community and foster further research in Nepal Bhasha.

Rishikesh Kumar Sharma, Safal Narshing Shrestha, Jenny Poudel +4

Information and Language Processing Research Lab, Kathmandu University

Automatic Speech Recognition Low-Resource Language Processing Low-Resource Speech Recognition Cross-Lingual Transfer

Ask PDF

Jun 14, 2026

Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech

Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data augmentation via Text-to-speech (TTS) has been explored, existing CS TTS approaches primarily optimise reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed approach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively.

Yue Heng Yeo, Haoyang Li, Yizhou Peng +6

College of Computing and Data Science, Nanyang Technological University, Singapore · HLT-COE & CLSP, Johns Hopkins University, USA · Institute for Infocomm Research (I2R), A⋆STAR, Singapore +1

Code-Switching Speech Recognition Synthetic Data Augmentation TTS Synthesis Automatic Speech Recognition Synthetic Data Augmentation for Speech Recognition Preference Learning+1

Ask PDF

May 18, 2026

Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German

Code-switching -- the natural alternation between two languages within a single utterance -- remains one of the most challenging and under-studied conditions for automatic speech recognition (ASR). We present a benchmark evaluating five commercial ASR providers across four language pairs: Egyptian Arabic--English, Saudi Arabic (Najdi/Hijazi)--English, Persian (Farsi)--English, and German--English, comprising 300 samples per pair selected by a two-stage pipeline combining heuristic filtering with a GPT-4o and Gemini 1.5 Pro ensemble scorer, reducing LLM costs by

\approx

91%. We evaluate on both WER and BERTScore, showing that while both metrics agree on the ordinal ranking of systems for all Arabic and Persian pairs (

τ= 1.0

), WER inflates the magnitude of quality gaps by approximately 3

\times

by penalising semantically correct transliteration choices. ElevenLabs Scribe v2 achieves the lowest WER (13.2% overall) and leads on BERTScore (0.936 overall). Difficulty-stratified analysis reveals performance gaps masked by aggregate averages, and BERT embedding projections confirm semantic proximity between reference and hypothesis despite surface-level script differences. The dataset is publicly available at https://huggingface.co/datasets/Perle-ai/ASR_Code_Switch.

Sajjad Abdoli, Ghassan Al-Sumaidaee, Clayton W. Taylor +2

Perle AI

Code-Switching Speech Recognition ASR Evaluation Automatic Speech Recognition Multilingual ASR

Ask PDF

Abstract

Code-Switching Speech Recognition

Explore similar work

Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR

Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech

Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German