Code-switched (CS) automatic speech recognition (ASR) remains difficult because real CS speech is scarce and synthetic speech does not fully match it. We investigate whether synthetic CS speech is more useful for learning an adaptation-friendly initialization than for ordinary continual fine-tuning. We propose SyRA-CS, a pair-specific synthetic-to-real adaptation framework based on Reptile meta-learning. For each language pair and pretrained model, we meta-train on the corresponding synthetic CS-FLEURS subset and then adapt using one hour of labeled real target speech. We evaluate four architectural settings: Encoder-CTC, FastConformer-RNN-T, Attention Encoder-Decoder (AED) and Speech-LLM. Experiments cover on Arabic-, French-, German-, and Hindi-English, against synthetic-only transfer, real-only fine-tuning, and continual fine-tuning. SyRA-CS improves over continual fine-tuning on WER in all 16 model-pair comparisons and on CER in 15 of 16, with relative macro WER reductions of 17.7% for XLS-R and 5.3-5.8% for the other three models. The gains hold across the tested models but vary in size across architectures, language pairs and metrics.
Figures & tables
Figure 1: Overview of the Pair-Specific Synthetic-to-Real Reptile Adaptation Framework for Code-Switched ASR.
Setting
XLS-R 300M
Nemotron 3.5 / Whisper-small / Audex-2B
Architecture
Encoder–CTC
FastConformer–RNN-T / AED / Speech-LLM
Trainable parameters
Full model
Full model
Meta-iterations
500
500
Inner steps ka
50
50
Inner learning rate αa
3×10−4
2×10−5
Outer step size βa
0.1
0.1
Table 1: Reptile configurations. The grouped column reports shared settings; architectures follow the model order.
Training regime
Real data
Hindi-English
French-English
Arabic-English
German-English
WER ↓
CER ↓
WER ↓
CER ↓
WER ↓
CER ↓
WER ↓
CER ↓
XLS-R 300M (Encoder–CTC)
Synthetic-only transfer
0h
0.770
0.590
0.814
0.364
0.976
0.527
0.830
0.333
Real-only fine-tuning
1h
0.980
0.960
0.531
0.182
0.827
0.367
0.784
0.219
CFT from synthetic
1h
0.479
0.288
0.370
0.117
0.615
0.212
0.432
0.137
SyRA-CS (ours)
1h
0.416
0.257
0.350
0.112
0.461
0.163
0.333
0.104
Table 2: Synthetic-to-real CS ASR performance across four pretrained models. WER and CER are reported as fractions. Real data denotes labeled target speech used for adaptation.
Figure 2: Relative WER and CER reductions of SyRA-CS against CFT, computed as 100(eCFT−eSyRA)/eCFT . Positive values indicate improvement; negative values indicate higher error.
Figure 3: Pair-level WER for all four models and training regimes. Values are percentages; lower is better.
Figure 4: Pair-level CER for all four models and training regimes. Values are percentages; lower is better.
Figure 5: Code-switching ratio in each target test set.
Figure 6: Sensitivity of relative macro WER/CER reduction to the retained language pairs. Positive values favor SyRA-CS. Lines connect subset summaries; they do not represent a time trend or uncertainty intervals.Each retained pair receives equal weight.
Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data augmentation via Text-to-speech (TTS) has been explored, existing CS TTS approaches primarily optimise reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed approach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively.
Yue Heng Yeo, Haoyang Li, Yizhou Peng +6
College of Computing and Data Science, Nanyang Technological University, Singapore · HLT-COE & CLSP, Johns Hopkins University, USA · Institute for Infocomm Research (I2R), A⋆STAR, Singapore +1
Code-switching (CSW) remains challenging for large multi-lingual ASR systems in real-world deployment. While fine-tuning on synthetic CSW data is possible, it generally degrades strong monolingual baselines. Our goal is to preserve these capabilities while extending models to handle complex code-switching, including morphological variations across languages. We propose Bayesian factorized adaptation, which learns to efficiently integrate switching-relevant knowledge into strong pretrained models without overwriting existing capabilities. Requiring only a small amount of synthetic data, our approach reduces transcription errors by 32.87% on code-switched words while improving overall WER by 5.31%, all while maintaining mono-lingual performance. Our results demonstrate that effective CSW adaptation depends more on knowledge integration than data complexity.
Enes Yavuz Ugan, Alexander Waibel
Interactive Systems Lab, Karlsruhe Institute of Technology (KIT), Germany · InterACT, Carnegie Mellon University (CMU), USA
Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.
Tung X. Nguyen, Hieu Minh Truong, Giang Son Nguyen +3
VinUniversity, Vietnam · University of Technology Sydney, Australia · Monash University, Australia