Organizations: Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, China · AI research Institute, iFLYTEK Company Limited, Hefei, Anhui, China · School of Electronic Information, Wuhan University, Wuhan, Hubei, China
Code-switching (CS) speech, characterized by language alternation within a single utterance, is prevalent in multilingual communities and poses challenges to automatic speech recognition (ASR) due to rapid language transitions. We propose LLM-HB, a language-aware large language model-guided hotword biasing framework for code-switching ASR (CS-ASR). The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting. Experiments on the ASRU2019 code-switching challenge test set demonstrate that LLM-HB achieves a 20.30% relative reduction in mixed error rate (MER) over the baseline, reaching 5.85% MER under a 15-distractor hotword setting. To the best of our knowledge, this is the first work to integrate LLM-guided biasing into code-switching ASR. We further release the extracted ASRU2019 hotword list to support future research.
Figures & tables
Figure 1: The framework of the proposed large language model-guided hotword biasing method and MoE adaptor.
Method
Training Dataset
MER
B-MER
CER
B-CER
WER
B-WER
Base Model
ASRU2019-CS-Train-Set
9.59
26.91
7.03
25.45
30.44
27.27
+ MoE Adaptor
9.65
27.18
7.06
25.57
30.73
27.58
+ Language Head
9.30
26.27
6.77
24.50
29.87
26.72
+ Hotword Prompt
7.48
16.00
5.92
10.49
20.21
17.75
+ MoE + LH + HP (LLM-HB)
7.36
14.14
6.02
8.11
18.29
16.07
Base Model (baseline)
ASRU2019-CS- Train-Set + Librispeech + Aishell-2
7.34
22.11
5.04
19.26
26.05
22.90
Table 1: Performance comparison of the proposed LLM-HB method and its variants across different datasets. Base model refers to the Whisper medium encoder with a linear projection layer and a frozen Qwen3-4B LLM. All metrics are reported in percentage (%).
Configuration
MER
CER
WER
4 choose 4
7.33
5.01
25.99
4 choose 2
7.47
5.10
26.79
2 choose 2 (Ours)
7.19
4.89
25.94
2 choose 1
7.21
4.89
26.03
1 choose 1
7.23
4.90
26.14
Table 2: Ablation on expert configurations. All metrics are reported in percentage (%).
Figure 2: Performance of backbone + hotword prompt models with different λ1 using MER, CER and WER.
Code-switching (CSW) remains challenging for large multi-lingual ASR systems in real-world deployment. While fine-tuning on synthetic CSW data is possible, it generally degrades strong monolingual baselines. Our goal is to preserve these capabilities while extending models to handle complex code-switching, including morphological variations across languages. We propose Bayesian factorized adaptation, which learns to efficiently integrate switching-relevant knowledge into strong pretrained models without overwriting existing capabilities. Requiring only a small amount of synthetic data, our approach reduces transcription errors by 32.87% on code-switched words while improving overall WER by 5.31%, all while maintaining mono-lingual performance. Our results demonstrate that effective CSW adaptation depends more on knowledge integration than data complexity.
Enes Yavuz Ugan, Alexander Waibel
Interactive Systems Lab, Karlsruhe Institute of Technology (KIT), Germany · InterACT, Carnegie Mellon University (CMU), USA
Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.
Tung X. Nguyen, Hieu Minh Truong, Giang Son Nguyen +3
VinUniversity, Vietnam · University of Technology Sydney, Australia · Monash University, Australia
Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data augmentation via Text-to-speech (TTS) has been explored, existing CS TTS approaches primarily optimise reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed approach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively.
Yue Heng Yeo, Haoyang Li, Yizhou Peng +6
College of Computing and Data Science, Nanyang Technological University, Singapore · HLT-COE & CLSP, Johns Hopkins University, USA · Institute for Infocomm Research (I2R), A⋆STAR, Singapore +1