Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding relevant utterances in large speech collections so that they can be included in training data. One way to find such utterances is to transcribe the speech and detect language alternation in the text, but this depends on obtaining accurate multilingual transcriptions. We propose a lightweight method that detects CS directly from speech. It uses frame-level language information in pretrained speech encoders to determine whether both languages are present within an utterance. Specifically, we identify language-selective hidden-state dimensions using monolingual recordings and aggregate their frame-level activations into a CS prediction. With a small CS set used to tune a decision rule, we show that the Whisper-small encoder achieves a mean F1 of 0.798 across 14 language pairs in CS-FLEURS, compared with 0.776 for a CNN and 0.751 for a HuBERT linear baseline trained for monolingual language identification. Our findings suggest that pretrained speech encoders already encode the information needed to identify CS utterances, and they provide a lightweight tool for building more representative multilingual corpora.
Figures & tables
CS-FLEURS
SEAME_combined
Calibrated
Calibrated
Method
Model
K
Oracle
Mean
Std
Oracle
Mean
Std
Pair-specific, K=1%
Whisper-small
8
0.798
0.784
.021
0.734
0.733
.008
Pair-specific, K=1%
mHuBERT-147
8
0.756
0.737
.022
0.732
0.731
.009
Pair-specific, K=1%
MMS-1B
13
0.762
0.745
.020
0.758
0.755
.006
Recurring dims. (all pairs)
Whisper-small
8
0.790
0.774
.020
0.736
0.734
.011
Table 1: Utterance-level CS detection F1. Oracle parameters are selected on the evaluation set; calibrated parameters are selected from limited labelled data. Calibrated results show mean and standard deviation over 200 repetitions.
Pair
Whisper-small
mHuBERT-147
MMS-1B
ara-eng
0.832
0.806
0.824
ces-eng
0.793
0.709
0.674
cmn-eng
0.762
0.693
0.728
deu-eng
0.754
0.672
0.677
fra-eng
0.792
0.712
0.727
hin-eng
0.846
0.762
0.835
Table 2: Utterance-level CS detection F1 for each CS-FLEURS language pair using pair-specific dimensions and oracle parameter selection.
Code-switched (CS) speech leaks through the monolingual language identification (LID) filters used to curate massive speech corpora, calling for CS-aware LID (CS-LID). We formulate utterance-level CS-LID as multi-label language-set prediction and propose a set generator that directly outputs the languages in an utterance, comparing it against atomic-pair and score-based classification baselines. Oracle Top-k is the strongest baseline, but thresholding fails because no single threshold separates CS from monolingual speech. Our set generator predicts the correct language count on unseen pairs without assuming the number of languages, but underperforms oracle Top-k in exact set accuracy. Our analysis identifies the key obstacles to robust CS-LID: oracle cardinality, threshold instability, language bias in CS training data, and the synthetic-to-real gap.
Shunsuke Mitsumori, Matthew Wiesner, Shigeo Morishima +1
Waseda University, Tokyo, Japan · Johns Hopkins University, Baltimore, USA · Carnegie Mellon University, Pittsburgh, USA
We present CS-YODAS, a Creative Commons-licensed dataset of in-the-wild code-switched speech mined from multilingual YouTube data. Code-switching (CS), or the alternation between languages within an utterance or conversation, is common in multilingual settings but remains underrepresented in existing CS speech resources, which are typically small, domain-specific, or artificially constructed. Building on the YODAS corpus, we develop a scalable, human-in-the-loop pipeline for identifying and validating naturally occurring code-switching. The resulting dataset, which totals 313 hours and spans 7 matrix languages, provides diverse, real-world examples of spontaneous code-switched speech. We further analyze the distribution and characteristics of code-switching in the wild, examining language-pair frequencies and switching patterns, and report baseline results for spoken language identification. We hope that CS-YODAS will encourage broader and more comprehensive research on code-switched speech. Dataset link: https://huggingface.co/datasets/byan/cs-yodas.
Brian Yan, Qingzheng Wang, Matthew Wiesner +9
Carnegie Mellon University · Johns Hopkins University · University of Texas at Austin +4
Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.
Tung X. Nguyen, Hieu Minh Truong, Giang Son Nguyen +3
VinUniversity, Vietnam · University of Technology Sydney, Australia · Monash University, Australia