Organizations: Human Geography, University of Bremen, Bremen, Germany · Adrienne Arsht Community-Based Resilience Solutions Initiative, Smithsonian Tropical Research Institute, Panama City, Panama · Avalon University, Willemstad, Netherlands Antilles · SISSTEM Program, Faculty of Arts and Science, University of Aruba, Oranjestad, Aruba · Research Center Trustworthy Data Science and Security, TU Dortmund, Dortmund, Germany · TU Dortmund University, Dortmund, Germany
Multilingual speakers choose languages by topic and conversation partner. Large language models (LLMs) support these languages unequally. Research on code-switching with conversational AI has mostly examined bilingual pairs of well-resourced languages. In Curaçao and Aruba, many people use Dutch, English, Spanish, and the low-resource Creole Papiamentu/o. In a preregistered study, 42 university students held two five-minute chatbot conversations, one about university life and one about local food and culture. Surveys assessed language background, AI use, and switching comfort. 20 international students in Curaçao formed a comparison sample. Participants switched between turns and rarely mixed languages within a message. English dominated both conversations. Participants switched more often in the cultural conversation, where Papiamentu/o and Spanish rose from 15% to 24% of messages. Self-reports showed no lower comfort switching with the chatbot than with a person. Conversational agents must follow turn-by-turn language changes and handle local low-resource languages well.
Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). Recent studies on synthetic code-switching for LLMs have primarily focused on bilingual transfer between English and a target language, leaving instruction-tuning settings that jointly mix three or more languages within each example largely unexplored. In this work, we investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese. We evaluate multilingual understanding on Belebele and MMLU. Our experiments show that simple sentence-level multilingual CSD consistently improves the four-language average accuracy across all bilingual, trilingual, and quadrilingual settings for Gemma-2-2B, indicating that multilingual code-switching can be effective beyond bilingual transfer settings.
Code-switching (CS) speech, characterized by language alternation within a single utterance, is prevalent in multilingual communities and poses challenges to automatic speech recognition (ASR) due to rapid language transitions. We propose LLM-HB, a language-aware large language model-guided hotword biasing framework for code-switching ASR (CS-ASR). The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting. Experiments on the ASRU2019 code-switching challenge test set demonstrate that LLM-HB achieves a 20.30% relative reduction in mixed error rate (MER) over the baseline, reaching 5.85% MER under a 15-distractor hotword setting. To the best of our knowledge, this is the first work to integrate LLM-guided biasing into code-switching ASR. We further release the extracted ASRU2019 hotword list to support future research.
Yuxuan He, Genshun Wan, Pengcheng Li +3
Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, China · AI research Institute, iFLYTEK Company Limited, Hefei, Anhui, China · School of Electronic Information, Wuhan University, Wuhan, Hubei, China
Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.
Dries Rooryck, Alex Cai, Yonatan Belinkov +2
Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel