Paper ID: 2410.11079 • Published Oct 14, 2024
Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Multilingual Large Language Models (LLMs) have demonstrated exceptional
performance in Machine Translation (MT) tasks. However, their MT abilities in
the context of code-switching (the practice of mixing two or more languages in
an utterance) remain under-explored. In this paper, we introduce Rule-Based
Prompting, a novel prompting technique to generate code-mixed sentences. We
measure and compare the code-mixed MT abilities of 3 popular multilingual LLMs:
GPT-3.5-turbo, GPT-4, and Gemini Pro across five language pairs:
English-{Hindi, Bengali, Gujarati, French, Spanish} using k-shot prompting
(k\in\{0, 1, 10, 20\}) and Rule-Based Prompting. Our findings suggest that
though k-shot prompting often leads to the best results, Rule-Based prompting
shows promise in generating unique code-mixed sentences that vary in their
style of code-mixing. We also use k-shot prompting to gauge the code-mixed to
English translation abilities of multilingual LLMs. For this purpose, we create
a gold-standard code-mixed dataset spanning five language pairs:
English-{Hindi, Bengali, Gujarati, French, Spanish}. As a real-world
application of our work, we create a code-mixed chatbot.