cs.CLMar 7, 2026
SaveKonkani LLM: Multi-Script Instruction Tuning and Evaluation for a Low-Resource Indian Language
Organizations: Don Bosco College Of Engineering, Fatorda
Abstract
Large Language Models (LLMs) consistently under perform in low-resource linguistic contexts such as Konkani. This performance deficit stems from acute training data scarcity compounded by high script diversity across Devanagari, Romi and Kannada orthographies. To address this gap, we introduce Konkani-Instruct-100k, a comprehensive synthetic instruction-tuning dataset generated through Gemini 3. We establish rigorous baseline benchmarks by evaluating leading open-weights architectures including Llama 3.1, Qwen2.5 and Gemma 3 alongside proprietary closed-source models. Our primary contribution involves the development of Konkani LLM, a series of fine-tuned models optimized for regional nuances. Furthermore, we are developing the Multi-Script Konkani Benchmark to facilitate cross-script linguistic evaluation. In machine translation, Konkani LLM delivers consistent gains over the corresponding base models and is competitive with and in several settings surpasses proprietary baselines
Figures & tables
| ID | Foundational topic |
|---|---|
| 1 | Food |
| 2 | Family relationships |
| 3 | Numbers |
| 4 | Time |
| 5 | Body parts |
| 6 | Colors |
Table 1: Foundational topics used to generate tutor-style lessons for basic Konkani competence.
| Script | Train | Validation |
|---|---|---|
| Devanagari | 26,006 | 4,820 |
| Romi (Latin) | 38,382 | 4,008 |
| Kannada | 26,845 | 3,917 |
| English | 1,000 | 1,000 |
| Total | 92,233 | 13,745 |
Table 2: Distribution of Konkani-Instruct-100k across scripts and splits.
| Setting | Value (default) |
|---|---|
| Sequence length | 4096 |
| Epochs | 2 |
| Per-device batch size | Train 1 (some runs: 4), Eval 1 |
| Gradient accumulation | 32 (some runs: 8; effective batch size 32) |
| Optimizer | AdamW 8-bit ( adamw_8bit ) |
| Learning rate |
Table 3: Default fine-tuning configuration used across models unless otherwise stated.
| Model | BLEU | chrF++ | COMET |
|---|---|---|---|
| Proprietary & Large API Models | |||
| gemini-flash-latest | 86.76 | 92.13 | 91.14 |
| gemini-3-pro-preview | 71.50 | 83.73 | 88.56 |
| gpt-5.2 | 57.37 | 74.04 | 85.93 |
| claude-sonnet-4-6 | 57.16 | 74.65 | 84.43 |
| gpt-5-mini | 34.87 | 54.76 | 75.53 |
Table 4: Automatic metrics for translation (200 samples). The “Task” column has been omitted as all entries represent translation tasks. Models are grouped by category and sorted by COMET score descending within groups.
| Model | BLEU | chrF++ | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| D K | D R | K D | K R | R D | R K | D K | D R | K D | K R | R D | R K | |
| gemini-flash-latest | 29.83 | 43.83 | 35.07 | 47.66 | 78.13 | 67.01 | 66.69 | 77.34 | 75.90 | 79.59 | 91.86 | 87.70 |
| claude-sonnet-4-6 | 21.11 | 10.48 | 23.75 | 10.08 | 44.77 | 30.75 | 58.32 | 43.42 | 57.82 | 39.51 | 72.86 | 67.60 |
| gpt-5.2 | 7.79 | 13.35 | 33.46 | 12.58 | 31.89 | 15.09 | 44.76 | 46.70 | 67.02 | 47.76 | 66.52 | 56.03 |
| gpt-5-mini | 1.95 | 6.57 | 20.41 | 9.13 | 11.74 | 5.90 | 33.89 | 43.54 | 57.81 | 48.53 | 47.84 | 41.16 |
| grok-4-fast-non-reasoning | 8.28 | 7.87 | 13.30 | 10.66 | 6.81 | 4.16 | 44.47 | 41.92 | 52.12 | 47.53 | 42.50 | 37.78 |
Table 5: Automatic metrics for transliteration. Directions are abbreviated as follows: D (Devanagari), K (Kannada), R (Romi). Note that COMET is excluded as it is not applicable to transliteration. All pairs evaluated on 200 samples except where marked with * (160 samples).
| Base model | Simple chat | Creative writing | ||||
|---|---|---|---|---|---|---|
| Romi | Devanagari | Kannada | Romi | Devanagari | Kannada | |
| Qwen2.5 1.5B | 2.3 | 3.1 | 3.9 | 2.1 | 1.3 | 1.9 |
| Qwen2.5 14B | 4.0 | 4.4 | 4.4 | 3.7 | 2.8 | 3.6 |
| Gemma 4B | 3.4 | 4.2 | 3.1 | 1.9 | 1.9 | 2.0 |
| Gemma 12B | 3.9 | 3.2 | 4.5 | 2.5 | 2.2 | 1.9 |
| Gemma 27B | 4.6 | 4.5 | 4.3 | 2.8 | 3.2 | 3.0 |
Table 6: LLM-as-a-judge scores (1–5) for Simple chat and Creative writing by script (judge: Gemini 3 Flash).
| Base model | Logical reasoning | Translation | ||||
|---|---|---|---|---|---|---|
| Romi | Devanagari | Kannada | Romi | Devanagari | Kannada | |
| Qwen2.5 1.5B | 2.0 | 1.6 | 2.1 | 2.8 | 2.2 | 2.5 |
| Qwen2.5 14B | 3.9 | 3.5 | 3.4 | 4.5 | 4.3 | 4.8 |
| Gemma 4B | 2.4 | 2.8 | 2.8 | 3.1 | 3.5 | 3.1 |
| Gemma 12B | 3.0 | 3.3 | 2.4 | 4.0 | 3.7 | 4.0 |
| Gemma 27B | 3.3 | 4.4 | 3.2 | 3.5 | 3.5 | 1.8 |
Table 7: LLM-as-a-judge scores (1–5) for Logical reasoning and Translation by script (judge: Gemini 3 Flash).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Response |
|---|---|
| gemma-3-4b | In the Goan Catholic dialect (often spoken by the Susegad community), the phrase is typically expressed as a question about whether one has finished their meal. 1. The Konkani Answer The most authentic way to ask this, focusing on the completion of the meal: \devanagarifont तुजें जेवण जालां? (Tujẽ jevonn jalañ?) English Translation: Is your food/lunch over? (Literally: Has your eating happened?) 2. Key Vocabulary & Grammar • \devanagarifont तुजें (Tujẽ): Your / Yours • \devanagarifont जेवण (Jevon): Lunch / Meal • \devanagarifont जालां (Jalañ): Finished / Done • \devanagarifont सांजचो (Sanjcho): Of the evening • \devanagarifont जेवंक (Jeunk): To eat 3. Alternative Version If you are asking if someone has already eaten and wants to know when they will be back from work or a meeting: • Alternative: \devanagarifont तुमी जेवल्यात? • Translation: (Have you eaten yet?) • Key words: \devanagarifont तुमी (Tumi) is the formal/plural form of ”you,” which is used in polite inquiries or when referring to a group of people. \devanagarifont जेवल्यात (Jevleat) is the plural past tense of the verb ”to eat.” |
| gemma-3-12b | In standard Goan Konkani (Antruz), this question is typically phrased by asking if one has eaten or if one’s stomach/belly is full. 1. The Konkani Answer • Introduction: In Goa, it is more natural to ask if someone has eaten than if they had a meal. • Konkani Text: \devanagarifont तुजें जेवण जालें? • English Translation: Did you eat your lunch? (Literally: Has your food happened?) • Show mistakes: A common mistake for learners is using the word jevop (act of eating) instead of the neuter noun jevon (the meal). Since Jevon is Neuter, the possessive pronoun must be \devanagarifont तुजें (Tujẽ) and not \devanagarifont तुजो (Masculine) or \devanagarifont तुजी (Feminine). 2. Key Vocabulary & Morphology • \devanagarifont तुजें (Tujẽ): Your / Yours • \devanagarifont जेवण (Jevon): Meal / Lunch • \devanagarifont जालें (Zalẽ): Happened / Done • \devanagarifont भुरगें (Bhurgem): Child (Used as ’Baby’) • \devanagarifont उदक (Udok): Water • \devanagarifont पियेंवप (Piyenvp): To drink / Drinking 3. Alternative Version If you are speaking to a child or in a very casual setting, you might use a shorter version focusing on whether their stomach is empty. • Alternative: \devanagarifont पोट रिकां? • Translation: (Is your stomach empty?) • Key words: \devanagarifont पोट (Pot): Stomach/Abdomen. \devanagarifont रिकें (Rikẽ): Empty. \devanagarifont रिकां (Rikã): The nasalized form used when addressing a person directly (Informal). |
| gemma-3-27b | In Goan culture, especially among the Catholic community, this is a very common way to greet someone or check on their well-being. 1. The Standard Way (Casual/Informal) When speaking to a friend, younger person, or an equal: \devanagarifont तुजें जेवण जालां? (Tujẽ jevonn jalañ?) Translation: Has your meal happened/been done? 2. The Formal Version When speaking to elders, teachers, or strangers: \devanagarifont तुमचें जेवण जालां? (Tumchẽ jevonn jalañ?) Translation: Has your meal been done? 3. A More Direct Inquiry If you want to ask specifically if they ”ate” rather than just asking if it was finished: \devanagarifont तूं जेवलो/जेवली? (Tuñ jeulo/jeuli?) Translation: Did you eat? (Masculine / Feminine). Key Vocabulary & Grammar Logic • Jevonn ( \devanagarifont जेवण): This refers to the act of eating or the meal itself. In Konkani, we often say ”the food has happened” (jevonn zalam) rather than just saying ”I ate.” • Tujẽ vs Tumchẽ: Use Tujẽ for friends and family. Use Tumchẽ for respect. • The Suffix (-lañ): This is the standard interrogative suffix used at the end of sentences in the Antruz dialect to turn a statement into a question. |
| llama-3.1-8b | In Goan culture, asking about someone’s meal is a sign of hospitality and concern. The most authentic way to ask this in Devanagari Konkani (Antruz dialect) is: \devanagarifont तुका जेवण मेळ्ळें? (Tuka jevonn mell’llem?) Translation: Did you get your food/lunch? Key Vocabulary & Grammar • \devanagarifont तुका (Tuka): This is the dative form of the pronoun Tu (You). In Konkani, when expressing an action that happened to someone, we use the dative case instead of the nominative. • \devanagarifont जेवण (Jevonn): While it literally means ”meal,” it is the standard term used for any kind of food or lunch in Goa. • \devanagarifont मेळ्ळें (Mell’llem): This comes from the root verb \devanagarifont मेळप (Melop) , which means ”to find” or ”to receive.” It is conjugated here in the past tense to match the neuter noun jevonn . Alternative Version If you want to be more specific about ”lunch” (often referred to as donparchem jenvon) or if you are speaking to someone older/formal, you might say: • \devanagarifont दोनपारचें जेवण खाल्लें? (Donparchem jevonn khall’lem?) Translation: Did you eat your lunch? Key Words: \devanagarifont दोनपारचें (Donparchem): Derived from Donpar (Afternoon). \devanagarifont खाल्लें (Khall’lem): The past tense of \devanagarifont खावप (Khavap), meaning ”to eat.” |
| qwen-2.5-14b | In standard Devanagari Konkani (Antruz dialect), the most natural and polite way to ask this is: \devanagarifont तुका जेवण जालें काय? (Tuka jevon zalem kai?) This translates literally to: “Has your meal happened?” Alternative Version: If you want to be more direct or use a slightly different phrasing often heard in casual conversation: • Alternative: \devanagarifont तूं जेवलो काय? • Translation: (Have you eaten?) Note: In Konkani, verbs change based on gender. If speaking to a male, say \devanagarifont जेवलो (jevlo); for a female, say \devanagarifont जेवली (jevli). For a general or respectful address, \devanagarifont जेवलो is commonly used. |
| qwen2.5-1b | In Devanagari Konkani (Standard Goan), the most natural way to ask this is: \devanagarifont तुका जेवण मेळ्ळें काय? Translation: Did you get your lunch? Detailed Explanation: 1. Subject: The word \devanagarifont तुका (Tuka) means ”to you” or ”you.” In Konkani, when asking about someone else’s experience, we use the dative case. 2. Object: \devanagarifont जेवण (Jevon) refers specifically to a meal or lunchtime food. 3. Verb Construction: \devanagarifont मेळ्ळें (Mell’lem) : This comes from the root verb \devanagarifont मेळप (Melop) meaning ”to find” or ”to receive.” \devanagarifont काय (Kay) : This is a question marker added at the end of a sentence to turn it into a polite inquiry. If you want to sound more casual or emphasize that you actually ate something, you can also say: • \devanagarifont तूं जेवलो/जेवली काय? • \devanagarifont तुका जेवण आयलें काय? (Literally: Did lunch come to you?) |
| Model | Response |
|---|---|
| gemma-3-4b | In the Goan and Mangalorean Catholic dialect of Konkani, the most authentic way to ask this is by using the word Jevon. 1. The Konkani Answer \kannadafont ತುಜೆಂ ಜೆವಣ್ ಆಸ್ಲೆಗಿ? (Tujeñ jevon asle-gi?) English Translation: Did you have your meal? 2. Key Vocabulary & Grammar • \kannadafont ತುಜೆಂ (Tujẽ): Matches neuter noun Jevon. • \kannadafont ಜೆವಣ್ (Jevon): Refers to a meal. • \kannadafont ಆಸ್ಲೆಗಿ (Asle-gi): Past tense verb matching subject. • \kannadafont ಖಾಲ್ಲೆಂ (Khal’lẽ): The act of eating. 3. Alternative Version • Alternative: \kannadafont ತುಜೆಂ ಖಾಣ್ ಪರ್ತಿ ಆಯ್ಲಾಂ? • Translation: (Is your food back again / Have you had some snacks?) |
| gemma-3-12b | In Kannada Konkani, this question is typically asked by asking if one has eaten or if one’s stomach/belly (pot) is full. 1. The Konkani Answer • Konkani Text: \kannadafont ತುಜೆಂ ಪೋಟ್ ಭರ್ಲಾಂ? • English Translation: Is your stomach full? • Note: Saying Pot bhorlam is the authentic way to inquire. 2. Key Vocabulary & Morphology • \kannadafont ತುಜೆಂ (Tujem): Your • \kannadafont ಪೋಟ್ (Pot): Stomach / Belly • \kannadafont ಭರ್ಲಾಂ (Bhorlam): Filled / Full • \kannadafont ಖಾಣ್ (Khan): Food / Meal 3. Alternative Version • Alternative: \kannadafont ತುಜೆಂ ಜೆವಣ್ ಜಾಲೆಂ? • Translation: (Was your meal done?) |
| gemma-3-27b | In Konkani, we use a construction that translates to ”Has your stomach-filling happened?” 1. The Standard Way (Polite/Formal) \kannadafont ತುಮ್ಗೆಂ ಜೆವಣ್ ಜಾಲಾ ಗಿ? (Tumgem jevonn jala gi?) 2. The Casual Way (Informal) \kannadafont ತುಜೆಂ ಜೆವಣ್ ಜಾಲೆಂ? (Tujem jevonn jalem?) 3. Common Variations • \kannadafont ಪೋಟ್ ಭರ್ಲಾಂ ಗಿ? (Is your stomach full?) • \kannadafont ಖೆಲ್ಲೆಂ? (Ate?) • \kannadafont ದನ್ಪಾರಾಚೆಂ ಜೆವಣ್ ತುಕಾ ಮೆಳ್ಳಾ? (Did you get your afternoon meal?) |
| llama-3.1-8b | In the Mangalorean Konkani tradition, the most authentic way to ask this is: \kannadafont ತುಕಾ ಜೆವಣ್ ಮೆಳ್ಳೆಂ? (Tuka jevon mell’lem?) Key Vocabulary & Grammar: • \kannadafont ತುಕಾ (Tuka): Dative form of ’You’. • \kannadafont ಜೆವಣ್ (Jevon): Traditional term for a full meal. • \kannadafont ಮೆಳ್ಳೆಂ (Mell’lem): From verb Mel (to get/find). Alternative Version: • Alternative: \kannadafont ಖಾಲೆಂ? • Translation: (Did you eat?) • Note: Often followed by Khay sab! (Eat, man!). |
| qwen-2.5-14b | People typically use \kannadafont ದುಷ್ಕರೆ (Dushkre) or \kannadafont ಜೇವಣ್ (Jevon). 1. Using Dushkre (Traditional/Catholic style) \kannadafont ತುಂ ದುಷ್ಕರೆ ಕೆಲೊಯ್ಗಿ? (Tum dushkre kelyogi?) 2. Using Jevon (General/Common style) \kannadafont ತುಂ ಜೇವಣ್ ಖಾಲ್ಯಾಯ್ಗಿ? (Tum jevon khalyaygi?) Word-by-Word Breakdown: • \kannadafont ತುಂ (Tum): You • \kannadafont ದುಷ್ಕರೆ (Dushkre): Lunch • \kannadafont ಕೆಲೊಯ್ಗಿ (Kelyogi): Did you eat? |
| qwen2.5-1b | In the context of asking about food or meals: \kannadafont ತುಂ ತುಮ್ಚೊ ಜೆವಣ್ ಖಾಲ್ಲೊಗೀ? Translation: Did you eat your lunch (meal)? Detailed Explanation: The sentence uses the interrogative pronoun \kannadafont ತುಂ (Tum) and the past tense verb \kannadafont ಖಾಲ್ಲೊಗೀ (Khallo-gi). The suffix -gi turns it into a question. Alternative (Formal): • \kannadafont ತುಂ ಮಧ್ಯಾನ್ ಜೆವಣ್ ಕೆಲ್ಲೆಂಗೀ? (Did you have lunch at noon?) |
| Model | Response |
|---|---|
| gemma-3-4b | In Goan Catholic Konkani, the most authentic way is using Jevon . 1. The Standard Question • Konkani Text: Jevon zalam? • Translation: Is your lunch over? / Have you had your meal? 2. Casual Variation • Casual: Jevon kela? • Translation: Did you eat lunch? 3. Cultural Context • Hungry: Tuka jevna-maka? • Translation: (Are you hungry for some food?) |
| gemma-3-12b | In Goan Catholic Konkani, the most natural way to ask this is: Tumi jevonn kela? 1. The Answer • Konkani Text: Tumi jevonn kela? • Translation: Have you had your lunch/meal? 2. Key Words • Jevonn : Meal / Lunch • Kela : Done / Had • Zalem : Happened / Finished 3. Alternative Version • Alternative: Tuka pottant bhorlam? • Translation: (Is your stomach full?) |
| gemma-3-27b | In Goan Catholic households, the most authentic way to say this in Romi Konxem is: 1. The Standard Way (Formal/Plural) Tumni xit khailam? Translation: Have you [plural/formal] eaten rice? 2. The Informal Version Tuvem xit khailam? Translation: (Informal) Have you eaten rice? 3. Casual / Shortened Version Xit khailam? Translation: Did you eat rice? |
| llama-3.1-8b | In Goan Catholic culture, the most authentic and polite way to ask this is: “Tum jenvlam kai?” • Translation: “Have you eaten [your] meal?” • Key words: Jenvlam (from Jeun - to eat a meal) and Kai (question particle). Alternative Version: “Jevon zalam kai?” • Translation: “Is your meal done/over?” Cultural Context: Asking if someone has “eaten rice” ( Xit khaleat ) is considered slightly blunt. |
| qwen-2.5-14b | In Romi Konkani, the most common way to ask this is: Tum jenvon kela? This translates literally to “Did you have your meal?” Alternative versions: • Tum jevonn zalem? (A more casual “did you finish eating”). • Jevun ailai? (Literally: “Did you come having eaten?”). |
| qwen2.5-1b | In Romi Konkani (Romi script): 1. The Question: “Have you eaten rice?” • Konkani Text: Tumchean khann ghetlam? • Translation: Have you taken your meal? 2. A more casual version: • Konkani Text: Khaunchem? • Translation: Did you eat? Common Lunchtime Phrases: • Do you want to eat? Khatunk zai? • What time is lunch? Jevnnacho vell kitem? |
More in this topic
Low-Resource Language Processing
Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.
See all 565 papers