Tiny Aya: Bridging Scale and Multilingual Depth

Alejandro R. Salamanca; Diana Abagyan; Daniel D'souza; Ammar Khairi; David Mora; Saurabh Dash; Viraat Aryabumi; Sara Rajaee

cs.CLMar 12, 2026

Tiny Aya: Bridging Scale and Multilingual Depth

Authors: Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza, Ammar Khairi, David Mora, Saurabh Dash, Viraat Aryabumi, Sara Rajaee, +18 more

Organizations: Cohere Labs · Cohere

Abstract

Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in translation quality, strong multilingual understanding, and high-quality target-language generation, all with just 3.35B parameters. The release includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models targeting languages from Africa, South Asia, Europe, Asia-Pacific, and West Asia. This report details the training strategy, data composition, and comprehensive evaluation framework behind Tiny Aya, and presents an alternative scaling path for multilingual AI: one centered on efficiency, balanced performance across languages, and practical deployment.

Figures & tables

Figure 1: Benchmark performance across regions. The Tiny Aya model family performs competitively across languages, regions and multilingual benchmark tasks. Comparing the Tiny Aya model that scores best for each region with similar-sized competitors aggregated across multiple massively multilingual benchmarks for a diverse set of tasks (mDolly, mArenaHard, GlobalMGSM, Flores, GlobalMMLU), we find that Tiny Aya advances the state of the art for languages from West Asia and Africa.

Figure 2: Posttraining pipeline and model construction. Starting from Tiny Aya Base, we run region-specific supervised finetuning on five regional data subsets and tune the final regional mixtures. In parallel, we train a global supervised fine-tuned model over all regions with minimal alignment. Each region model is then merged with the global model to produce the final region-specialized releases.

Figure 3: Tokenization efficiency across scripts. We report the average tokens per character for each script, comparing the Tiny Aya tokenizer with Gemma3-4B , Qwen3-4B , and SmolLM3-3B tokenizers. Scripts are sorted by total stacked height from smallest to largest. The label beneath each script reports the number of languages using that script in Tiny Aya . Lower values indicate better tokenization efficiency. Our tokenizer (green) achieves competitive or superior compression across most scripts, with particularly strong performance on scripts underserved by other models such as Khmer, Telugu, Gujarati, Lao, and Ge’ez.

Region	Languages
Europe	English, Dutch, French, Italian, Portuguese, Romanian, Spanish, Czech, Polish, Ukrainian (Cyrillic), Russian (Cyrillic), Greek (Greek), German, Danish, Swedish, Bokmål, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Hungarian, Serbian (Cyrillic), Bulgarian (Cyrillic)
West Asia	Arabic (Arabic), Persian (Perso-Arabic), Turkish, Maltese, Hebrew (Hebrew)
South Asia	Hindi (Devanagari), Marathi (Devanagari), Bengali (Bengali), Gujarati (Gujarati), Punjabi (Gurmukhi), Tamil (Tamil), Telugu (Telugu), Nepali (Devanagari), Urdu (Urdu)
Asia Pacific	Tagalog, Malay, Indonesian, Vietnamese, Javanese (Javanese), Khmer (Khmer), Thai (Thai), Lao (Lao), Chinese (Traditional and Simplified Han), Burmese (Mon-Burmese), Japanese (Japanese), Korean (Hangul)
African	Amharic (Ge’ez), Hausa, Igbo, Malagasy, Shona, Swahili, Wolof, Xhosa, Yoruba, Zulu

Table 1: Language coverage by region. Languages grouped into Europe, West Asia, South Asia, Asia Pacific, and Africa for training and evaluation reporting. The majority of languages are written in Latin script—we indicate the non-Latin scripts that we support in brackets.

Figure 4: Regional composition of posttraining data clusters. Share of posttraining data drawn from each region for each cluster mixture used to train region specific SFT models. These SFT models are later used for merging as shown in Figure 2 . English and code are present in all clusters, and the remaining proportions reflect region- and language-level dataset availability.

Tiny Aya Model Architecture
Parameter	Value	Parameter	Value
Embedding dims	2048	Num layers	36
FFN hidden dims	11008	Vocab size	262k
Num heads	16	Embedding parameters	0.5B
Num KV heads	4	Non-Embedding parameters	2.8B
Sliding window	4096	Total parameters	3.35B

Table 2: Tiny Aya architecture summary. Key model hyperparameters and parameter counts for Tiny Aya .

Name	langs	Prompt
Generative
Dolly	66	Describe a morning routine from a 2 year olds perspective
m-ArenaHard-v2	66	How to code an api in python to download and chat with a huggingface llm?
GlobalMGSM	35	Julia’s boat sprang a leak while she was out on the lake. The boat was taking on two liters of water for every ten feet she rowed back towards shore. It took her sixteen seconds to row twenty feet. The shore was 64 seconds away. How much water had the boat taken on by the time Julia reached shore?
Flores	66	translation_template ( Tuy nhiên, tài xế vẫn sống sót dù bị chấn thương nặng ở đầu.)
WMT24++	55	translation_template (A final push for female equality)

Table 3: Our Benchmark Suite with examples. WMT24++ and Flores prompts are additionally wrapped with translation templates, see Appendix D , GlobalMGSM, BLEnD, GlobalPIQA, NormAd with CoT specifications and output format instructions.

Model	Global MMLU	INCLUDE	Global PIQA
Tiny Aya Global	$44.9\pm 7.3$	$45.1\pm 11.1$	$68.3\pm 10.6$
Gemma3-4B	$45.3\pm 6.6$	$48.9\pm 9.6$	$70.8\pm 9.8$
Qwen3-4B	$49.3\pm 11.9$	$52.2\pm 11.2$	$74.6\pm 11.2$
Qwen3.5-4B	$\textbf{54.8}\pm 10.3$	$\textbf{56.9}\pm 11.6$	$\textbf{79.31}\pm 10.0$
Ministral-3-3B	$46.8\pm 10.7$	$52.6\pm 10.5$	$70.7\pm 10.9$
SmolLM3-3B	$37.2\pm 8.2$	$39.4\pm 10.1$	$63.2\pm 9.8$

Table 4: Discriminative benchmark results. Average accuracy (and standard deviation) on Global MMLU (42 languages), INCLUDE (44 languages) and Global PIQA (116 languages).

Figure 5: Open-ended generation quality versus web presence. mDolly judge scores plotted against an approximate web-presence proxy based on Common Crawl bucketed into five equal-width bins. The trend highlights robustness in lower-web-presence languages relative to same-scale competitors.

Model	mDolly (66)	mArenaHard (66)	GlobalMGSM (35)	Flores (66)	WMT24++ (55)
Tiny Aya Global	86.9 (6.2)	67.4 (6.1)	52.8 (11.7)	43.5 (14.0)	46.0 (10.4)
Gemma3-4B	77.6 ( 24.9 )	65.8 ( 23.2 )	55.4 ( 26.5 )	38.9 (20.0)	41.9 (13.2)
Qwen3-4B	67.3 ( 30.2 )	70.1 ( 28.5 )	60.9 ( 33.6 )	30.5 (19.5)	32.9 (16.0)
Qwen3.5-4B	81.1 ( 18.3 )	76.7 ( 15.5 )	62.5 ( 26.7 )	36.1 (19.4)	40.1 (14.0)
Ministral-3-3B	61.6 ( 29.7 )	63.7 ( 24.4 )	49.6 ( 34.0 )	32.0 (18.7)	31.5 (15.3)
SmolLM3-3B	43.5 ( 25.0 )	31.8 ( 20.2 )	22.6 ( 27.0 )	16.8 (18.5)	18.4 (17.7)

Table 5: Generative and translation benchmark summary. Mean with standard deviation for open-ended and reasoning benchmarks (in percentages), plus ChrF for Flores and WMT24++ translation. Higher is better for all metrics (but standard deviation is better to be lower). Outliers in standard deviation are marked in red, as these indicate increased language disparities.

Model	mDolly (66)	mArenaHard (66)	GlobalMGSM (35)
Tiny Aya Global	97.4 (8.6)	91.4 (8.5)	96.9 (8.5)
Gemma3-4B	94.1 (11.6)	85.7 (13.2)	95.6 (10.6)
Qwen3-4B	94.3 (13.8)	92.3 (15.0)	96.6 (10.6)
Qwen3.5-4B	96.4 (10.5)	94.2 (10.4)	98.4 (5.8)
Ministral-3-3B	91.2 (19.5)	86.1 (18.5)	94.1 (20.0)
SmolLM3-3B	88.3 (19.5)	82.2 (19.8)	86.2 (30.1)

Table 6: Language confusion in open-ended generations and mathematical reasoning. Results are given in percentages. Mean and standard deviation (in brackets) line level pass rates ( Marchisio et al., 2024b ) according to FastText’s language identification for Tiny Aya Global and competitors. Higher is better (except for standard deviation). Note: Averages are computed over FastText’s supported languages.

Figure 6: Translation quality on focus languages from Flores ( Team et al., 2022 ) . ChrF ( Popović, 2015 ) for English-to-target translation on Flores across the Tiny Aya evaluation language set. Tiny Aya Global performs competitively against TranslateGemma-4B , outperforming it on 27 of 66 languages. Notably, TranslateGemma-4B is explicitly optimized for translation, whereas Tiny Aya Global is trained as a general-purpose multilingual model, underscoring the strength of our balanced training approach

Figure 7: Effect of regional specialization on translation. Across Flores translation tasks, region-specialized Tiny Aya variants consistently outperform the Global model, with the strongest gains on English-to-target translation.

Figure 8: Effect of merging on multilingual safety. Merging improves safety across languages for all variants, increasing safe-response behavior under multilingual safety evaluation.

	MultiJail (10 langs)		XSTest (en)
Model	Min Safe Rate $\uparrow$	Mean Safe Rate $\uparrow$	Over-Refusal $\downarrow$	Under-Refusal $\downarrow$
Tiny Aya Global	87.0	91.1	10.4	15.5
Tiny Aya Earth	77.5	87.8	10.4	19.0
Tiny Aya Fire	78.1	90.0	10.4	15.0
Tiny Aya Water	82.9	89.7	10.0	19.5
Gemma3-4B	79.9	88.7	3.6	44.0

Table 7: Safety evaluation summary across benchmarks. Minimum and mean safe response rate across 10 languages of MultiJail and over-refusal and under-refusal rates from English XSTest. Tiny Aya variants behave consistently, delivering the overall highest safe response rates, with minimal disparities across languages, with the trade-off of being slightly more prone to over-refusal than the competitors.

Figure 9: Cultural norm reasoning across countries on NormAd. Country-level performance comparing Gemma3-4B to Tiny Aya variants when prompted in the local language. A Tiny Aya variant is best in 39 of 75 countries, with considerable gains from the Fire and Global models.

Panel of Figure 10 — Figure 10: Impact of test-time reasoning on model performance evaluated on NormAd . We conduct the analysis in two settings: (a) Local Chain-of-Thought reasoning (conducted in the source language) and (b) English Chain-of-Thought reasoning. Results show a consistent accuracy improvement across the Tiny Aya variants in both settings, highlighting the models’ capabilities to leverage test-time compute to reach more accurate judgments.

Figure 11: Cultural commonsense on BLEnD SQA. Country-level accuracy on BLEnD short question answering when prompts are issued in each country’s source language. Results compare Gemma3-4B to all Tiny Aya variants.

Figure 12: Effect of prompt language on BLEnD accuracy. Mean accuracy difference between prompting in English and prompting in local language on BLEnD, aggregated by region. The largest prompt-language sensitivity appears in Africa and South Asia, with smaller gaps in Europe and the Americas.

Figure 13: Tiny Aya throughput across quantization levels. Prefill (left) and Decode (right) throughput (tokens/s) for a standardized workload of 100 input tokens and 100 output tokens.

Figure 14: Open-ended generation quality versus quantization level. mDolly judge scores plotted for various quantization levels show the impact of quantization on open-ended generation quality.

Figure 15: Open-ended generation quality degradation versus web presence. Degradation in mDolly judge scores compared to unquantized (BF16) models plotted against an approximate web-presence proxy based on Common Crawl bucketed into five equal-width bins. The trend highlights robustness to degradation in lower-web-presence languages.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Language	All Regions	South Asia	Europe+WA+AP	Europe+WA+Af
English	13.8	46.2	17.0	18.1

Appendix

Table 8: English data proportion (%) across data mixes.

Language	All Regions	South Asia	Europe+WA+AP	Europe+WA+Af
Dutch	1.6	0.2	2.0	2.2
French	1.8	0.2	2.2	2.3
Italian	1.7	0.3	2.1	2.2
Portuguese	1.7	0.3	2.1	2.2
Romanian	1.6	0.1	2.0	2.1
Spanish	1.7	0.2	2.1	2.2

Appendix

Table 9: European languages data proportion (%) across data mixes.

Language	All Regions	South Asia	Europe+WA+AP	Europe+WA+Af
Arabic	1.8	0.3	2.2	2.3
Persian	1.7	0.1	2.1	2.2
Urdu	1.0	3.4	1.2	1.3
Turkish	1.7	0.3	2.0	2.2
Maltese	1.0	0.0	1.3	1.4
Hebrew	1.7	0.3	2.1	2.3

Appendix

Table 10: West Asia languages data proportion (%) across data mixes.

Language	All Regions	South Asia	Europe+WA+AP	Europe+WA+Af
Hindi	1.7	5.8	0.1	0.1
Marathi	1.1	3.7	0.1	0.1
Bengali	1.1	3.6	0.0	0.0
Gujarati	1.1	3.6	0.0	0.0
Punjabi	1.0	3.4	0.0	0.0
Tamil	1.0	3.4	0.0	0.0

Appendix

Table 11: South Asia languages data proportion (%) across data mixes.

Language	All Regions	South Asia	Europe+WA+AP	Europe+WA+Af
Tagalog	1.0	0.1	1.2	0.0
Malay	0.9	0.0	1.1	0.0
Indonesian	1.6	0.1	2.0	0.1
Vietnamese	1.7	0.3	2.1	0.1
Javanese	0.9	0.0	1.1	0.0
Khmer	1.0	0.0	1.2	0.0

Appendix

Table 12: Asia Pacific languages data proportion (%) across data mixes.

Language	All Regions	Europe+WA+Af
Amharic	1.0	1.3
Hausa	1.0	1.3
Igbo	1.1	1.4
Malagasy	0.9	1.2
Shona	1.0	1.4
Swahili	1.0	1.3

Appendix

Table 13: African languages data proportion (%) across data mixes.

Language	All Regions	South Asia	Europe+WA+AP	Europe+WA+Af
Code	3.5	11.7	4.3	4.5

Appendix

Table 14: Code data proportion (%) across data mixes.

Country	Language	Country	Language	Country	Language
Afghanistan	Farsi	Iraq	Iraqi Arabic	Russia	Russian
Argentina	Spanish	Ireland	Irish	Samoa	English
Australia	English	Israel	Hebrew	Saudi Arabia	Saudi Arabic
Austria	German	Italy	Italian	Serbia	Serbian
Bangladesh	Bengali	Japan	Japanese	Singapore	Malay
Bosnia & Herz.	Bosnian	Kenya	Swahili	Somalia	Somali

Appendix

Table 15: Mapping from countries to the language used for translation in the multilingual NormAd evaluation.

	English Stories
Region	Gemma3-4B	Qwen3-4B	SmolLM3-3B	Ministral-3-3B	Qwen3.5-4B	Tiny Aya Global	Tiny Aya Water	Tiny Aya Earth	Tiny Aya Fire
Europe	72.35	71.23	77.92	80.85	76.64	74.91	75.90	74.22	75.18
West Asia	64.63	73.84	76.06	58.06	76.15	68.17	72.98	68.09	70.99
South Asia	71.20	81.17	81.67	68.70	80.33	69.03	69.87	68.37	73.43
Asia-Pacific	78.29	82.85	84.83	82.27	82.82	77.73	77.94	77.23	78.07
Africa	68.23	71.68	78.72	73.59	76.62	71.43	74.07	73.14	72.82

Appendix

Table 16: Per-Region accuracy on NormAd with stories all in English.

	Source-Language Stories
Region	Gemma3-4B	Qwen3-4B	SmolLM3-3B	Ministral-3-3B	Qwen3.5-4B	Tiny Aya Global	Tiny Aya Water	Tiny Aya Earth	Tiny Aya Fire
Europe	75.37	67.24	74.40	77.42	72.70	72.27	71.95	71.98	75.02
West Asia	63.11	66.47	68.83	53.32	69.53	67.44	69.27	66.35	68.87
South Asia	67.77	65.97	64.97	57.03	77.80	69.07	69.60	70.47	67.77
Asia-Pacific	73.56	73.70	73.11	74.04	80.03	73.77	73.41	75.40	72.80
Africa	65.46	57.48	61.52	60.36	68.82	64.70	65.37	66.19	66.28

Appendix

Table 17: Per-Region accuracy on NormAd with stories in the official language of the country.

Country/Region	Code	Language	Geographic Region
United States	US	English	North America
United Kingdom	GB	English	Europe
China	CN	Chinese	Asia Pacific
Spain	ES	Spanish	Europe
Mexico	MX	Spanish	North America
Indonesia	ID	Indonesian	Asia Pacific

Appendix

Table 18: Countries and Languages in BLEnD dataset with Geographic Grouping

		Source-Language Prompts
Region	Country	Gemma3-4B	Ministral-3-3B	Qwen3-4B	Qwen3.5-4B	SmolLM3-3B	Tiny Aya Global	Tiny Aya Earth	Tiny Aya Fire	Tiny Aya Water
Africa	DZ	27.73	15.03	15.90	28.54	23.53	24.89	27.29	24.40	38.34
	ET	7.92	0.00	5.21	60.42	0.21	11.67	11.88	11.46	12.08
	NG	3.66	1.29	1.29	5.39	1.94	11.21	10.34	11.45	11.21
Americas	MX	52.04	42.13	44.90	45.08	44.49	40.70	43.06	42.13	41.02
Americas	US	64.04	66.97	62.83	61.87	65.66	64.24	63.23	62.20	64.50

Appendix

Table 19: BLEnD SQA Results Accuracy for BLEnD Short Question Answer Split with Source-Language Prompts.

		English-Only Prompts
Region	Country	Gemma3-4B	Ministral-3-3B	Qwen3-4B	Qwen3.5-4B	SmolLM3-3B	Tiny Aya Global	Tiny Aya Earth	Tiny Aya Fire	Tiny Aya Water
Africa	DZ	38.34	35.89	42.48	42.05	35.51	39.08	39.08	41.18	39.04
	ET	26.46	30.90	27.77	28.18	25.68	31.25	32.08	31.87	31.46
	NG	26.29	26.46	23.28	25.86	19.05	23.54	23.53	25.49	22.46
Americas	MX	45.51	28.22	51.12	42.33	47.03	42.94	46.31	45.50	46.94
Americas	US	64.24	67.61	62.22	65.18	65.31	63.41	65.38	65.45	65.45

Appendix

Table 20: BLEnD SQA Results Accuracy for BLEnD Short Question Answer Split with English-Only Prompts.

Language	Gemma3-4B	Ministral-3-3B	Qwen3-4B	Qwen3.5-4B	SmolLM3-3B	Tiny Aya Earth	Tiny Aya Fire	Tiny Aya Global	Tiny Aya Water	TranslateGemma 4B
am	0.01	0.01	0.00	0.05	0.00	0.16	0.07	0.19	0.03	0.16
ar	0.50	0.38	0.42	0.49	0.44	0.53	0.48	0.52	0.53	0.52
bg	0.60	0.48	0.48	0.57	0.22	0.60	0.59	0.60	0.61	0.59
bn	0.42	0.31	0.30	0.34	0.02	0.21	0.32	0.31	0.19	0.46
ca	0.57	0.55	0.55	0.61	0.38	0.58	0.60	0.58	0.61	0.60
cs	0.49	0.48	0.44	0.51	0.16	0.56	0.55	0.54	0.56	0.53

Appendix

Table 21: Flores output quality for translating from English into Tiny Aya focus languages, evaluated with ChrF ( sacrebleu default implementation). We do not recommend using Tiny Aya regional models for languages that were not in focus in their training (see Table 1 and Figure 4 ), where translation quality is significantly lower than for the Tiny Aya Global model.

Language	Gemma3-4B	Ministral-3-3B	Qwen3-4B	Qwen3.5-4B	SmolLM3-3B	Tiny Aya Earth	Tiny Aya Fire	Tiny Aya Global	Tiny Aya Water	TranslateGemma 4B
ar_EG	0.30	0.21	0.27	0.29	0.24	0.30	0.31	0.31	0.30	0.35
ar_SA	0.34	0.24	0.30	0.31	0.28	0.34	0.36	0.35	0.34	0.39
bg_BG	0.54	0.41	0.44	0.49	0.14	0.56	0.56	0.56	0.56	0.56
bn_IN	0.35	0.24	0.27	0.31	0.02	0.29	0.39	0.37	0.25	0.42
ca_ES	0.54	0.48	0.49	0.57	0.29	0.57	0.55	0.56	0.56	0.57
cs_CZ	0.44	0.40	0.37	0.43	0.13	0.48	0.48	0.47	0.48	0.48

Appendix

Table 22: WMT24++ output quality for translating from English, evaluated with ChrF.

Language	Gemma3-4B	Qwen3-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global	Gemma3-4B	Qwen3-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global
	Accuracy					LPR
amh	0.36	0.15	0.00	0.00	0.36	0.92	0.86	0.99	0.99	0.98
ar	0.72	0.84	0.74	0.21	0.65	1.00	1.00	1.00	1.00	1.00
bn	0.59	0.77	0.10	0.05	0.53	1.00	1.00	1.00	1.00	1.00
ca	0.78	0.90	0.83	0.57	0.59	1.00	1.00	0.99	0.02	0.98
cs	0.72	0.84	0.78	0.19	0.56	1.00	1.00	1.00	0.99	0.99

Appendix

Table 23: GlobalMGSM results by language. Per-language answer accuracy and language-consistency pass rate based on FastText, reporting how often the answer is in the prompt language.

	Mean Score						Naturalness Score						LPR
Lang.	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global
am	0.376	0.174	0.665	0.076	0.131	0.586	0.298	0.208	0.732	0.035	0.185	0.686	0.665	0.990	0.970	0.827	0.819	0.889
ar	0.820	0.915	0.868	0.740	0.590	0.722	0.943	0.957	0.919	0.745	0.748	0.887	0.932	0.980	0.968	0.891	0.942	0.922
bg	0.826	0.891	0.865	0.760	0.352	0.718	0.937	0.922	0.911	0.729	0.396	0.881	0.924	0.968	0.974	0.849	0.829	0.922
bn	0.776	0.810	0.781	0.758	0.095	0.658	0.900	0.880	0.847	0.776	0.142	0.809	0.918	0.982	0.978	0.892	0.960	0.899
ca	0.772	0.904	0.869	0.804	0.427	0.718	0.850	0.918	0.908	0.753	0.292	0.883	0.896	0.958	0.974	0.805	0.356	0.944

Appendix

Table 24: Open-ended generation quality on mArenaHard-v2. We report the aggregated score by the LLM judge (averaged across four rubrics, see Section 4.3 ), the judge’s score assigned to the naturalness rubric, and language id line pass rate from FastText.

	Mean Score						Naturalness Score						LPR
Lang.	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global
am	0.436	0.193	0.637	0.010	0.181	0.778	0.343	0.208	0.687	0.007	0.222	0.836	0.768	0.995	0.990	0.955	0.995	0.995
ar	0.937	0.926	0.933	0.773	0.821	0.915	0.989	0.956	0.971	0.794	0.900	0.981	1.000	1.000	1.000	0.990	1.000	1.000
bg	0.945	0.860	0.927	0.760	0.453	0.898	0.987	0.848	0.942	0.719	0.424	0.966	0.990	0.980	0.990	0.894	0.879	0.995
bn	0.930	0.826	0.900	0.803	0.210	0.880	0.971	0.869	0.944	0.871	0.258	0.949	1.000	1.000	1.000	0.995	0.995	1.000
ca	0.895	0.883	0.920	0.845	0.576	0.905	0.901	0.863	0.914	0.801	0.368	0.961	0.975	0.980	0.985	0.929	0.515	1.000

Appendix

Table 25: Open-ended generation quality on mDolly. We report the aggregated score by the LLM judge (averaged across four rubrics, see Section 4.3 ), the judge’s score assigned to the naturalness rubric, and language id line pass rate from FastText.

Language	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global	Gemma3-4B	Qwen3-4B	Qwen3.5-4B	Ministral-3-3B	SmolLM3-3B	Tiny Aya Global
	safe response rate $\uparrow$						invalid response rate $\downarrow$
ar	0.91	0.94	0.94	0.73	0.65	0.87	0.00	0.00	0.00	0.00	0.00	0.00
bn	0.88	0.89	0.92	0.72	0.00	0.88	0.00	0.00	0.01	0.00	0.75	0.00
en	0.93	0.99	0.97	0.79	0.70	0.93	0.00	0.00	0.00	0.00	0.00	0.00
it	0.91	0.95	0.94	0.65	0.67	0.90	0.00	0.00	0.00	0.00	0.00	0.00
jv	0.82	0.93	0.72	0.84	0.47	0.88	0.04	0.04	0.02	0.02	0.19	0.04

Appendix

Table 26: Results on MultiJail ( Deng et al., 2024 ) . We report the rate of safe and invalid responses.

Explore similar work

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi
Mar 3, 2026Shiza Fatimah, Aniket Sen, Sophia Falk +3Language Modeling Language Model Pretraining
Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Helmholtz-Institut für Strahlen- und Kernphysik +1
SozKZ: Training Efficient Small Language Models for Kazakh from Scratch
Mar 21, 2026Saken TukenovLanguage Model Pretraining Small Language Models
Independent Researcher
TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation
Mar 9, 2026Toms Bergmanis, Martins Kronis, Ingus Jānis Pretkalniņš +5Multilingual Language Models Open-Weight Language Models
Tilde, Latvia

Mar 3, 2026J/K move · Enter open · S save

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

Shiza Fatimah, Aniket Sen, Sophia Falk, Florian Mai+2

Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Helmholtz-Institut für Strahlen- und Kernphysik+1

Language Modeling Language Model Pretraining Small Language Models Low-Resource Language Processing

Open paper AskPDF arXiv

The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

Abstract

Explore similar work

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation