Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

Shiza Fatimah; Aniket Sen; Sophia Falk; Florian Mai; Lucie Flek; Nicholas Kluge Corrêa

cs.CLMar 3, 2026

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

Authors: Shiza Fatimah, Aniket Sen, Sophia Falk, Florian Mai, Lucie Flek, Nicholas Kluge Corrêa

Organizations: Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Helmholtz-Institut für Strahlen- und Kernphysik · Bonn Sustainable AI Lab

Abstract

The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

Figures & tables

Subset	Documents	Size	Tokens
default	83,081,507	260.68 GB	90,705,245,239
excluded	498,892	1.76 GB	1,545,290,236
Total	83,580,399	262.44 GB	92,250,535,475

Table 1: Statistics for subsets.

Tokenizer	Tokens (31500 words)	Vocab Size	Fertility	PCW	UNK
LilMoo	46129	49152	1.46	0.50	0
HindRoBERTa	50697	250002	1.61	0.31	0
Nemotron-Hindi-4B	52111	256000	1.65	0.30	0
IndicBERT	54213	200000	1.72	0.40	2
Gemma-2-2b	54379	256000	1.73	0.47	0
Llama-3.2	63891	128256	2.02	0.61	0

Table 2: Tokenizer Evaluation Metrics Comparison Across Multiple Tokenizers for Hindi (Ordered by Fertility)

Parameter	Value
Activation	SwiGLU
Hidden d (Residual Stream)	1,536
Feed-forward (MLP)	3,072
Context Length	4,096 tokens
Attention Heads	16
Attention Head d	96

Table 3: Architectural Specifications.

Parameter	Value / Description
Total batch size	2,097,152 tokens
Precision	BF16 with TF32 enabled
Optimizer	AdamW
Maximum learning rate	7e-4
Adam $\beta_{1},\beta_{2}$	0.9, 0.95
Adam $\epsilon$	1e-8

Table 4: Shared Training Hyperparameters for Both Recipes

Dataset Name	Subset	Size (Tokens)
GigaLekh		50B (Total)
	Edu Score of 3	34.95B
	Edu Score of 4	14.30B
	Edu Score of 5	163.96M
HuggingFaceFW/fineweb-edu		49.29B (Total)
	Edu Score of 3	35.00B

Table 5: Summary of Datasets Used in Multi-Stage Recipe

Stage	Hindi% (tokens)	English% (tokens)
Warmup+Stable	48.5 ( $\sim$ 70B)	51.5 ( $\sim$ 74B)
Stable	52 ( $\sim$ 81B)	48 ( $\sim$ 75B)
Stable+SqrtDecay	53 ( $\sim$ 44B)	47 ( $\sim$ 39B)

Table 6: Language Proportions Across Stages in Multi-Stage Recipe

Figure 1: Pretraining loss curve for the LilMoo pair.

Panel of Figure 2 — Figure 2: Gradient Statistics for the LilMoo pair.

Panel of Figure 3 — Figure 3: Per-benchmark performance across our evaluation suite (Qwen2.5/3 serve as Baselines).

Figure 4: NPM scores for both LilMoo models and both Qwen baselines.

Model	Aggregate NPM	MILU	CSQA	ARC	MMLU	HellaSwag	PIQA
LilMoo-v0.2	9.94	0.28	0.42	0.29	0.27	0.32	0.58
LilMoo-v0.1	8.70	0.28	0.42	0.25	0.26	0.32	0.58
Qwen3-1.7B	7.80	0.37	0.34	0.26	0.36	0.28	0.50
OpenHathi-7B-Hi-v0.1-Base	7.64	0.30	0.36	0.23	0.27	0.31	0.58
Airavata	6.63	0.28	0.36	0.22	0.27	0.29	0.58
gemma-2-2b-it	6.53	0.36	0.36	0.22	0.31	0.29	0.50

Table 7: Performance comparison against other models, beyond the Qwen2.5/3 baselines.

Model	Total Duration (hours)	GPU-hours	Energy consumed (kWh)	CO 2 emitted (kg CO 2 eq)	Total Compute (FLOPs)
LilMoo-v0.2	$\sim$ 327.50	$\sim$ 2648.7	1346.76	513.05	$\sim$ 1.285e21
LilMoo-v0.1	$\sim$ 346.97	$\sim$ 2797.1	1414.21	538.74	$\sim$ 1.357e21

Table 8: Training Resource Consumption

Figure 5: Performance vs Compute for LilMoo Models vs Qwen Baselines

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source Type	Dataset / Crawl	License(s)
Common Crawl	CC-MAIN-2025-30	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2025-26	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2025-05	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2024-51	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2023-50	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2022-49	ODC-By v1.0, CommonCrawl ToU

Appendix

Table B.1: Data sources organized by source type.

Model Name	Task	Precision	Recall	F1 Macro	Accuracy
HindRoBERTa	Educational	0.52	0.47	0.49	0.72
HindRoBERTa	Toxicity	0.59	0.45	0.47	0.70
IndicBERT	Educational	0.51	0.47	0.48	0.71
IndicBERT	Toxicity	0.56	0.43	0.44	0.69

Appendix

Table C.2: Evaluation results for educational-quality and toxicity classifiers on the test set using the original 5-class labels.

Dataset Name	Subset	Size (Tokens)	Repetition Factor
GigaLekh	Edu Score of 3	34.9B	2
HuggingFaceFW/fineweb-edu	Edu Score of 3	40.00B	1
HuggingFaceTB/finemath	Edu Score of 4	8.59B	2
allenai/big-reasoning-traces	All	2.44B	2
allenai/math-meta-reasoning-filtered ∗	All	1.24B	2
nvidia/OpenScience	All	9.7B	1

Appendix

Table D.3: Data mixture for Stage-1

Dataset Name	Subset	Size (Tokens)	Repetition Factor
GigaLekh	Edu Score of 3	34.9B	1.5
	Edu Score of 4	14.3B	2
HuggingFaceFW/fineweb-edu	Edu Score of 4	14.22B	1
HuggingFaceTB/smollm-corpus	All	29.4B	1
HuggingFaceTB/finemath	Edu Score of 4	8.59B	1
	Edu Score of 5	1.08B	2

Appendix

Table D.4: Data mixture for Stage-2

Dataset Name	Subset	Size (Tokens)	Repetition Factor
GigaLekh	Edu Score of 4	14.3B	3
	Edu Score of 5	0.16B	5
HuggingFaceFW/fineweb-edu	Edu Score of 5	0.068B	5
HuggingFaceTB/smollm-corpus	All	29.4B	1
HuggingFaceTB/finemath	Edu Score of 5	1.08B	3
allenai/big-reasoning-traces	All	2.44B	2

Appendix

Table D.5: Data mixture for Stage-3

Benchmark	n-shot	Baseline	Metric
ARC	5-shot	25	acc
MMLU	5-shot	25	acc
HellaSwag	5-shot	25	acc
TruthfulQA	0-shot	22.5	mc1
MILU	5-shot	25	acc
CSQA	5-shot	25	acc

Appendix

Table E.6: Summary of Evaluation Benchmarks

Panel of Figure 20 — Figure E.1: Per-benchmark Performance across training checkpoints.

Figure E.2: NPM aggregated score of all benchmarks.

Benchmark	Baseline	Threshold	First Step	Tokens (B)	Score	%
ARC	0.250	0.300	80000	167.770	0.301	+0.051
HellaSwag	0.250	0.300	5000	10.490	0.311	+0.061
MMLU	0.250	0.300	Not Reached	-	-	-
TruthfulQA MC1	0.225	0.275	Not Reached	-	-	-
COPA	0.500	0.550	10000	20.970	0.559	+0.059
IITP MR	0.330	0.380	5000	10.490	0.387	+0.057

Appendix

Table E.7: Training tokens required for v0.1 to surpass random baselines by 5%.

Panel of Figure 23 — Figure E.3: Noise analysis for the 10 benchmarks in our Hindi evaluation suite.

Explore similar work

Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language
Jun 15, 2026Rabindra Nath NandiDecoder-Only Language Models Small Language Models
Independent Researcher
Tiny Aya: Bridging Scale and Multilingual Depth
Mar 12, 2026Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza +23Multilingual Language Models Machine Translation
Cohere Labs · Cohere
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE
May 18, 2026Hao Zhou, Tianhao Li, Zhijun Wang +6Multilingual Language Models Mixture-of-Experts Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Tongyi Lab, Alibaba Group · Zhejiang University

Jun 15, 2026J/K move · Enter open · S save

Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language

Rabindra Nath Nandi

Independent Researcher

Decoder-Only Language Models Small Language Models Low-Resource Language Processing

Open paper AskPDF arXiv

While the NLP landscape is dominated by multi-billion parameter architectures, their deployment in low-resource, non-Latin scripts remains computationally prohibitive for edge configurations, mobile systems, and decentralized local hardware. This paper presents bangla-smollm-135m, a highly compact 135-million parameter decoder-only foundational model engineered explicitly for high-efficiency language modeling in the Bangla script. By leveraging a deterministic intersect-and-append token merging strategy between TituLLMs and SmolLM2-135M, the model overcomes subword script fragmentation without destabilizing early pretrained parameter states. In zero-shot multi-task benchmark evaluations (PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, and Bangla_MMLU), bangla-smollm-135m matches or outperforms models twice its size (Gemma-3-270m) and achieves parity with models in the 1B parameter tier. The model is available at rnnandi/bangla-smollm-135m

Abstract

Explore similar work

Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language

Tiny Aya: Bridging Scale and Multilingual Depth

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔΔΔ Integration into Upcycled MoE

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE