Lara Mini Base: Language Diversity as a Resource, Not a Trade-off
Abstract
General-purpose language models are trained mostly on English: over open models we review, at most, non-English pretraining data makes up only a 12% share. Models built for European languages give them far more, but score well below general-purpose models of the same size. A common concern is that, in small models, languages compete for capacity. We present Lara Mini base, a model trained from scratch on 6T tokens in English and 44 other languages, which together cover all 24 official languages of the European Union. As training progresses, the 44 other languages together come to outweigh English. To give these languages enough capacity at low cost, Lara Mini base uses a mixture of experts architecture with 7.2B total and 0.6B active parameters. Among open base models with up to 2B active parameters, it achieves the highest average score across standard English, multilingual, math, and code benchmarks, ahead of models trained on 4 to 6 times as many tokens. Its margin is largest on multilingual benchmarks in the languages it supports and on math benchmarks. It supports contexts of up to 128k tokens. We find that our MoE architecture reduces capacity competition between languages. For a single user on one GPU, without quantization, it generates text at least 1.8 times faster than any model with a higher average score. We release the weights of Lara Mini base under the CC BY-NC-SA 4.0 license. In addition, we will release T-HellaSwag, a higher-quality machine translation of the commonsense reasoning benchmark HellaSwag into 110 language variants.
Figures & tables
| Model | Active (total) parameters | Training tokens | Multilingual share | Multilingual tokens |
|---|---|---|---|---|
| General-purpose models | ||||
| SmolLM2 ( Ben Allal et al., 2025 ) | 1.7B | 11T | 0% | – |
| Olmo 3 ( Team Olmo, 2025 ) | 7B, 32B | 5.9T | 0% | – |
| Nemotron Nano 2 ( NVIDIA, 2025 ) | 12B | 20T | 5% | 1.0T |
| Llama 3 ( Grattafiori et al., 2024 ) | 8B–405B | 15.6T | 8% | 1.2T |
| SmolLM3 ( Hugging Face, 2025c ) | 3B | 11.2T | 12% | 1.3T |
| Domain | Main sources | Pool | Training |
|---|---|---|---|
| English | Nemotron-CC and its synthetic data ( Su et al., 2025 ) , FineWeb-Edu ( Penedo et al., 2024 ) , FinePDFs ( Hugging Face, 2025a ) , FineWiki ( Hugging Face, 2025b ) | 5.65T | 2.49T (41.4%) |
| Multilingual | FineWeb-2 ( Penedo et al., 2025 ) , FineWeb2-HQ ( Messmer et al., 2025 ) , HPLT 3.0 ( Oepen et al., 2026 ) , CulturaX ( Nguyen et al., 2024 ) , FinePDFs, FineWiki, synthetic question-answer data | 6.31T | 2.18T (36.4%) |
| Code | Nemotron-CC code pages, The Stack v3 ( Hugging Face, 2026 ) (permissive files), Nemotron synthetic code | 0.90T | 0.77T (12.8%) |
| Math | FineMath ( Ben Allal et al., 2025 ) , MegaMath ( Zhou et al., 2025 ) , Nemotron-CC-Math ( Karimi Mahabadi et al., 2026 ) | 0.33T | 0.35T (5.9%) |
| SFT-like | Nemotron pretraining collections ( NVIDIA, 2025 ; NVIDIA, 2026a ; NVIDIA, 2026b ) | 0.61T | 0.21T (3.5%) |
| Total | 13.8T | 6T |
| Decision | Choice | Basis |
|---|---|---|
| English : multilingual | 54:46 between English and multilingual text, excluding code, math and SFT-like data (main stage) | Most high-quality and synthetic data are English; in small proxy runs, shifting tokens between English and multilingual text hurt the side that lost them more than it helped the other |
| Code | 12%, 14.5%, 17% | Code improves natural-language reasoning ( Aryabumi et al., 2025 ) ; SmolLM3 uses 12–24% ( Hugging Face, 2025c ) , Llama 3 17% ( Grattafiori et al., 2024 ) |
| Math | 5%, 8%, 11% | Limited by supply: about 330B unique tokens |
| SFT-like data | 2%, 7%, 12% | High-quality and question-answer data help most at the end of training ( Blakeney et al., 2024 ; Grattafiori et al., 2024 ; Team OLMo, 2025 ; NVIDIA, 2025 ) |
| Multilingual anneal | 33%, mostly high-quality web and question-answer data | In a 100B-token continuation of a 1B model, translated question-answer data scored 10 points higher on Global MMLU than heuristically filtered Common Crawl text ( NVIDIA, 2025 ) |
| Hyperparameter | Value |
|---|---|
| Layers | 34 |
| Gated DeltaNet | 14 |
| Sliding-window attention | 13 (window of 512 tokens) |
| Global attention | 7 (no positional encoding) |
| Hidden size | 1,024 |
| Attention heads (query / key-value) | 16 / 4, dimension 128 |
| Hyperparameter | Value |
|---|---|
| Optimizer | Muon + AdamW |
| Peak learning rate | |
| Warmup | 2,000 steps (17B tokens) |
| Schedule | constant to 90% of training, then 1 sqrt decay to 0 |
| Global batch size | 2,048 sequences ( 8.4M tokens) |
| Sequence length | 4,096 |
| Parameters | Domain average | |||||||
| Model | Active | Total | Tokens | English | Multilingual | Math | Code | Overall |
| Models with more than 4B active parameters | ||||||||
| Llama-3.1-8B | 8.0B | 8.0B | 15T | 69.6 | 55.2 | 36.6 | 43.6 | 51.2 |
| Apertus-v1.5-8B | 8.0B | 8.0B | 19T | 69.0 | 59.2 | 50.6 | 44.3 | 55.8 |
| Qwen2.5-7B | 7.6B | 7.6B | 18T | 72.3 | 57.3 | 64.9 | 55.5 | 62.5 |
| Qwen3-8B | 8.2B | 8.2B | 36T | 71.9 | 59.2 | 71.3 | 61.0 | 65.9 |
| Up to 2B active parameters | Larger models | |||||||
| Benchmark | LFM2.5 8B-A1B | Hunyuan 1.8B | Falcon-H1 1.5B-Deep | Qwen3 1.7B | granite-4.0 h-tiny | Lara Mini base | Qwen3 4B 2–4B | Qwen3 8B 4B |
| English | ||||||||
| NLI | 52.3 | 50.7 | 55.9 | 54.2 | 54.2 | 52.7 | 54.4 | 46.8 |
| PAWS | 62.9 | 71.0 | 68.0 | 61.2 | 63.3 | 65.3 | 68.6 | 55.5 |
| Belebele | 69.1 | 75.4 | 70.9 | 75.9 | 69.9 | 76.0 | 89.3 | 91.3 |
| COPA | 85.4 | 78.6 | 82.2 | 82.4 | 85.4 | 84.8 | 85.8 | 89.0 |
| Parameters | RULER | ONERULER | HELMET | |||||
| Model | Active | Total | 32k | 128k | 32k | 128k | 32k | 128k |
| End of the anneal | 0.6B | 7.2B | 15.0 | 10.9 | 0.4 | 0.0 | 9.3 | 8.9 |
| Lara Mini base | 0.6B | 7.2B | 76.6 | 63.7 | 90.1 | 80.9 | 63.0 | 54.7 |
| Trinity-Nano | 1.0B | 6.1B | 71.0 | 61.8 | 73.9 | 62.1 | 54.9 | 51.8 |
| granite-4.0-h-tiny | 1.0B | 6.9B | 29.5 | 26.4 | 19.5 | 14.1 | 41.0 | 36.3 |
| Falcon-H1-1.5B-Deep | 1.6B | 1.6B | 59.7 | 38.7 | 60.5 | 38.8 | 45.5 | 34.4 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Group | Sources | Licenses |
|---|---|---|---|
| English | Web, high quality | Nemotron-CC high and medium-high (v1 and v2), FineWeb-Edu | NVIDIA DA, ODC-BY |
| Web, medium quality | Nemotron-CC medium (v1 and v2) | NVIDIA DA | |
| Synthetic | Nemotron-CC high-quality synthetic, v2 synthetic and diverse QA | NVIDIA DA | |
| PDFs | FinePDFs (English) | ODC-BY | |
| Wikipedia | FineWiki (English) | CC BY-SA 4.0, GFDL | |
| Multilingual | Web | HPLT 3.0, FineWeb-2, CulturaX | CC0, ODC-BY |
| Group | Pool | Main | Transition | Anneal | Epochs |
|---|---|---|---|---|---|
| English web, high quality | 1391.4 | 1152.9 | 108.2 | 76.1 | 0.96 |
| English web, medium quality | 1100.0 | 143.5 | 7.2 | 0.0 | 0.14 |
| English synthetic, high quality | 1916.3 | 538.9 | 64.5 | 56.7 | 0.34 |
| English PDFs | 1236.6 | 266.1 | 29.4 | 24.3 | 0.26 |
| English Wikipedia | 8.2 | 10.6 | 3.7 | 4.9 | 2.34 |
| Multilingual web | 3326.5 | 724.7 | 50.8 | 12.8 | 0.24 |
| Language | Class | Tokens | Epochs | Language | Class | Tokens | Epochs |
|---|---|---|---|---|---|---|---|
| Russian | P | 177.2 | 0.29 | Ukrainian | P | 31.4 | 0.16 |
| Spanish | EU, P | 169.2 | 0.33 | Thai | P | 31.4 | 0.16 |
| German | EU, P | 162.4 | 0.35 | Romanian | EU | 30.5 | 0.44 |
| French | EU, P | 155.7 | 0.37 | Slovak | EU, P | 30.5 | 0.37 |
| Chinese (Simpl.) | P | 138.3 | 0.48 | Persian | – | 28.3 | 0.30 |
| Italian | EU, P | 119.3 | 0.44 | Croatian | EU, P | 25.7 | 0.44 |
| 32k step | 128k step | Overall | |
| Sequence length (tokens) | 32,768 | 131,072 | – |
| Global batch size (sequences) | 256 | 64 | – |
| Steps | 4,768 | 7,153 | 11,921 |
| Tokens | 40B | 60B | 100B |
| Anneal mixture, as is | 33.3% | 50.0% | 43.3% |
| Anneal mixture, reweighted toward long documents | 25.5% | 9.0% | 15.6% |
| Layers | Routed experts | Shared | Active | Total | Loss | English | Multilingual |
| 28 | dense | 0.60B | 0.60B | 2.431 | 50.4 | 36.9 | |
| 28 | dense | 1.24B | 1.24B | 2.285 | 53.7 | 38.3 | |
| 24 | , top-4 | – | 0.51B | 3.91B | 2.201 | 55.8 | 38.8 |
| 24 | , top-4 | – | 0.51B | 7.53B | 2.160 | 56.3 | 39.5 |
| 34 | , top-4 | – | 0.55B | 7.18B | 2.159 | 56.7 | 39.7 |
| 34 | , top-4 | 0.60B | 7.24B | 2.140 | 57.3 | 40.0 | |
| Token mixers | loss | benchmarks | Memory at 32k |
|---|---|---|---|
| Global attention (34 layers) | – | – | 100% |
| Gated DeltaNet (26 layers) | 0.017 | 0.37 | 24.2% |
| Sliding window (26 layers) | 0.002 | 0.17 | 24.7% |
| Gated DeltaNet (14) + sliding window (13) | 0.015 | 0.67 | 21.5% |
| Oracle: better of Gated DeltaNet and sliding window | – | 0.65 | – |
| Family | Benchmarks | Sliding window | Hybrid |
|---|---|---|---|
| Inference | NLI, XNLI | 0.59 | 0.53 |
| Paraphrase | PAWS, PAWS-X | 0.26 | 0.05 |
| Causality | COPA, XCOPA | 0.58 | 0.17 |
| Physical | PIQA, Global PIQA | 0.39 | 0.35 |
| Completion | HellaSwag, T-HellaSwag | 0.96 | 0.07 |
| Coreference | WinoGrande | 2.84 | 0.16 |
| RULER | ONERULER | HELMET | |||||
|---|---|---|---|---|---|---|---|
| Token mixers | Short | 16k | 32k | 16k | 32k | 16k | 32k |
| Gated DeltaNet (26 layers) | 49.66 | 57.3 | 46.1 | 63.5 | 55.9 | 44.9 | 39.0 |
| Sliding window (26 layers) | 49.51 | 55.9 | 48.3 | 65.1 | 60.1 | 40.0 | 35.1 |
| Hybrid (14 + 13 layers) | 50.13 | 57.5 | 52.8 | 71.7 | 64.0 | 43.4 | 39.6 |
| Domain | Benchmark | Shots | Metric | Lang. |
| English | XNLI (English) ( Conneau et al., 2018 ) | 0 | Acc. | – |
| PAWS (English) ( Zhang et al., 2019 ) | 0 | Acc. | – | |
| Belebele (English) ( Bandarkar et al., 2024 ) | 0 | Acc. | – | |
| COPA ( Roemmele et al., 2011 ) | 0 | Acc. | – | |
| PIQA ( Bisk et al., 2020 ) | 0 | Acc. (norm.) | – | |
| HellaSwag ( Zellers et al., 2019 ) | 10 | Acc. (norm.) | – |
| Parameters | Domain average | ||||||
| Model | Active | Total | English | Multilingual | Math | Code | Overall |
| Models with more than 4B active parameters | |||||||
| Teuken-7B | 7.5B | 7.5B | 59.8 | 47.1 | 7.4 | 19.1 | 33.4 |
| Mistral-7B-v0.3 | 7.3B | 7.3B | 68.1 | 49.0 | 24.6 | 37.4 | 44.8 |
| Llama-3-8B | 8.0B | 8.0B | 68.8 | 53.5 | 35.8 | 43.3 | 50.4 |
| Gemma-7B | 8.5B | 8.5B | 68.5 | 55.2 | 39.0 | 40.5 | 50.8 |
| Model | NLI | PAWS | Bele. | COPA | PIQA | HS | WG | ARC-C | MMLU | MMLU-P | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Models with more than 4B active parameters | |||||||||||
| Teuken-7B | 56.9 | 59.4 | 45.9 | 86.0 | 78.7 | 77.4 | 71.7 | 54.0 | 49.8 | 17.9 | 59.8 |
| Mistral-7B-v0.3 | 55.3 | 68.4 | 72.1 | 88.2 | 82.1 | 83.2 | 78.8 | 60.2 | 62.5 | 30.5 | 68.1 |
| Llama-3-8B | 54.8 | 70.1 | 75.2 | 87.8 | 80.7 | 82.3 | 78.5 | 58.4 | 65.5 | 34.7 | 68.8 |
| Gemma-7B | 52.9 | 69.2 | 76.0 | 89.4 | 81.0 | 82.4 | 77.4 | 60.0 | 63.1 | 33.5 | 68.5 |
| Llama-3.1-8B | 55.3 | 70.4 | 81.0 | 88.2 | 80.9 | 82.3 | 78.3 | 58.0 | 65.3 | 35.8 | 69.6 |
| Model | XNLI | PAWS-X | Bele. | XCOPA | GP-NP | GP-P | T-HS | INCL. | G-MMLU | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Models with more than 4B active parameters | ||||||||||
| Teuken-7B | 44.2 | 54.6 | 34.8 | 61.1 | 66.1 | 29.9 | 51.9 | 43.2 | 37.9 | 47.1 |
| Mistral-7B-v0.3 | 42.6 | 59.3 | 47.3 | 58.8 | 62.7 | 28.4 | 48.5 | 47.0 | 46.3 | 49.0 |
| Llama-3-8B | 45.7 | 59.8 | 50.4 | 65.7 | 66.2 | 32.5 | 54.9 | 54.2 | 52.1 | 53.5 |
| Gemma-7B | 43.5 | 59.7 | 57.4 | 68.2 | 69.0 | 33.0 | 58.5 | 54.8 | 52.8 | 55.2 |
| Llama-3.1-8B | 46.2 | 60.8 | 60.3 | 67.1 | 66.5 | 32.6 | 55.2 | 54.7 | 52.9 | 55.2 |
| Model | XNLI | PAWS-X | Bele. | XCOPA | GP-NP | GP-P | T-HS | INCL. | G-MMLU | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Models with more than 4B active parameters | ||||||||||
| Teuken-7B | 42.8 | 54.6 | 30.2 | 57.8 | 57.4 | 24.9 | 41.9 | 40.4 | 36.6 | 43.0 |
| Mistral-7B-v0.3 | 41.5 | 59.3 | 36.1 | 56.7 | 56.5 | 24.9 | 40.5 | 44.0 | 44.2 | 44.9 |
| Llama-3-8B | 44.4 | 59.8 | 40.3 | 62.0 | 59.7 | 27.1 | 46.8 | 51.7 | 50.0 | 49.1 |
| Gemma-7B | 42.9 | 59.7 | 43.1 | 64.1 | 61.3 | 27.5 | 48.7 | 53.1 | 50.6 | 50.1 |
| Llama-3.1-8B | 44.9 | 60.8 | 45.7 | 62.7 | 60.0 | 27.6 | 46.9 | 52.1 | 50.6 | 50.1 |
| Math | Code | |||||||
| Model | GSM8K | MATH | MGSM | MBPP | MBPP+ | CRUX-I | CRUX-O | Avg. |
| Models with more than 4B active parameters | ||||||||
| Teuken-7B | 10.9 | 4.4 | 7.0 | 13.4 | 14.0 | 22.4 | 26.6 | 13.3 |
| Mistral-7B-v0.3 | 36.4 | 14.0 | 23.5 | 38.0 | 41.5 | 36.0 | 34.1 | 31.0 |
| Llama-3-8B | 51.0 | 19.0 | 37.2 | 49.8 | 53.4 | 32.9 | 37.3 | 39.6 |
| Gemma-7B | 51.9 | 24.8 | 40.2 | 45.2 | 47.1 | 33.3 | 36.5 | 39.7 |
| Language | Bele. | G-MMLU | INCL. | XNLI | PAWS-X | XCOPA | GP-NP | GP-P | T-HS | Avg. | Peer |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Russian | 67.1 | – | 54.4 | 48.6 | – | – | 76.0 | 31.1 | 63.5 | 56.8 | 56.2 b |
| Spanish | 64.3 | 59.6 | 64.0 | 45.9 | 63.8 | – | 87.3 | 41.4 | 62.0 | 61.0 | 58.8 f |
| German | 62.9 | 58.9 | 51.8 | 47.9 | 59.2 | – | 77.0 | 36.9 | 56.0 | 56.3 | 54.7 f |
| French | 63.2 | 58.8 | 60.9 | 49.8 | 59.3 | – | 75.0 | 43.7 | 60.8 | 58.9 | 55.7 b |
| Chinese (Simpl.) | 65.7 | 57.3 | 51.2 | 36.7 | 53.6 | 70.6 | 58.0 | 35.0 | 57.5 | 53.9 | 56.6 d |
| Italian | 67.2 | 59.4 | 71.2 | – | – | 71.8 | 77.0 | 39.8 | 58.0 | 63.5 | 60.0 f |
| Benchmark | What we changed | Why |
|---|---|---|
| RULER | Fixed samples: 100 per task and length, generated once, with lengths counted in each model’s tokens | The harness draws new random samples at every run; fixed samples give every model the same inputs |
| ONERULER | No answer template at the end of the prompt; instead, the prompt ends with the start of the answer, the needle sentence up to the number (“The special magic number for … is:”) | Base models copy the answer template instead of answering |
| In the task with four numbers for one key, the start of the answer states how many there are (“The 4 special magic numbers for … are:”) | Without the count, models often stop after one to three numbers, so the score measures when they stop rather than what they find | |
| An answer is correct if it contains the right numbers, whatever text follows them | The original scoring gives zero to any answer that contains the word for “none”, which a base model often copies from the question after a correct answer | |
| The task whose correct answer is “none” (no needle has the key asked for) is left out | Without examples in the prompt, the base models we tested did not answer “none”; with examples, the score depends on how closely a model follows them rather than on what it retrieves, and a model can then answer “none” even when the needle is there | |
| The two common-word tasks are left out | They score close to zero for every model from 16k tokens |
| MMLU | MBPP | |||||
|---|---|---|---|---|---|---|
| Model | Clean | Contam. | Clean | Contam. | ||
| Llama-3.2-1B | 31.5 | 30.8 | 39.5 | 23.9 | ||
| granite-4.0-h-tiny | 64.4 | 66.4 | 65.4 | 57.0 | ||
| Qwen3-1.7B | 61.5 | 63.4 | 66.7 | 53.2 | ||
| Lara Mini base | 63.6 | 63.4 | 61.7 | 52.5 | ||
| MMLU | MBPP | |||||
|---|---|---|---|---|---|---|
| Model | Clean | Contam. | Ratio | Clean | Contam. | Ratio |
| Llama-3.2-1B | 0.847 | 0.816 | 0.964 | 0.562 | 0.429 | 0.763 |
| granite-4.0-h-tiny | 0.855 | 0.847 | 0.990 | 1.207 | 0.972 | 0.806 |
| Qwen3-1.7B | 0.826 | 0.776 | 0.940 | 0.408 | 0.306 | 0.749 |
| Lara Mini base | 0.790 | 0.776 | 0.982 | 0.507 | 0.387 | 0.763 |
| 1 user | 64 users | 256 users | 128k | |||||
| Model | Active | Chars/s | TTFT | k chars/s | TTFT | k chars/s | TTFT | k chars/s |
| Models with more than 4B active parameters | ||||||||
| Mistral-7B-v0.3 † | 7.25 | 257 | 77.6 | 5.4 | 487 | 5.5 | 939 | – |
| Gemma-7B † | 8.54 | 258 | 81.9 | 4.5 | 509 | 4.4 | 36,924 | – |
| Teuken-7B † | 7.45 | 274 | 66.4 | 7.1 | 437 | 6.9 | 1,054 | – |
| Qwen3-8B † | 8.19 | 281 | 74.2 | 5.9 | 499 | 5.7 | 3,799 | 9.2 |
| 44 non-English languages supported by Lara Mini base (48 variants) | Arabic, Azerbaijani, Bengali, Bosnian, Bulgarian, Catalan, Chinese (Simplified), Chinese (Traditional), Croatian, Czech, Danish, Dutch, Estonian, Finnish, French (Canada), French (France), German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Maltese, Norwegian (Bokml), Persian, Polish, Portuguese (Brazil), Portuguese (Portugal), Romanian, Russian, Serbian (Cyrillic), Serbian (Latin), Slovak, Slovenian, Spanish (Mexico), Spanish (Spain), Swedish, Thai, Turkish, Ukrainian, Vietnamese |
|---|---|
| Other languages (62 variants) | Afrikaans, Albanian, Amharic, Armenian, Assamese, Basque, Belarusian, Bhojpuri, Burmese, Cebuano, Chichewa, Galician, Georgian, Gujarati, Haitian Creole, Hausa, Icelandic, Igbo, Javanese, Kannada, Kazakh, Khmer, Kinyarwanda, Kurdish (Kurmanji), Kurdish (Sorani), Kyrgyz, Lao, Lingala, Luganda, Luxembourgish, Macedonian, Maithili, Malagasy, Malayalam, Marathi, Mongolian, Nepali, Northern Sotho, Odia, Oromo (West Central), Pashto, Punjabi, Shona, Sindhi, Sinhala, Somali, Southern Sotho, Sundanese, Swahili, Tagalog, Tajik, Tamil, Telugu, Tswana, Urdu, Uyghur, Uzbek, Welsh, Xhosa, Yiddish, Yoruba, Zulu |
| Developer | Translated |
|---|---|
| Model type | Decoder-only mixture of experts base model; hybrid of Gated DeltaNet, sliding-window and global attention ( Section 3 ) |
| Parameters | 7.2B total, 0.6B active per token (embeddings included) |
| Layers | 34: 14 Gated DeltaNet (GDN), 13 sliding-window attention (window 512, RoPE), 7 global attention (no positional encoding) |
| Layer pattern | GDN, sliding, GDN, global, then (sliding, GDN, sliding, GDN, global) 6 |
| Hidden size | 1,024 |
| Attention | 16 query heads, 4 key-value heads, head dimension 128, QK-norm |
Multilingual Language Models
Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.
See all 276 papers