cs.CLOct 19, 2026
SaveScaling Synthetic Pre-training Data and Evaluation for Indic LLMs
Organizations: NVIDIA · AI4Bharat
Abstract
Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.
Figures & tables
| Corpus | # Toks (B) | # Indic Langs | Type | Roman | Data Open? | Pipeline Open? |
| Sangraha [ 41 ] | 251 | 22 | Mixed | ✓ | ✓ | ✓ |
| BhashaKritika [ 57 ] | 540 | 10 | Synth | – | ❍ | ✗ |
| IndicCorp-v2 [ 16 ] | 20.9 | 23 | Native | ✗ | ✓ | ✗ |
| FineWeb2 [ 62 ] | – | 22 | Native | ✗ | ✓ | ✓ |
| Sarvam-1 [ 70 ] | 2,000 | 10 | ✗ | ✗ | ✗ | ✗ |
| Param-1 [ 65 ] | 2,770 | 1 | Mixed | – | ✗ | ✗ |
Table 2.1: Comparison of Indic pretraining corpora. # Toks (B): billions of Indic-language tokens as per each source’s own tokenizer and reports. Type: Native is human-authored text (web, books, OCR, transcripts); Synth is machine-generated or machine-translated text; Mixed combines both. Roman: whether the corpus includes romanized Indic text. Data Open?: whether the corpus is publicly downloadable. Pipeline Open?: whether the code used to collect, filter or generate the data is public. ✓yes; ✗no; ❍partially released.
Figure 3.1 : Released target tokens per language, complementing Table 3.1 . Bars report Gemma-4 target tokens in billions. For the twelve priority languages (Variant A), each native-script bar (left) and romanized bar (right, hatched) is split into tokens accepted directly by the quality judge (first pass) and tokens recovered through the rewriting stage. Variants B and C have no rewrite stage, so only totals are shown. The two rows use different vertical scales.
| Source (English) | Target (Gemma-4 tokens, B) | ||||||||
| Language | Config | Script | Tokens (B) | 1st pass (%) | 1st pass | Rewrite | Total | Exp. | Docs (M) |
| Variant A: Gemma-4-31B-it pipeline (Sections 4.1 to 4.4 ); native-script and romanized configurations | |||||||||
| Bengali | bn-Beng-IN | Bengali | 114.0 | 88 | 124.9 | 14.0 | 138.9 | 1.22 | 166.3 |
| bn-Latn-IN | Latin | 12.2 | 84 | 15.6 | 3.4 | 19.0 | 1.56 | 16.7 | |
| Gujarati | gu-Gujr-IN | Gujarati | 115.5 | 83 | 168.8 | 27.6 | 196.4 | 1.70 | 169.3 |
| gu-Latn-IN | Latin | 12.0 | 89 | 15.2 | 2.0 | 17.2 | 1.43 | 16.5 | |
Table 3.1 : Overview of the released pretraining corpus by language–script configuration. Source reports the number of English tokens corresponding to the released documents, while Target reports their translated token counts using the Gemma-4 tokenizer. 1st pass denotes examples accepted directly by the quality judge, and Rewrite denotes examples recovered through the rewriting stage. Exp. is the ratio of target to source tokens, and Docs denotes the number of released documents. Romanized ( Latn ) configurations are transliterations of subsets of the corresponding native-script data and therefore provide additional script coverage rather than additional content. We describe this corpus counting methodology in more detail in Appendix B .
| Subset | Share (%) | Subset | Share (%) |
| cc_v2_diverse_qa | 40.40 | specialized_v1_math_textbooks | 2.97 |
| pretraining_sft_v1 | 12.03 | specialized_v1_wiki_rewrite | 2.68 |
| cc_v2_english_cc | 9.56 | specialized_v1_stem_sft | 1.61 |
| cc_math_v1_4plus | 8.65 | specialized_v11_multiple_choice | 1.08 |
| cc_v21_hq_synthetic | 7.46 | specialized_v11_formal_logic | 0.15 |
| cc_v21_hq_organic | 5.59 | specialized_v11_economics | 0.10 |
Table 3.2 : Nemotron-CC v2 subsets sampled for translation and their shares of the sampling target (percent). Names are the subset identifiers of the Nemotron-CC v2 release. The retrieval-style question-answer subset specialized_v1_rqa (1.00 percent in the initial plan) was dropped before sampling and the remaining shares renormalised.
Figure 4.1 : Workflow for the top-12 Indic languages (Variant A in Section 3.2). Gemma-4-31B-it translates whole English documents with fewer than 4,096 Gemma-4 tokens, assesses the translations, and rewrites low-scoring candidates before reassessment. Corpus assembly retains native translations that meet the score threshold of 75. A subset of the retained native-script text is romanized and checked against its native-script input. NeMo Data Designer runs generation and assessment; corpus acceptance follows these steps. Appendix D describes inference optimizations and execution details.
Figure 4.2 : Workflows for the remaining ten Indic languages (Variants B and C in Section 3.2), using Bodhan-MT for translation and Gemma-4-31B-it for quality assessment. Eight languages use whole-document translation. Longer Santali documents use overlapping windows followed by reconstruction. Manipuri uses Bengali-script translation followed by conversion to Meitei Mayek, with assessment before and after conversion. Corpus assembly applies language-specific acceptance rules; these workflows do not use the top-12 feedback-rewrite loop. Appendix D describes inference optimizations and execution details.
| Label | Pool | Description | Available tokens (billions) |
| SG-V | Sangraha verified | Web text from human-verified sources | 85.6 |
| SG-U | Sangraha unverified | Perplexity-filtered web text | 32.6 |
| SG-T | Sangraha translated | English Wikimedia translated with IndicTrans2 | 139.0 |
| FW+FP | FineWeb2 + FinePDFs | Web text and PDF-extracted text | 94.8 |
| NPI-N | NPI-v1 , native script | First-pass accepted translations, score at least 75 | Varies by study |
| NPI-R | NPI-v1 , romanized | First-pass accepted romanized text | Varies by study |
Table 5.1: Data available for constructing the Indic training mixtures. Counts describe the source pools, before sampling for a training run. NPI-N and NPI-R identify the native-script and romanized subsets of NPI-v1 (Section 3.1).
| Comparison | What changes | Total budget | Indic budget |
| Public Indic corpora ( 5.4.1 ) | Allocation across natural and translated sources | 100B | 45B |
| NPI-v1 contribution ( 5.4.2 ) | Mixtures with NPI-v1 data; script mix | 100B | 45B |
| Pool size ( 5.4.3 ) | Punjabi or Malayalam pool configurations | 100B | 45B |
| Romanized Malayalam ( 5.4.4 ) | Native-script versus romanized training recipes | 100B | 45B |
| Indic share ( 5.4.5 ) | 10%, 20%, or 30% Indic sampling | 200B | 20B/40B/60B |
| English results ( 5.4.6 ) | Evaluation of the same Indic-share runs | 200B | 20B/40B/60B |
Table 5.2: Experiment overview. Budgets denote sampled training tokens, not available corpus size.
| Recipe | Indic tokens (B) | MILU |
| Base | – | 47.1 |
| FW+FP 45 | 45 | 61.7 |
| SG-T 45 | 45 | 60.8 |
| SG-V 33 + SG-U 12 | 45 | 62.0 |
| SG-V 18 + SG-U 6 + FW+FP 21 | 45 | 62.1 |
| SG-V 15 + SG-U 4 + SG-T 8 + FW+FP 18 | 45 | 63.3 |
Table 5.3: Source survey: 100B-token runs with 45B Indic tokens. Recipe numbers are percentages of all training tokens; the remaining 55% uses the base mixture. MILU is averaged over eleven languages, including English.
| Reported inclusive means | ||||
| Recipe | MILU | MMLU-Pro | ARC-C | GSM8K |
| Base | 44.4 | 29.3 | 53.7 | 40.4 |
| Public-corpora baseline | 63.2 | 40.2 | 84.2 | 67.7 |
| NPI-N mixture | 64.5 | 41.8 | 87.6 | 75.7 |
| NPI-N + NPI-R mixture | 64.3 | 42.0 | 87.3 | 75.1 |
| Gemma-26B | 64.7 | 34.8 | 87.3 | 61.9 |
Table 5.4: Mixtures containing NPI-v1 data at a 100B-token budget and 45% Indic share. The second panel excludes Odia as a sensitivity analysis. Recipe shares are percentages of all training tokens.
| Punjabi | ||||
| Pool configuration | MILU | MMLU-Pro | ARC-C | GSM8K |
| Web only | 59.06 | 38.05 | 81.45 | 66.72 |
| 10B | 63.06 | 42.99 | 87.49 | 78.24 |
| 20B | 65.24 | 43.96 | 89.04 | 78.85 |
| 30B | 65.80 | 44.30 | 89.04 | 79.83 |
| Gemma-26B | 65.58 | 34.90 | 88.20 | 64.70 |
Table 5.5: Single-language pool-size comparisons. “Web only” contains Indic data but excludes our synthetic data.
| Training recipe | MILU, romanized | GSM8K, romanized |
| Base | 44.25 | 19.64 |
| Sangraha web | 45.73 | 21.91 |
| Sangraha + NPI-N (50B pool) | 47.00 | 30.10 |
| Sangraha + NPI-R | 64.43 | 73.09 |
| Gemma-26B | 56.84 | 63.46 |
Table 5.6: Performance on romanized Malayalam MILU and GSM8K. GSM8K uses ASCII-digit scoring.
| Configuration | MILU | MMLU-Pro | ARC-C | GSM8K | Mean-4 |
| Base | 46.8 | 33.5 | 57.5 | 44.1 | 45.5 |
| Indic 10% (20B) | 58.8 | 41.9 | 82.2 | 69.5 | 63.1 |
| Indic 20% (40B) | 60.2 | 43.0 | 83.9 | 71.4 | 64.6 |
| Indic 30% (60B) | 61.1 | 43.8 | 85.0 | 72.3 | 65.6 |
| Gemma-26B | 64.7 | 34.8 | 87.3 | 61.9 | 62.2 |
Table 5.7: Indic-share study: 200B total training tokens. Indic token allocations include the full Indic mixture, not only our synthetic data. Mean-4 is the arithmetic mean of the four displayed benchmark scores.
| Benchmark and metric | 10% Indic | 20% Indic | 30% Indic |
| BigCodeBench instruct, pass@1 | +13.4 | +14.3 | +11.7 |
| HMMT, pass@32 | +10.0 | +3.3 | +10.0 |
| LiveCodeBench sampled, pass@32 | +3.0 | +0.1 | +1.5 |
| HMMT, pass@1 | 0.0 | +2.0 | +0.3 |
| MATH-500, pass@1 | +1.3 | +0.9 | +1.6 |
| HumanEval, pass@1 | +0.6 | +1.0 | +0.9 |
Table 5.8: English benchmark changes for the 200B-token Indic-share study. Values are percentage-point changes relative to the base comparator.
| Benchmark | Error Type | # dropped rows | # repaired rows |
| MILU | Incomplete question | 2,234 | 37 |
| MILU | Translation defect | 525 | 482 |
| MILU | Missing figure or passage | 480 | 0 |
| MILU | Malformed question | 172 | 232 |
| MILU | Duplicate options | 156 | 0 |
| IndicIFEval | Silently unenforced constraint | 0 | 2171 |
Table 6.1 : Summary of errors found in the MILU and IndicIFEval audits, by type. Dropped : rows removed because they could not be fixed. Repaired : rows corrected and retained. IndicIFEval errors are in its constraint checkers rather than its questions, so they were all repaired.
Figure 6.1 : Benchmark cleaning shifts accuracy in proportion to how unclean each language set was. Left: 0-shot accuracy on MILU vs. MILU-1.5; all three models gain 1.4–1.6 points. Right: mean gain per language across models, with the share of entries fixed in brackets.
| Failure type | Source example | Translation risk | Effect on evaluation |
| ARC-Challenge | |||
| Negation / polarity loss | “Scientists consider all of the following except …” ( Mercury_184975 ) | Except or similar cues ( not , least , unlikely ) may be weakened or omitted | Reverses which option is correct |
| Option distinction collapse | decrease / increase / remain the same ( NYSEDREGENTS_2012_8_42 ) | Opposing choices may receive similar or identical translations | Distinct distractors become indistinguishable |
| Technical terminology collapse | observation / prediction / conclusion / theory / hypothesis ( TIMSS_2007_8_pg53 ) | Closely related scientific terms may be translated with the same everyday word | Changes the conceptual distinction being tested |
| Symbol corruption | R , rr , Rr , RR ( Mercury_7238945 ) | Case-sensitive symbols may be transliterated or normalized in scripts without case | Genetically distinct choices collapse |
| Formula / unit modification | ; 32 °F | Formulae, signs, units or capitalization may be altered or localized | Changes the scientific content |
Table 6.2 : Failure modes observed when translating ARC-Challenge and MBPP with a generic LLM prompt. Each row gives a source example (with instance ID where available), how direct translation can alter it, and the effect on evaluation.
| Capability | Benchmarks |
| General chat | Arena Hard [ 48 ] , BiGGen Bench [ 42 ] , AlpacaEval [ 19 ] , IndiVibe [ 69 ] |
| Instruction following | IndicIFEval [ 36 ] |
| Math | GSM8K [ 13 ] , MATH-500 [ 32 ] , AIME 2026 [ 58 ] , FrontierMath (sample) [ 27 ] |
| STEM | JEEBench [ 2 ] , GPQA [ 67 ] , HLE [ 63 ] |
| Commonsense reasoning | ARC-Challenge [ 12 ] |
| Knowledge | MMLU-Pro [ 88 ] , MILU-1.5 [ 85 ] , BhashaBench-multi [ 15 ] , IndicQuest-v2 [ 35 ] |
Table 6.3: Capabilities and benchmarks in the Indic-LLM-Eval-Suite.
Figure 6.2 : Two-pass translation quality control. An LLM judge scores entries (pass ≥ 98) and flags error spans. Failed entries are corrected and re-judged. Remaining failures get 100% human review, and 20% of corrected passes are spot-checked.
| Task | Nemotron-3.5-Super-120B | Nemotron-3-Super-120B | sarvam-105b-1026 | Gemma-4-31B-it | Gnani-evon-v3.3-30B | Muse-Glimmer-30B | Nemotron-3.5-Lightning-30B | Qwen3-30B | Sarvam-30b | Gemma-4-26B-A4B-it | |
| Mathematics | |||||||||||
| Indic AIME 2026 | pass@4, acc | 95.28 | 90.83 | 94.44 | 94.44 | 83.61 | 93.89 | 71.39 | 79.44 | 90.83 | 93.33 |
| Indic MATH500 | acc_norm | 97.20 | 92.62 | 96.47 | 97.77 | 92.33 | 95.17 | 81.10 | 92.20 | 95.50 | 97.32 |
| Indic FrontierMath | acc_norm | 57.64 | 20.14 | 22.22 | 27.08 | — | — | — | — | — | — |
| STEM | |||||||||||
| Indic GPQA Diamond | avg[8]acc_norm | 88.23 | 76.40 | 72.45 | 84.37 | 64.36 | 82.85 | 62.35 | 58.52 | 59.71 | 79.80 |
Table 6.4 : Zero-shot comparison of post-trained models across mathematics, STEM, knowledge, code, translation, and instruction-following benchmarks. Best available results are marked in bold .
More in this topic
Language Modeling
Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.
See all 518 papers