Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.
Figures & tables
Corpus
# Toks (B)
# Indic Langs
Type
Roman
Data Open?
Pipeline Open?
Sangraha [ 41 ]
251
22
Mixed
✓
✓
✓
BhashaKritika [ 57 ]
540
10
Synth
–
❍
✗
IndicCorp-v2 [ 16 ]
20.9
23
Native
✗
✓
✗
FineWeb2 [ 62 ]
–
22
Native
✗
✓
✓
Sarvam-1 [ 70 ]
2,000
10
✗
✗
✗
✗
Param-1 [ 65 ]
2,770
1
Mixed
–
✗
✗
Table 2.1: Comparison of Indic pretraining corpora. # Toks (B): billions of Indic-language tokens as per each source’s own tokenizer and reports. Type: Native is human-authored text (web, books, OCR, transcripts); Synth is machine-generated or machine-translated text; Mixed combines both. Roman: whether the corpus includes romanized Indic text. Data Open?: whether the corpus is publicly downloadable. Pipeline Open?: whether the code used to collect, filter or generate the data is public. ✓yes; ✗no; ❍partially released.
Figure 3.1 : Released target tokens per language, complementing Table 3.1 . Bars report Gemma-4 target tokens in billions. For the twelve priority languages (Variant A), each native-script bar (left) and romanized bar (right, hatched) is split into tokens accepted directly by the quality judge (first pass) and tokens recovered through the rewriting stage. Variants B and C have no rewrite stage, so only totals are shown. The two rows use different vertical scales.
Source (English)
Target (Gemma-4 tokens, B)
Language
Config
Script
Tokens (B)
1st pass (%)
1st pass
Rewrite
Total
Exp.
Docs (M)
Variant A: Gemma-4-31B-it pipeline (Sections 4.1 to 4.4 ); native-script and romanized configurations
Bengali
bn-Beng-IN
Bengali
114.0
88
124.9
14.0
138.9
1.22
166.3
bn-Latn-IN
Latin
12.2
84
15.6
3.4
19.0
1.56
16.7
Gujarati
gu-Gujr-IN
Gujarati
115.5
83
168.8
27.6
196.4
1.70
169.3
gu-Latn-IN
Latin
12.0
89
15.2
2.0
17.2
1.43
16.5
Table 3.1 : Overview of the released pretraining corpus by language–script configuration. Source reports the number of English tokens corresponding to the released documents, while Target reports their translated token counts using the Gemma-4 tokenizer. 1st pass denotes examples accepted directly by the quality judge, and Rewrite denotes examples recovered through the rewriting stage. Exp. is the ratio of target to source tokens, and Docs denotes the number of released documents. Romanized ( Latn ) configurations are transliterations of subsets of the corresponding native-script data and therefore provide additional script coverage rather than additional content. We describe this corpus counting methodology in more detail in Appendix B .
Subset
Share (%)
Subset
Share (%)
cc_v2_diverse_qa
40.40
specialized_v1_math_textbooks
2.97
pretraining_sft_v1
12.03
specialized_v1_wiki_rewrite
2.68
cc_v2_english_cc
9.56
specialized_v1_stem_sft
1.61
cc_math_v1_4plus
8.65
specialized_v11_multiple_choice
1.08
cc_v21_hq_synthetic
7.46
specialized_v11_formal_logic
0.15
cc_v21_hq_organic
5.59
specialized_v11_economics
0.10
Table 3.2 : Nemotron-CC v2 subsets sampled for translation and their shares of the sampling target (percent). Names are the subset identifiers of the Nemotron-CC v2 release. The retrieval-style question-answer subset specialized_v1_rqa (1.00 percent in the initial plan) was dropped before sampling and the remaining shares renormalised.
Figure 4.1 : Workflow for the top-12 Indic languages (Variant A in Section 3.2). Gemma-4-31B-it translates whole English documents with fewer than 4,096 Gemma-4 tokens, assesses the translations, and rewrites low-scoring candidates before reassessment. Corpus assembly retains native translations that meet the score threshold of 75. A subset of the retained native-script text is romanized and checked against its native-script input. NeMo Data Designer runs generation and assessment; corpus acceptance follows these steps. Appendix D describes inference optimizations and execution details.
Figure 4.2 : Workflows for the remaining ten Indic languages (Variants B and C in Section 3.2), using Bodhan-MT for translation and Gemma-4-31B-it for quality assessment. Eight languages use whole-document translation. Longer Santali documents use overlapping windows followed by reconstruction. Manipuri uses Bengali-script translation followed by conversion to Meitei Mayek, with assessment before and after conversion. Corpus assembly applies language-specific acceptance rules; these workflows do not use the top-12 feedback-rewrite loop. Appendix D describes inference optimizations and execution details.
Label
Pool
Description
Available tokens (billions)
SG-V
Sangraha verified
Web text from human-verified sources
85.6
SG-U
Sangraha unverified
Perplexity-filtered web text
32.6
SG-T
Sangraha translated
English Wikimedia translated with IndicTrans2
139.0
FW+FP
FineWeb2 + FinePDFs
Web text and PDF-extracted text
94.8
NPI-N
NPI-v1 , native script
First-pass accepted translations, score at least 75
Varies by study
NPI-R
NPI-v1 , romanized
First-pass accepted romanized text
Varies by study
Table 5.1: Data available for constructing the Indic training mixtures. Counts describe the source pools, before sampling for a training run. NPI-N and NPI-R identify the native-script and romanized subsets of NPI-v1 (Section 3.1).
Comparison
What changes
Total budget
Indic budget
Public Indic corpora ( 5.4.1 )
Allocation across natural and translated sources
100B
45B
NPI-v1 contribution ( 5.4.2 )
Mixtures with NPI-v1 data; script mix
100B
45B
Pool size ( 5.4.3 )
Punjabi or Malayalam pool configurations
100B
45B
Romanized Malayalam ( 5.4.4 )
Native-script versus romanized training recipes
100B
45B
Indic share ( 5.4.5 )
10%, 20%, or 30% Indic sampling
200B
20B/40B/60B
English results ( 5.4.6 )
Evaluation of the same Indic-share runs
200B
20B/40B/60B
Table 5.2: Experiment overview. Budgets denote sampled training tokens, not available corpus size.
Recipe
Indic tokens (B)
MILU
Base
–
47.1
FW+FP 45
45
61.7
SG-T 45
45
60.8
SG-V 33 + SG-U 12
45
62.0
SG-V 18 + SG-U 6 + FW+FP 21
45
62.1
SG-V 15 + SG-U 4 + SG-T 8 + FW+FP 18
45
63.3
Table 5.3: Source survey: 100B-token runs with 45B Indic tokens. Recipe numbers are percentages of all training tokens; the remaining 55% uses the base mixture. MILU is averaged over eleven languages, including English.
Reported inclusive means
Recipe
MILU
MMLU-Pro
ARC-C
GSM8K
Base
44.4
29.3
53.7
40.4
Public-corpora baseline
63.2
40.2
84.2
67.7
NPI-N mixture
64.5
41.8
87.6
75.7
NPI-N + NPI-R mixture
64.3
42.0
87.3
75.1
Gemma-26B
64.7
34.8
87.3
61.9
Table 5.4: Mixtures containing NPI-v1 data at a 100B-token budget and 45% Indic share. The second panel excludes Odia as a sensitivity analysis. Recipe shares are percentages of all training tokens.
Punjabi
Pool configuration
MILU
MMLU-Pro
ARC-C
GSM8K
Web only
59.06
38.05
81.45
66.72
10B
63.06
42.99
87.49
78.24
20B
65.24
43.96
89.04
78.85
30B
65.80
44.30
89.04
79.83
Gemma-26B
65.58
34.90
88.20
64.70
Table 5.5: Single-language pool-size comparisons. “Web only” contains Indic data but excludes our synthetic data.
Training recipe
MILU, romanized
GSM8K, romanized
Base
44.25
19.64
Sangraha web
45.73
21.91
Sangraha + NPI-N (50B pool)
47.00
30.10
Sangraha + NPI-R
64.43
73.09
Gemma-26B
56.84
63.46
Table 5.6: Performance on romanized Malayalam MILU and GSM8K. GSM8K uses ASCII-digit scoring.
Configuration
MILU
MMLU-Pro
ARC-C
GSM8K
Mean-4
Base
46.8
33.5
57.5
44.1
45.5
Indic 10% (20B)
58.8
41.9
82.2
69.5
63.1
Indic 20% (40B)
60.2
43.0
83.9
71.4
64.6
Indic 30% (60B)
61.1
43.8
85.0
72.3
65.6
Gemma-26B
64.7
34.8
87.3
61.9
62.2
Table 5.7: Indic-share study: 200B total training tokens. Indic token allocations include the full Indic mixture, not only our synthetic data. Mean-4 is the arithmetic mean of the four displayed benchmark scores.
Benchmark and metric
10% Indic
20% Indic
30% Indic
BigCodeBench instruct, pass@1
+13.4
+14.3
+11.7
HMMT, pass@32
+10.0
+3.3
+10.0
LiveCodeBench sampled, pass@32
+3.0
+0.1
+1.5
HMMT, pass@1
0.0
+2.0
+0.3
MATH-500, pass@1
+1.3
+0.9
+1.6
HumanEval, pass@1
+0.6
+1.0
+0.9
Table 5.8: English benchmark changes for the 200B-token Indic-share study. Values are percentage-point changes relative to the base comparator.
Benchmark
Error Type
# dropped rows
# repaired rows
MILU
Incomplete question
2,234
37
MILU
Translation defect
525
482
MILU
Missing figure or passage
480
0
MILU
Malformed question
172
232
MILU
Duplicate options
156
0
IndicIFEval
Silently unenforced constraint
0
2171
Table 6.1 : Summary of errors found in the MILU and IndicIFEval audits, by type. Dropped : rows removed because they could not be fixed. Repaired : rows corrected and retained. IndicIFEval errors are in its constraint checkers rather than its questions, so they were all repaired.
Figure 6.1 : Benchmark cleaning shifts accuracy in proportion to how unclean each language set was. Left: 0-shot accuracy on MILU vs. MILU-1.5; all three models gain 1.4–1.6 points. Right: mean gain per language across models, with the share of entries fixed in brackets.
Failure type
Source example
Translation risk
Effect on evaluation
ARC-Challenge
Negation / polarity loss
“Scientists consider all of the following except …” ( Mercury_184975 )
Except or similar cues ( not , least , unlikely ) may be weakened or omitted
Reverses which option is correct
Option distinction collapse
decrease / increase / remain the same ( NYSEDREGENTS_2012_8_42 )
Opposing choices may receive similar or identical translations
Closely related scientific terms may be translated with the same everyday word
Changes the conceptual distinction being tested
Symbol corruption
R , rr , Rr , RR ( Mercury_7238945 )
Case-sensitive symbols may be transliterated or normalized in scripts without case
Genetically distinct choices collapse
Formula / unit modification
2H2O(l)→2H2(g)+O2(g) ; 32 °F
Formulae, signs, units or capitalization may be altered or localized
Changes the scientific content
Table 6.2 : Failure modes observed when translating ARC-Challenge and MBPP with a generic LLM prompt. Each row gives a source example (with instance ID where available), how direct translation can alter it, and the effect on evaluation.
Table 6.3: Capabilities and benchmarks in the Indic-LLM-Eval-Suite.
Figure 6.2 : Two-pass translation quality control. An LLM judge scores entries (pass ≥ 98) and flags error spans. Failed entries are corrected and re-judged. Remaining failures get 100% human review, and 20% of corrected passes are spot-checked.
Task
Nemotron-3.5-Super-120B
Nemotron-3-Super-120B
sarvam-105b-1026
Gemma-4-31B-it
Gnani-evon-v3.3-30B
Muse-Glimmer-30B
Nemotron-3.5-Lightning-30B
Qwen3-30B
Sarvam-30b
Gemma-4-26B-A4B-it
Mathematics
Indic AIME 2026
pass@4, acc
95.28
90.83
94.44
94.44
83.61
93.89
71.39
79.44
90.83
93.33
Indic MATH500
acc_norm
97.20
92.62
96.47
97.77
92.33
95.17
81.10
92.20
95.50
97.32
Indic FrontierMath
acc_norm
57.64
20.14
22.22
27.08
—
—
—
—
—
—
STEM
Indic GPQA Diamond
avg[8]acc_norm
88.23
76.40
72.45
84.37
64.36
82.85
62.35
58.52
59.71
79.80
Table 6.4 : Zero-shot comparison of post-trained models across mathematics, STEM, knowledge, code, translation, and instruction-following benchmarks. Best available results are marked in bold .
Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.8 trillion target-language tokens across 36 languages, produced by translating 100 billion high-quality Nemotron-CC tokens with Tower+ and OPUS-MT/HPLT-MT systems. For many medium- and lower-resource European languages, this is the largest openly available pre-training resource. Across five high- and medium-resource languages, reference LLMs trained on MultiSynt/MT reach the final score of HPLT 2.0, a native-data baseline, using roughly 72% fewer pre-training tokens, and outperform it by approximately 15% relative at a matched 100B-token training budget. Our analyses also identify evaluation blind spots: standard multiple-choice benchmarks miss translation-quality differences that a fluency-sensitive LLM-as-judge protocol recovers on the trained LLMs without detecting a deficit relative to its native-data baseline, while Norwegian idiomatic and culturally grounded tasks remain better served by native data. We release the corpus, including row-aligned translations from multiple systems, to support controlled research on multilingual pre-training data and evaluation.
Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19
ellamind · Leibniz University Hannover · University of Helsinki +14
We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.
Rinit Jain, Tirthraj Mahajan, Advait Joshi +1
Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras
Large Language Models (LLMs) have transformed conversational AI, yet high-quality multilingual code-mixed dialogue resources remain scarce, particularly for Indic languages where speakers naturally alternate between English and their native language in both native-script and Romanized forms. We present IndicTalk, one of the largest multilingual Indic code-mixed conversational corpora, comprising over 13,28,604 event-grounded multi-turn conversations across 18 language varieties covering 9 Indic languages. The corpus is generated through a fully automated pipeline that combines real-world news grounding, persona-conditioned dialogue generation using multilingual LLMs, and automatic quality validation. Extensive linguistic, automatic, and human evaluations demonstrate that IndicTalk produces fluent, coherent, and naturally code-mixed conversations across both script variants. We will release IndicTalk to support the development and evaluation of multilingual conversational AI for underrepresented Indic languages. The dataset is available at: https://huggingface.co/datasets/LingoIITGN/IndicTalk .