In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B

Avinash Benki; Prasanjith Pasunuti; Sutanshu Saksena Raj; Parijat Chatterjee; Nayanjit Sarkar; Daksh Maan

cs.CLOct 11, 2026

In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B

Authors: Avinash Benki, Prasanjith Pasunuti, Sutanshu Saksena Raj, Parijat Chatterjee, Nayanjit Sarkar, Daksh Maan

Abstract

We adapted Nemotron-3-Nano-30B-A3B for eight Indian languages without changing its backbone architecture. We expanded the vocabulary in place from 131,072 to 163,072 entries using continued byte-pair encoding and AdaptMix rebalancing, reducing tokens per word by 38.6%. We initialized new input and output rows by averaging their original constituent rows, then trained them on 90% Indic and 10% English text before updating the backbone. Continual pretraining then progressed through broad training, high-quality annealing, and long-context training over approximately 1.27 trillion tokens. The annealed checkpoint gained 12.9 points on the Indic MILU benchmark but lost 3.4 on English MMLU. We also report practical negative results. Direct token addition underperformed continued byte-pair encoding. AdamW weight decay changed rows intended to stay fixed. Perplexity on an Indic-heavy holdout did not select the learning rate that best preserved English performance.

Figures & tables

Figure 1: English scores during initialization and embedding warmup, shown in run order. These runs form a debugging sequence, not a controlled ablation. These scores are not directly comparable with the CPT results in Table 1 .

Benchmark	Base	Broad	Anneal
MILU [ 12 ]	40.63	53.54	53.52
ARC-Challenge-Indic [ 30 ]	35.23	48.20	48.20
IndicGLUE-selected [ 14 ]	52.12	57.86	57.22
IndicMMLU-Pro [ 15 ]	14.44	20.03	20.02
BharatMath	29.99	41.06	39.70
Indic BPB overall ( $\downarrow$ ) [ 13 ]	0.8453	0.6189	0.6132

Table 1: Results under the same evaluation setup. Both adapted checkpoints precede long-context training. Appendix F lists the full benchmark results for these checkpoints.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Run	Extension	Mean fertility	Notes
Base Nemotron	None	3.5472	Original tokenizer
Direct add_tokens	32K	2.4815	14.3% above continued BPE
Continued BPE	32K	2.1715	Used in subsequent experiments
Raw / NFKC / NFKC+regex	32K	2.1716–2.1823	Similar fertility across variants
TOK-2	32K	2.1773	Selected for CPT
TOK-3	64K	2.0996	Lowest mean, not used for CPT

Appendix

Table 2: Tokenizer fertility on Samanantar, using seed 42 and 2,000 samples per language.

Field	Value
Initialization	Checkpoint after warmup, vocabulary 163,072
Precision	BF16
Sequence length	8,192 (broad and anneal), 131,072 (long context)
Global / micro batch	2,048 / 1
Parallelism	TP=1, EP=8, PP=1, remaining ranks use DP
Optimizer	Distributed AdamW

Appendix

Table 3: Configuration used for broad continual pretraining.

Language	Base	120B CPT	$\Delta$
Kannada	32.8	47.2	$+$ 14.4
Tamil	34.2	46.6	$+$ 12.4
Gujarati	34.4	46.5	$+$ 12.2
Telugu	35.8	47.9	$+$ 12.1
Malayalam	35.5	45.7	$+$ 10.1
Bengali	39.4	47.7	$+$ 8.4

Appendix

Table 4: MILU scores before and after 120B tokens of CPT with 55% English and 45% Indic data.

Bucket	Weight	Shards	Pool tokens
Indic long documents	0.22	9	22.0B
Indic high-quality	0.15	132	15.0B
English long documents	0.20	7	20.0B
Mathematics and STEM	0.14	1	14.0B
Long-context supervised data	0.10	3	10.0B
Code	0.08	1	8.0B

Appendix

Table 5: Composition of the 100B-token long-context data pool. The selected run consumed approximately 99.86B tokens from this pool.

Benchmark	Base	Broad	Anneal
MILU [ 12 ]	40.63	53.54	53.52
ARC-Challenge-Indic [ 30 ]	35.23	48.20	48.20
Belebele [ 20 ]	37.49	42.17	42.27
BharatMath	29.99	41.06	39.70
Global-MMLU-Lite [ 21 ]	32.50	36.17	36.00
IndicGLUE-selected [ 14 ]	52.12	57.86	57.22

Appendix

Table 6: Results for the base, broad, and annealed checkpoints. Two benchmarks are omitted because they lack results for one or more checkpoints.

Metric or context length	Score (%)
Mean RULER recall (55 slices)	76.7
Mean QA exact match (10 slices)	45.7
Recall at 8,192 tokens	96.6
Recall at 16,384 tokens	90.6
Recall at 32,768 tokens	82.4
Recall at 65,536 tokens	67.3

Appendix

Table 7: RULER recall and QA exact-match scores for the long-context checkpoint. These tasks differ from those used for the broad and annealed checkpoint comparisons.

Stage	GPU-hours
CPT broad ( ${\sim}$ 974B)	${\sim}$ 25,147
CPT annealing (195B)	${\sim}$ 4,988
CPT long context ( ${\sim}$ 100B)	4,977
CPT total	${\sim}$ 35,112

Appendix

Table 8: GPU-hours for CPT, excluding embedding warmup and ablations.

Field	Value
Total / active parameters	31.75B / 3.31B
Layers	52 (23 Mamba-2, 23 MoE, 6 GQA)
Model dim / heads / KV heads	2,688 / 32 / 2
MoE experts (routed / active / shared)	128 / 6 / 1
Vocabulary / max sequence	163,072 / 131,072
Position encoding	none (NoPE)

Appendix

Table 9: Nemotron-3-Nano-30B-A3B architecture and expanded vocabulary. Parameter counts refer to the base model. Vocabulary expansion adds approximately 172 million parameters.

Language	Upstream	Expanded	Change
Hindi	2.05	1.66	$-$ 19.2%
Bengali	3.06	2.14	$-$ 30.0%
Gujarati	3.61	1.91	$-$ 47.1%
Kannada	3.86	2.34	$-$ 39.5%
Malayalam	4.97	2.77	$-$ 44.3%
Marathi	3.10	1.94	$-$ 37.4%

Appendix

Table 10: Fertility of the original tokenizer and TOK-2 (163,072 entries).

Bucket	Weight	Target tokens
Reasoning	22%	62.7B
Indic long documents	15%	42.8B
Math & STEM	14%	39.9B
Knowledge retention	12%	34.2B
English long documents	11%	31.4B
Code	9%	25.7B

Appendix

Table 11: Planned data allocation for the 285B mid-training mixture.

Figure 2: Tokens per word before and after the 32K expansion. Lower is better.

Figure 3: Fertility of the original tokenizer and the exploratory TOK-3 tokenizer with 64,000 added entries. TOK-3 was not used in any CPT run.

Figure 4: Learning-rate comparison. Validation perplexity and English benchmark performance favored different learning rates in the four-run sweep.

Figure 5: Tokens consumed during broad training, high-quality annealing, and long-context training.

Figure 6: MILU accuracy before and after the 120B-token ablation with 55% English and 45% Indic data. This run informed mixture planning and was not released as a checkpoint.

Figure 7: Indic and English benchmark results across CPT checkpoints under the same evaluation setup. Lower FLORES-IN bits per byte indicate better performance on that metric.

Figure 8: Weights & Biases traces for broad training at 8K context. These plots end before the broad checkpoint reported in the benchmark evaluation.

Figure 9: Weights & Biases traces for high-quality annealing. Learning-rate decay begins late in this stage. Validation loss fluctuates around 1.0 over the recorded interval.

Figure 10: Weights & Biases traces for long-context training at sequence length 131,072. Gradient norm and language-model loss fall early in training and then stabilize. Sequence load-balancing loss settles near 1.17 during the recorded run.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 441 papers

Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs
Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language Modeling Language Model Pretraining
NVIDIA · AI4Bharat
A Sovereign, Indic-First, Reasoning Model
Oct 11, 2026Sutanshu Saksena Raj, Prasanjith Pasunuti, Parijat Chatterjee +4Language Modeling Cross-Lingual Reasoning in Language Models
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
Aug 4, 2026Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan +1Efficient Language Model Training Continued Pretraining
NVIDIA

Oct 19, 2026J/K move · Enter open · S save

Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker, Anusha Kamath+23

NVIDIA · AI4Bharat

Language Modeling Language Model Pretraining Synthetic Data Pretraining Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.

Abstract

Language Model Pretraining

Explore similar work

Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

A Sovereign, Indic-First, Reasoning Model

Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension