Language Model Vocabulary Expansion

Follow this topic Ask about this topic

Latest papers

8 papers0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
Aug 4, 2026Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan +1Efficient Language Model Training Continued Pretraining
NVIDIA
Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya
Jul 16, 2026Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang NejdlLanguage Model Pretraining Multilingual Language Models
L3S Research Center, Leibniz University Hannover, Germany · Addis Ababa University, Ethiopia
ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models
Jun 21, 2026Mahnoor Khan, Afsheen Asif, Milhan Afzal Khan +2Multilingual Language Models Word Embeddings
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Computer Science, University of Agriculture Faisalabad, Faisalabad, Pakistan
Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping
May 5, 2026Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang +1Continual Learning for LLMs Memorization in Language Models
New Jersey Institute of Technology · University of California, Berkeley
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
Apr 17, 2026Maitrey Mehta, Nishant Subramani, Zhichao Xu +2Multilingual Language Models Subword Tokenization
Kahlert School of Computing, University of Utah · Language Technologies Institute, Carnegie Mellon University
RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment
Apr 10, 2026Jaber Jaber, Osama JaberLanguage Model Vocabulary Expansion Arabic NLP
RightNow AI
Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation
Apr 2, 2026Daiwei Chen, Zhoutong Fu, Chengming Jiang +12Language Model Vocabulary Expansion Large Language Model-Based Recommendation
University of Wisconsin-Madison · LinkedIn Corporation · Northeastern University +1
Task-Centric Acceleration of Small-Language Models
Feb 27, 2026Dor Tsur, Sharon Adar, Ran LevyLanguage Model Vocabulary Expansion Efficient Language Model Inference
Amazon

Aug 4, 2026J/K move · Enter open · S save

Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension

Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar

NVIDIA

Efficient Language Model Training Continued Pretraining Language Model Vocabulary Expansion

Open paper AskPDF arXiv

Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency. We present a systematic study of more than 20 initialization strategies for Hindi vocabulary extension in Nemotron-3-Nano-30B-A3B. Our comparison spans vocabulary-averaging baselines; external and learned initialization methods, including FOCUS, top-k semantic retrieval, and residual MLP mappings; subword composition; norm calibration; and input-output asymmetry. We find that subword composition methods outperform both vocabulary averaging and external/learned initialization approaches. Within subword composition, asymmetric variants achieve the lowest observed early validation loss and reveal distinct preferences for input and output embedding initialization. The best observed configuration initializes the input embedding matrix with uniform subword averaging and Hindi-specific norm calibration, and the output language modeling head with character-length-weighted subword averaging. Relative to the standard Mean-all baseline, this full initialization pipeline reaches comparable validation loss with over a 6x reduction in CPT steps and exceeds the baseline's 3,500-step MILU-Hindi accuracy after only 500 steps. Finally, we show that initialization loss and initialization bits-per-byte (Init BPB) are unreliable predictors of downstream convergence, whereas lightweight CPT, as few as 50 steps, provides a cost-effective and reliable signal for selecting the best initialization strategy.