Tokenizer Adaptation

Follow this topic Ask about this topic

Tokenizer adaptation studies how changing token vocabularies and segmentation can improve text coverage, shorten sequences, and reduce computation for languages, domains, or modalities poorly served by existing models. Common approaches include expanding or replacing BPE or unigram vocabularies, aligning new tokens to existing embeddings, and combining vocabulary changes with continued pretraining, selective fine-tuning, or distillation. Evaluations span low-resource language classification and tagging, domain summarization, retrieval benchmarks such as BEIR, and ASR or OCR measures such as word error rate and character error rate. A central challenge is improving target-language efficiency and task performance without damaging capabilities in other languages; better token compression alone does not guarantee downstream gains.

Latest papers

19 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B
Oct 11, 2026Avinash Benki, Prasanjith Pasunuti, Sutanshu Saksena Raj +3Language Model Pretraining Tokenizer Adaptation
BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models
Sep 9, 2026Shivam Singh, Aditya Yadavalli, Catherine Arnett +1Speech Foundation Models Automatic Speech Recognition
UC San Diego · EleutherAI
Writing-System-Level Tokenizer Adaptation for Byte-Level BPE
Aug 1, 2026Bohdan DidenkoLow-Resource Language Processing Subword Tokenization
Lviv Polytechnic National University
Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR
Jul 10, 2026Sanjid Hasan, Md. Abdur RahmanAutomatic Speech Recognition Low-Resource Speech Recognition
Khulna University of Engineering & Technology, Bangladesh · Military Institute of Science and Technology, Bangladesh
LangMAP: A Language-Adaptive Approach to Tokenization
Jun 22, 2026Clara Meister, Suchir Salhan, Andrzej Szablewski +3Multilingual Language Models Tokenization
EPFL · University of Cambridge · ETH Zürich
AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens
Jun 5, 2026Xiaocheng Lu, Yuxi Chen, Jie Zhang +5Image Tokenization Visual Tokenization
The Hong Kong University of Science and Technology · The Hong Kong Polytechnic University
Modular Monolingual Adaptation using Pretrained Language Models
Jun 4, 2026Nalin Kumar, Ondřej DušekLow-Resource Language Processing Tokenizer Adaptation
Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Prague, Czechia
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Representation Learning Autoencoders
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China
Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering
Jun 1, 2026Dongxing Mao, Jinpeng Wang, Jiahao Tang +6Image Tokenization Visual Tokenization
Central South University · University of Oxford · Microsoft Research
Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation
May 28, 2026M. Ali Bayram, Banu Diri, Savaş YıldırımMultilingual Language Models Sentence Embeddings
Department of Computer Engineering Yıldız Technical University Istanbul, Turkey · Department of Computer Engineering Istanbul Bilgi University Istanbul, Turkey
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
May 28, 2026Rohan ShravanToken Efficiency Subword Tokenization
The School of AI India
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
May 17, 2026Gunjan Balde, Soumyadeep Roy, Mainack Mondal +1Domain-Adaptive Pretraining Text Summarization
Dept. of Computer Science and Engg., IIT Kharagpur, Kharagpur, India · Dept. of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
May 13, 2026Chong Li, Yingzhuo Deng, Wen Yang +2Multilingual Language Models Tokenizer Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
May 12, 2026Mingcheng Zhu, Zhiyao Luo, Yu Liu +1LLM Compression Clinical Prediction
Department of Engineering Science, University of Oxford, Oxford, United Kingdom.
Adapting TrOCR for Printed Tigrinya Text Recognition: Word-Aware Loss Weighting for Cross-Script Transfer Learning
Apr 22, 2026Yonatan Haile Medhanie, Yuanhua NiTransfer Learning Optical Character Recognition
College of Software Engineering, Nankai University, Tianjin, China · College of Artificial Intelligence, Nankai University, Tianjin, China
Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
Apr 20, 2026Zhichao Geng, Yang YangLearned Sparse Retrieval Information Retrieval
Amazon Web Service · Shanghai, China
Toward domain-specific machine translation and quality estimation systems
Mar 26, 2026Javad Pourmostafa Roshan SharamiMachine Translation Domain Adaptation
SinLlama -- A Large Language Model for Sinhala
Aug 12, 2025H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3Language Model Pretraining Multilingual Language Models
Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India. +1
Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian
May 22, 2024Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2Language Model Pretraining Multilingual Language Models
ITMO University · HSE University · Vikhr +2

Oct 11, 2026J/K move · Enter open · S save

In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B

Avinash Benki, Prasanjith Pasunuti, Sutanshu Saksena Raj, Parijat Chatterjee+2

Language Model Pretraining Tokenizer Adaptation Multilingual Language Models Domain Adaptation for LLMs

Open paper AskPDF arXiv

We adapted Nemotron-3-Nano-30B-A3B for eight Indian languages without changing its backbone architecture. We expanded the vocabulary in place from 131,072 to 163,072 entries using continued byte-pair encoding and AdaptMix rebalancing, reducing tokens per word by 38.6%. We initialized new input and output rows by averaging their original constituent rows, then trained them on 90% Indic and 10% English text before updating the backbone. Continual pretraining then progressed through broad training, high-quality annealing, and long-context training over approximately 1.27 trillion tokens. The annealed checkpoint gained 12.9 points on the Indic MILU benchmark but lost 3.4 on English MMLU. We also report practical negative results. Direct token addition underperformed continued byte-pair encoding. AdamW weight decay changed rows intended to stay fixed. Perplexity on an Indic-heavy holdout did not select the learning rate that best preserved English performance.