Tokenization in Language Models

Follow this topic Ask about this topic

Latest papers

43 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX
Oct 15, 2026Wesley Kambale, Roland SankaraLanguage Modeling Low-Resource Language Processing
AI for Uganda Initiative · ISBAT University
The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction
Oct 15, 2026Hyunsik KimTokenization in Language Models Byte-Level Language Model
Samsung Research
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
Oct 15, 2026Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava +3Tokenization in Language Models LLM Evaluation
EleutherAI · University of Basel · National Taiwan University of Science and Technology +2
Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs
Oct 14, 2026Zhenyu Zhang, Zhichao CaoTokenization Tokenization in Language Models
Arizona State University
Latent Core Tokenizer: Compress, but Meaningfully
Oct 8, 2026Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie +3Tokenization in Language Models Tokenization
Microsoft Research Africa · Microsoft Research Accelerator · Microsoft Research India
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
More Than Words: Compositional Tokenization for Efficient Language Models
Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model Pretraining Efficient Language Model Training
The Hebrew University of Jerusalem
SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models
Oct 2, 2026Mohsen Larni, Sobhan Ebrahimi Azar, Pouyan Nahed +1Tokenization in Language Models LLM Evaluation
Department of Computer Science, University of Nevada, Las Vegas · Department of Computer Science University of Nevada, Las Vegas
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword Tokenization Byte-Pair Encoding
ETH Zürich · EPFL
Nameless Tokenization: A Lossless Tokenizer-Level Defense Against Control-Token Forgery in Open-Weight LLMs
Sep 15, 2026Kisu Yang, Yoonna Jang, Heuiseok LimAdversarial Attacks on LLMs LLM Security
1VAIV Company · 2Hanwha Aerospace · 3Korea University
When Tokenization is Secretly Output Supervision
Sep 1, 2026Tanja Baeumel, Josef van Genabith, Simon OstermannTokenization in Language Models LLM Training
1German Research Center for AI (DFKI) · Saarland University · Center for European Research in Trusted AI (CERTAIN)
Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility
Aug 26, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunTokenization in Language Models Tokenization
Karela Technologies Inc.
When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models
Aug 26, 2026Yefan Tao, Gerald Friedland, Luyang KongLanguage Model Robustness Tokenization in Language Models
Amazon
TokEval: A Tokenizer Evaluation Suite
Aug 18, 2026Clara MeisterLLM Evaluation Tokenization
EPFL
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
Aug 12, 2026Avijit Roy, Proma RoyMultilingual Language Models Tokenization in Language Models
John Jay College of Criminal Justice, CUNY · The City College of New York, CUNY
Explicit Boundary Markers for Subword Vocabularies
Aug 9, 2026Sander Land, Clara MeisterToken Efficiency Language Modeling
EPFL
Disentangling Language Modeling and Boundaries
Aug 4, 2026Mykola HaltiukTokenization in Language Models Byte-Level Language Model
Faculty of Computer Science AGH University of Krakow Krakow, Poland
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Jul 31, 2026Zhenyu Zhang, Zhichao CaoLLM Serving LLM Inference Acceleration
Arizona State University · Tempe, Arizona, USA
Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
Jul 29, 2026Poulami Ghosh, Preethi JyothiMultilingual Language Model Evaluation Language Model Robustness
*IIT Bombay, India
Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
Jul 17, 2026Ingo Ziegler, Martin Krebs, Desmond ElliottInformation Bottleneck Tokenization in Language Models
Department of Computer Science, University of Copenhagen
Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers
Jul 4, 2026Shuyang Xiang, Hao GuanLanguage Modeling Tokenization in Language Models
Independent Researcher. · Institute of Software, Chinese Academy of Sciences.
LangMAP: A Language-Adaptive Approach to Tokenization
Jun 22, 2026Clara Meister, Suchir Salhan, Andrzej Szablewski +3Multilingual Language Models Tokenization
EPFL · University of Cambridge · ETH Zürich
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
Emergent retokenization symmetry in large language models: phenomenology and applications
Jun 14, 2026Kanishk Jain, Matthew Day, Tankut CanLanguage Model Robustness Tokenization in Language Models
Department of Physics, Emory University, Atlanta, GA 30322
Incremental BPE Tokenization
May 29, 2026Shenghu Jiang, Ruihao GongSubword Tokenization Byte-Pair Encoding
1Beihang University · 2SenseTime Research.
Tokenisation via Convex Relaxations
May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2Token Efficiency Linear Programming
ETH Zurich · Kensho Technologies
Tokenization with Split Trees
May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4Token Efficiency Subword Tokenization
Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
May 13, 2026Chong Li, Yingzhuo Deng, Wen Yang +2Multilingual Language Models Tokenizer Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
Apr 18, 2026Disen Liao, Freda ShiLLM Fine-Tuning Language Modeling
University of Waterloo · Vector Institute
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
Apr 17, 2026Maitrey Mehta, Nishant Subramani, Zhichao Xu +2Multilingual Language Models Subword Tokenization
Kahlert School of Computing, University of Utah · Language Technologies Institute, Carnegie Mellon University

Oct 15, 2026J/K move · Enter open · S save

Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX

Wesley Kambale, Roland Sankara

AI for Uganda Initiative · ISBAT University

Language Modeling Low-Resource Language Processing Language Model Pretraining Tokenization in Language Models

Open paper AskPDF arXiv

Autoregressive foundation language models predominantly target high-resource Indo-European languages, imposing computational and morphological penalties on under-represented African languages. Generic multilingual tokenizers routinely exceed fertility rates of 3.2 tokens per word on agglutinative Bantu languages, degrading context efficiency and inflating latency. We present Pearl-Chat, an 8,473,122-parameter (

\sim

8.5M) causal decoder-only transformer engineered for Luganda (Oluganda, ISO 639-2: lg), a low-resource Niger-Congo Bantu language spoken by over 10 million people in East Africa. Built from first mathematical principles in JAX and Flax NNX, Pearl-Chat uses explicit functional state decomposition (nnx.split and nnx.merge), deterministic resumable data streaming via Google Grain, and SPMD tensor parallelism using jax.sharding.Mesh. We introduce a byte-level byte-pair encoding (BPE) subword tokenizer trained from scratch on curated Luganda corpora (

V = 8,192

), achieving 4.88 characters per token compression and 1.60 tokens per word fertility, a two-fold compression improvement over LLaMA-3 (128K) and Gemma (256K). The model is trained with a two-stage curriculum: Stage 1 pre-training on 59,501 sequences across parallel, biblical, encyclopedic, and journalistic corpora, and Stage 2 instruction fine-tuning on 510 dialogues with assistant-only loss masking. Across 4,000 optimization steps, validation cross-entropy loss drops monotonically from 9.4695 (perplexity: 12,958.98) to 0.1818 (perplexity: 1.20), a decrease of 9.2877 nats. On a three-tier held-out diagnostic benchmark, Pearl-Chat achieves 0.9792 ROUGE-L and 0.9405 BLEU while maintaining concordial agreement across Bantu noun classes. We provide a Python Server-Sent Events (SSE) streaming runtime and open-source all weights, tokenizers, and code.