Tokenization in Language Models

Follow this topic Ask about this topic

Latest papers

43 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution
Apr 15, 2026Pavel Chizhov, Egor Bogomolov, Ivan P. YamshchikovTokenization Tokenization in Language Models
CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · JetBrains Research · TU Delft
Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries
Mar 30, 2026Jon-Paul CacioliTokenization in Language Models Representation Geometry in Language Models
Independent Researcher, Melbourne, Australia
Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
Mar 25, 2026Ben GublerTokenization in Language Models Tokenization
What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?
Mar 19, 2026Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard +1Tokenization in Language Models Temporal Reasoning in Language Models
University of Aberdeen · Université Grenoble Alpes & CNRS
A Family of LLMs Liberated from Static Vocabularies
Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language Modeling Byte-Level Language Model
Aleph Alpha Research
Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
Mar 11, 2026Zhipeng Yang, Shu Yang, Lijie Hu +1Language Model Robustness Tokenization in Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology
VerChol -- Grammar-First Tokenization for Agglutinative Languages
Mar 6, 2026Prabhu RajaTokenization in Language Models Tokenization
Bengaluru, India
Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography
Feb 16, 2026Gianluca Vico, Jindřich LibovickýTokenization in Language Models Low-Resource Language Processing
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic
You Can Learn Tokenization End-to-End with Reinforcement Learning
Feb 15, 2026Sam Dauncey, Roger WattenhoferSubword Tokenization LLM Training
Department of Electrical Engineering, ETH Zurich, Zurich, Switzerland.
Understanding LLM Failures: A Multi-Tape Turing Machine Analysis of Systematic Errors in Language Model Reasoning
Jan 27, 2026Magnus BomanTokenization in Language Models LLM Interpretability
Karolinska Institutet, Stockholm, Sweden
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Dec 23, 2025Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4Subword Tokenization Tokenization in Language Models
University of Toronto · Vector Institute · Google DeepMind +4
Less Is More: Reducing Token Counts Without Compromising Performance
Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference Acceleration Token Efficiency
Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9LLM Interpretability Distributional Semantics
Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3

Apr 15, 2026J/K move · Enter open · S save

From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution

Pavel Chizhov, Egor Bogomolov, Ivan P. Yamshchikov

CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · JetBrains Research · TU Delft

Tokenization Tokenization in Language Models Byte-Pair Encoding

Open paper AskPDF arXiv

Efficiency and safety of Large Language Models (LLMs), among other factors, rely on the quality of tokenization. A good tokenizer not only improves inference speed and language understanding but also provides extra defense against jailbreak attacks and lowers the risk of hallucinations. In this work, we investigate the efficiency of code tokenization, in particular from the perspective of data source diversity. We demonstrate that code tokenizers are prone to producing unused, and thus under-trained, tokens due to the imbalance in repository and language diversity in the training data, as well as the dominance of source-specific, repetitive tokens that are often unusable in future inference. By modifying the BPE objective and introducing merge skipping, we implement different techniques under the name Source-Attributed BPE (SA-BPE) to regularize BPE training and minimize overfitting, thereby substantially reducing the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. This provides an effective tool suitable for production use.