Tokenization

Follow this topic Ask about this topic

Latest papers

44 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs
Oct 14, 2026Zhenyu Zhang, Zhichao CaoTokenization Tokenization in Language Models
Arizona State University
Online Tokenization with Optimal Delay in Linear Time
Oct 13, 2026Mina Dalirrooyfard, Christopher YeTokenization Subword Tokenization
Morgan Stanley · University of California San Diego
Taking Tokenization off the Host for Agentic LLM Serving
Oct 9, 2026Mohammad Siavashi, Pascal Schwinte, Ali Banaei +3Tokenization LLM Serving
KTH Royal Institute of Technology, Stockholm, Sweden
Latent Core Tokenizer: Compress, but Meaningfully
Oct 8, 2026Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie +3Tokenization in Language Models Tokenization
Microsoft Research Africa · Microsoft Research Accelerator · Microsoft Research India
More Than Words: Compositional Tokenization for Efficient Language Models
Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model Pretraining Efficient Language Model Training
The Hebrew University of Jerusalem
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
Sep 18, 2026Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le +4Tokenization Language Model Pretraining
University of Information Technology, Vietnam National University, Ho Chi Minh city, Viet Nam · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Abu Dhabi, UEA
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword Tokenization Byte-Pair Encoding
ETH Zürich · EPFL
Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility
Aug 26, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunTokenization in Language Models Tokenization
Karela Technologies Inc.
Tlow: Flow-based Item Tokenizer for Recommendation
Aug 25, 2026Nian Li, Chonggang Song, Jingtao Ding +3Tokenization Recommender Systems
Shenzhen International Graduate School, Tsinghua University Shenzhen, China
Type-Driven Tokenization for Brahmic Scripts
Aug 21, 2026Sai Hemanth Kapila, Rakshika BagavathyTokenization Subword Tokenization
Microsoft India
TokEval: A Tokenizer Evaluation Suite
Aug 18, 2026Clara MeisterLLM Evaluation Tokenization
EPFL
How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness
Aug 18, 2026Yi WangTokenization Token Efficiency
Department of Electronic Engineering, Tsinghua University
ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization
Aug 13, 2026Lixing LiTokenization Adaptive Tokenization
Cornell University Ithaca, NY 14853, USA
Explicit Boundary Markers for Subword Vocabularies
Aug 9, 2026Sander Land, Clara MeisterToken Efficiency Language Modeling
EPFL
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Jul 31, 2026Zhenyu Zhang, Zhichao CaoLLM Serving LLM Inference Acceleration
Arizona State University · Tempe, Arizona, USA
Tokenizing Crosslingual Homographs
Jul 20, 2026Rotem Brillant, Yuval PinterMultilingual Language Models Subword Tokenization
Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer Sheva, Israel
Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
Jul 17, 2026Ingo Ziegler, Martin Krebs, Desmond ElliottInformation Bottleneck Tokenization in Language Models
Department of Computer Science, University of Copenhagen
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Jun 25, 2026Sander LandToken Efficiency Subword Tokenization
Writer, Inc.
wav2tok 2.0: Scalable Audio Tokenization Maintaining Explicit Pairwise Token Alignment for Efficient Audio Retrieval
Jun 25, 2026Adhiraj Banerjee, Vipul AroraAudio Representation Learning Speech Processing
Department of Electrical Engineering, Indian Institute of Technology Kanpur, India · Department of Electrical Engineering, KU Leuven, Belgium
Speaking Numbers to LLMs: Multi-Wavelet Number Embeddings for Time Series Forecasting
Jun 25, 2026Defu Cao, Zijie Lei, Muyan Weng +2Time Series Forecasting Tokenization
University of Southern California · Meta · Google DeepMind
LangMAP: A Language-Adaptive Approach to Tokenization
Jun 22, 2026Clara Meister, Suchir Salhan, Andrzej Szablewski +3Multilingual Language Models Tokenization
EPFL · University of Cambridge · ETH Zürich
Structuring and Tokenizing Distributed User Interest Context for Generative Recommendation
Jun 18, 2026Ruizhong Qiu, Yinglong Xia, Dongqi Fu +6Sequential Recommendation Tokenization
University of Illinois Urbana–Champaign · 2Meta MRS
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
Incremental BPE Tokenization
May 29, 2026Shenghu Jiang, Ruihao GongSubword Tokenization Byte-Pair Encoding
1Beihang University · 2SenseTime Research.
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
May 28, 2026Rohan ShravanToken Efficiency Subword Tokenization
The School of AI India
The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty
May 23, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Low-Resource Language Processing
LEX AI Platform, legal.org.ua · Kyiv, Ukraine
Tokenisation via Convex Relaxations
May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2Token Efficiency Linear Programming
ETH Zurich · Kensho Technologies
Tokenization with Split Trees
May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4Token Efficiency Subword Tokenization
Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA
Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenLanguage Modeling Subword Tokenization
Department of Communications and Electronics Institut Polytechnique de Paris
Lost in Translation? Exploring the Shift in Grammatical Gender from Latin to Occitan
May 9, 2026Ahan Chatterjee, Matthias Schöffel, Matthias Aßenmacher +2Tokenization
Bavarian Academy of Sciences (BAdW), Munich · LMU Munich · Munich Center for Machine Learning (MCML) +1

Oct 14, 2026J/K move · Enter open · S save

Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs

Zhenyu Zhang, Zhichao Cao

Arizona State University

Tokenization Tokenization in Language Models LLM Inference Acceleration Subword Tokenization

Open paper AskPDF arXiv

An agent that drives an LLM sends the model the same conversation again and again, and each turn adds only a little text at the end. Inference engines reuse the KV cache of the shared prefix, so only the new tokens pass through the model. However, tokenization has no such shortcut. The CPU front end of the serving engine converts every request back into token IDs from its first character, which lengthens the time to first token (TTFT) and consumes CPU on every turn. We present Delta Tokenizer, a CPU tokenization front end that tokenizes only the delta between a request and the previous request of the same session. It retains the earlier token sequence, encodes again only the appended text and the short tail whose tokens the append could affect, and joins the two only at a position the tokenizer's splitting rules themselves cannot move, which keeps the result exact. For long requests with no state to reuse, it cuts the text where practical tokenizers already keep neighboring pieces independent, encodes the pieces on several cores at once, and concatenates the outputs into the reference encoding, exactly. A shared CPU pool schedules these chunks together with the short incremental updates, so an update runs between two chunks rather than behind an entire long job. The incremental path shortens the tokenization of a session turn by roughly 40x relative to the Hugging Face reference tokenizer, and on 16 cores the chunk-parallel path encodes a four-million-character request 8.2x faster than a serial encode and 1.6-2.1x faster than LoPT. Inside SGLang, Delta Tokenizer reduces the median TTFT through the chat interface by 59% at 100K tokens and brings the P99 TTFT of agent sessions down 9.5x under load and 23x when long requests tie up the front end. Every output we checked matches the reference, among them 41.8 billion documents encoded by its fast encoder.