Tokenization

Follow this topic Ask about this topic

Latest papers

47 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.0% of all new papers.

Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers
Oct 29, 2026Jiongfan Yang, Weichen Fan, Haiwen Diao +2Efficient ViTs Tokenization
Peking University · SenseTime · Nanyang Technological University
Rethinking BPE: Merge Selection as Model Selection
Oct 19, 2026Vijini Liyanage, François YvonTokenization in Language Models Tokenization
Sorbonne-Université, CNRS, ISIR, Paris
Finitely Verifiable Segmentation: Decidability for Every Tokenizer with a Finite-State Verifier
Oct 16, 2026Nicklas NidhöggTokenization Tokenization in Language Models
Independent Researcher
Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs
Oct 14, 2026Zhenyu Zhang, Zhichao CaoTokenization Tokenization in Language Models
Arizona State University
Online Tokenization with Optimal Delay in Linear Time
Oct 13, 2026Mina Dalirrooyfard, Christopher YeTokenization Subword Tokenization
Morgan Stanley · University of California San Diego
Taking Tokenization off the Host for Agentic LLM Serving
Oct 9, 2026Mohammad Siavashi, Pascal Schwinte, Ali Banaei +3Tokenization LLM Serving
KTH Royal Institute of Technology, Stockholm, Sweden
Latent Core Tokenizer: Compress, but Meaningfully
Oct 8, 2026Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie +3Morphology-Aware Tokenization Tokenization in Language Models
Microsoft Research Africa · Microsoft Research Accelerator · Microsoft Research India
More Than Words: Compositional Tokenization for Efficient Language Models
Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model Pretraining Efficient Language Model Training
The Hebrew University of Jerusalem
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
Sep 18, 2026Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le +4Tokenization Language Model Pretraining
University of Information Technology, Vietnam National University, Ho Chi Minh city, Viet Nam · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Abu Dhabi, UEA
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword Tokenization Byte-Pair Encoding
ETH Zürich · EPFL
Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility
Aug 26, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunTokenization in Language Models Tokenization
Karela Technologies Inc.
Tlow: Flow-based Item Tokenizer for Recommendation
Aug 25, 2026Nian Li, Chonggang Song, Jingtao Ding +3Tokenization Recommender Systems
Shenzhen International Graduate School, Tsinghua University Shenzhen, China
Type-Driven Tokenization for Brahmic Scripts
Aug 21, 2026Sai Hemanth Kapila, Rakshika BagavathyTokenization Subword Tokenization
Microsoft India
TokEval: A Tokenizer Evaluation Suite
Aug 18, 2026Clara MeisterLLM Evaluation Tokenization
EPFL
How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness
Aug 18, 2026Yi WangTokenization Token Efficiency
Department of Electronic Engineering, Tsinghua University
ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization
Aug 13, 2026Lixing LiTokenization Adaptive Tokenization
Cornell University Ithaca, NY 14853, USA
Explicit Boundary Markers for Subword Vocabularies
Aug 9, 2026Sander Land, Clara MeisterToken Efficiency Language Modeling
EPFL
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Jul 31, 2026Zhenyu Zhang, Zhichao CaoLLM Serving LLM Inference Acceleration
Arizona State University · Tempe, Arizona, USA
Tokenizing Crosslingual Homographs
Jul 20, 2026Rotem Brillant, Yuval PinterMultilingual Language Models Subword Tokenization
Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer Sheva, Israel
Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
Jul 17, 2026Ingo Ziegler, Martin Krebs, Desmond ElliottInformation Bottleneck Tokenization in Language Models
Department of Computer Science, University of Copenhagen
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Jun 25, 2026Sander LandToken Efficiency Subword Tokenization
Writer, Inc.
wav2tok 2.0: Scalable Audio Tokenization Maintaining Explicit Pairwise Token Alignment for Efficient Audio Retrieval
Jun 25, 2026Adhiraj Banerjee, Vipul AroraAudio Representation Learning Speech Processing
Department of Electrical Engineering, Indian Institute of Technology Kanpur, India · Department of Electrical Engineering, KU Leuven, Belgium
Speaking Numbers to LLMs: Multi-Wavelet Number Embeddings for Time Series Forecasting
Jun 25, 2026Defu Cao, Zijie Lei, Muyan Weng +2Time Series Forecasting Tokenization
University of Southern California · Meta · Google DeepMind
LangMAP: A Language-Adaptive Approach to Tokenization
Jun 22, 2026Clara Meister, Suchir Salhan, Andrzej Szablewski +3Multilingual Language Models Tokenization
EPFL · University of Cambridge · ETH Zürich
Structuring and Tokenizing Distributed User Interest Context for Generative Recommendation
Jun 18, 2026Ruizhong Qiu, Yinglong Xia, Dongqi Fu +6Sequential Recommendation Tokenization
University of Illinois Urbana–Champaign · 2Meta MRS
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
Incremental BPE Tokenization
May 29, 2026Shenghu Jiang, Ruihao GongSubword Tokenization Byte-Pair Encoding
1Beihang University · 2SenseTime Research.
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
May 28, 2026Rohan ShravanToken Efficiency Subword Tokenization
The School of AI India
The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty
May 23, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Low-Resource Language Processing
LEX AI Platform, legal.org.ua · Kyiv, Ukraine
Tokenisation via Convex Relaxations
May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2Token Efficiency Linear Programming
ETH Zurich · Kensho Technologies

Oct 29, 2026J/K move · Enter open · S save

Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers

Jiongfan Yang, Weichen Fan, Haiwen Diao, Zhongang Cai+1

Peking University · SenseTime · Nanyang Technological University

Efficient ViTs Tokenization T2I Generation Diffusion Transformer Pixel-Space Diffusion Models

Open paper AskPDF arXiv

Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control while converging faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within only 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, SpecSlot surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.