Tokenization

Follow this topic Ask about this topic

Recent momentum

+12%

19 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

9 new papers

A weekly snapshot of new work published in Tokenization.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Tokenization.

Period ending 2026-09-07

4 new papers

A weekly snapshot of new work published in Tokenization.

139 papers

Latest in Tokenization

Browse all topics →

Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet
Jun 18, 2026Milan Miletić, Julie Kallini, Ekaterina ShutovaSubword Tokenization Multilingual Language Models
Structuring and Tokenizing Distributed User Interest Context for Generative Recommendation
Jun 18, 2026Ruizhong Qiu, Yinglong Xia, Dongqi Fu +6Recommendation Tokenization
BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
Jun 17, 2026Max Van Puyvelde, Ibrahim Gulluk, Wim Van Criekinge +13D Latent Diffusion Model Brain Magnetic Resonance Imaging
MotionVLA: Vision-Language-Action Model for Humanoid Motion
Jun 13, 2026Nonghai Zhang, Siyu Zhai, Yanjun Li +5Human Motion Generation Video Latents
From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion
Jun 10, 2026Yuchen Xian, Yunqiu Xu, Yang He +1Visible Image Fusion Tokenization
Augmenting Molecular Language Models with Local $n$ -gram Memory
Jun 10, 2026Xinni Zhang, Zijing Liu, He Cao +2N$-Grams Features Graph Language Models
SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations
Jun 10, 2026Peijie Chen, Wenhao Guan, Weijie Wu +7Zero-Shot Text-To-Speech Speech Encoder
UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data
Jun 9, 2026Dong Fang, Youjun Wu, Yuanxin Zhong +4Dexterous Robot Hands Hand
What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks
Jun 8, 2026Qin Yang, Lu Malloy, Joshua Lee +4Language Model Evasion Attacks Content Moderation
Inside the LLM Word Factory
Jun 7, 2026Benzi Busigin, Yuval PinterTokenization Model Activations
X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining
Jun 7, 2026Miracle Kang, Lights Shi, Lucy Liang +10Vision-Language-Action Models Visual Tokens
AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens
Jun 5, 2026Xiaocheng Lu, Yuxi Chen, Jie Zhang +5Visual Tokens Tokenization
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Tokens Tokenization
LLM-Conditioned Synthesis of Pathological Gaits via Structured Gait-Language Representations
Jun 4, 2026Mritula Chandrasekaran, Sanket Kachole, Jarek Francik +1Single-Subject Gait Dataset Gait Recognition
Balancing Image Compression and Generation with Bootstrapped Tokenization
Jun 4, 2026Haozhe Chi, Jinghan Li, Hao Jiang +4Tokenization Self-Supervised Reconstruction Method
GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data
Jun 3, 2026Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Kumar Gyawali +2Tabular Foundation Models Tabpfn
LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
Jun 3, 2026Yuanrui Wang, Xingxuan Zhang, Han Yu +7Tabular Foundation Models Tensor Programs
ChannelTok: Efficient Flexible-Length Vision Tokenization
Jun 3, 2026Sukriti Paul, Arpit Bansal, Tom GoldsteinVisual Tokens Tokenization
CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
Jun 3, 2026Eugene Kwek, Feng Liu, Rui Zhang +1Neural Audio Codecs Audio Tokens
MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers
Jun 3, 2026Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang +2Neural Partial Differential Equation Solvers Tokenization
TiWeaver: Unified Temporal Dynamics Modeling via Contextual Patching
Jun 2, 2026Zhe Li, Jindong Tian, Hao Miao +3Multivariate Time Series Forecasting Temporal Dynamics
Large Byte Model: Teaching Language Models About Compiled Code
Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4Malware Detection Tokenization
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Jun 1, 2026Hui Li, Yangfan Gao, Junlin Shang +4Audio Tokens Tokenization
Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning
Jun 1, 2026Ziyang Yao, Zeyu Zhu, YunCheng Jiang +2Autonomous Driving Visual Tokens
Time Series as Language: A Universal Tokenizer for General-Purpose Time Series Foundation Models
May 31, 2026Yunhao Zhang, Ruiying Qi, Jiale Zheng +3Time Series Foundation Models Time Series
HiTokSR: A Coarse-to-Fine Tokenizer with Hierarchical Codebooks for High-Fidelity Real-World Image Super-Resolution
May 31, 2026Mingxi LiGenerative Super-Resolution Calibration-Free Low-Bit Vector Quantization
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
May 29, 2026Yuhan Song, Linhao Zhang, Aiwei Liu +6Audio Tokens Tokenization
UniRTL: Unifying Code and Graph for Robust RTL Representation Learning
May 29, 2026Yi Liu, Hongji Zhang, Lei Chen +2Flexible Register Transfer Level Design Automation Register Transfer Level
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
May 29, 2026Luyuan Zhang, Siyuan Li, Zedong Wang +7Visual Tokens Tokenization
Incremental BPE Tokenization
May 29, 2026Shenghu Jiang, Ruihao GongByte-Pair Encoding Tokenization
Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs
May 28, 2026Zhihao Wu, Gracia Gong, Qinglin Zhu +2Large Language Model Watermarks Watermarking
Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation
May 28, 2026M. Ali Bayram, Banu Diri, Savaş YıldırımTokenization Multilingual
HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding
May 28, 2026Bohan Li, Shi Lian, Hankun Wang +6Audio Tokens Tokenization
Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
May 28, 2026Yizhuo Lu, Changde Du, Qingyu Shi +5Brain-Computer Interface Multi--Task Learning
BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization
May 25, 2026Zhengyang Ni, Feng Yan, Yu Guo +1Building Blocks Geometric Reconstruction
The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty
May 23, 2026Volodymyr OvcharovSubword Tokenization Tokenization
Tokenisation via Convex Relaxations
May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2Tokenization Token-Level Uncertainty
Tokenization with Split Trees
May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4Tokenization Subword Tokenization
Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
May 21, 2026Maya Bechler-Speicher, Gilad Yehudai, Gil Harari +3Tokenization Expressivity
Vision Foundation Models as Generalist Tokenizers for Image Generation
May 18, 2026Anlin Zheng, Qi Han, Xin Wen +5Visual Tokens Image Generation
SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark
May 18, 2026Khalid Yusuf DahirText Corpora Low-Resource Languages
WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens
May 18, 2026Yiwei Guo, Shaobin Zhuang, Zhipeng Huang +4Visual Tokens Tokenization
Echoes in Filter Bubble: Diagnosing and Curing Popularity Bias in Generative Recommenders
May 16, 2026Jun Yin, Bangguo Zhu, Peng Huo +5Large-Scale Recommendation Systems Sequential Recommendation
Hypergraph Pattern Machine: Compositional Tokenization for Higher-Order Interactions
May 15, 2026Kyrie Zhao, Zehong Wang, Tianyi Ma +5Hypergraphs Drug-Drug Interactions
Modeling Music as a Time-Frequency Image: A 2D Tokenizer for Music Generation
May 15, 2026Yuqing Cheng, Xingyu Ma, Guochen Yu +1Audio Tokens Music Generation
Eradicating Negative Transfer in Multi-Physics Foundation Models via Sparse Mixture-of-Experts Routing
May 14, 2026Ellwil Sharma, Arastu SharmaFluid Dynamics Mixture-of-Experts Models
A Calculus-Based Framework for Determining Vocabulary Size in End-to-End ASR
May 14, 2026Sunil Kumar KopparapuAutomatic Speech Recognition Systems Vocabulary Expansion
ENSEMBITS: an alphabet of protein conformational ensembles
May 13, 2026Kaiwen Shi, Carlos OliverProtein Data Bank Conformational Ensembles
Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenTokenization Transformer Architectures
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
May 13, 2026Chong Li, Yingzhuo Deng, Wen Yang +2Tokenization C++
ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation
May 12, 2026Inwoo Hwang, Hojun Jang, Bing Zhou +3Human Motion Generation Motion Generation
Beyond Spatial Compression: Interface-Centric Generative States for Open-World 3D Structure
May 11, 2026Xiang Chen, Alexander Binder3D Assets 3D World
Yeti: A compact protein structure tokenizer for reconstruction and multi-modal generation
May 11, 2026Nabin Giri, Steven Farrell, Kristofer E. BouchardProtein Data Bank Tokenization
MambaNetBurst: Direct Byte-level Network Traffic Classification without Tokenization or Pretraining
May 11, 2026Gayan K. Kulatilleke, Siamak Layeghy, Mahsa Baktashmotlagh +1Network Traffic Bidirectional Mamba
What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion
May 8, 2026Zhengrong Yue, Taihang Hu, Mengting Chen +8Latent Diffusion Model Masked Autoencoders
PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization
May 7, 2026Adhiraj Banerjee, Vipul AroraAudio Tokens Tokenization
Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF
May 5, 2026Mullosharaf K. ArabovNatural Language Processing Large Language Model Pretraining
A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language
May 5, 2026Thibault Bañeras-Roux, Mickael Rouvier, Jane Wottawa +1Subword Tokenization Word Error Rate
TimeTok: Granularity-Controllable Time-Series Generation via Hierarchical Tokenization
May 2, 2026Seokhyun Lee, Jaeho Kim, Changjun Oh +2Synthetic Time Series Autoregressive Generation
End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer
May 1, 2026Wenda Chu, Bingliang Zhang, Jiaqi Han +4Autoregressive Image Generation Generative Models

Jun 18, 2026 · cs.CLJ/K move · Enter open · S save

Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

Milan Miletić, Julie Kallini, Ekaterina Shutova

University of Amsterdam · Stanford University

Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio. To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution across languages. We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.

Subword Tokenization Multilingual Language Models Tokenization Cross-Lingual Consistency Phoneme-Level Analysis Rescue

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Recent digests

9 new papers

3 new papers

4 new papers

Latest in Tokenization