Subword Tokenization

Ask about this topic

Momentum

8 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 52

All topics
CardsList

  1. More Than Words: Compositional Tokenization for Efficient Language Models

    Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model PretrainingEfficient Language Model Training

    The Hebrew University of Jerusalem

  2. Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

    Oct 1, 2026Hyunsik Kim, Youngmoon JungMultilingual Language ModelsToken Efficiency

    Samsung Research

  3. Counting and Min-Cost Encoding for Tokenization in Large Language Models

    Oct 1, 2026Shuming Shi, Xiang Zhang, Hao Yu +7Subword Tokenization

    Mashang Consumer Finance Co., Ltd., China · National-Mathematics Artificial Intelligence Institute in Chongqing (NMAII), China

  4. Component-Weighted Centroid Search for Exact Incremental BPE

    Sep 30, 2026Harshit Verma, Rex YingSubword TokenizationByte-Pair Encoding

    Yale University

  5. Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali

    Sep 27, 2026Kalash Shrestha, Nikhil PradhanLanguage Model PretrainingLow-Resource Language Processing

    Kathmandu University · IOE Thapathali Campus

  6. Objective vs. Search: Decomposing What Makes a Good Tokeniser

    Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword TokenizationByte-Pair Encoding

    ETH Zürich · EPFL

  7. Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

    Sep 15, 2026Adam Zachary Wasserman, David BeaucheminLLM EvaluationCross-Lingual Representation Alignment

    1Open Honest Foundation · 2Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Québec, Canada

  8. Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition

    Sep 14, 2026Devansh SharmaToken EfficiencyLow-Resource Language Processing

    Independent researcher

  9. Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining

    Sep 1, 2026Francois MeyerLanguage Model PretrainingAutoregressive Language Modeling

    Department of Computer Science University of Cape Town

  10. Explicit Boundary Markers for Subword Vocabularies

    Aug 9, 2026Sander Land, Clara MeisterToken EfficiencyLanguage Modeling

    EPFL

  11. Morphology Aware Reversible Semantic Tokenization and Hierarchical Word Composition for Tamil Language Models

    Aug 2, 2026Anand MuruganSubword TokenizationMachine Translation

  12. Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

    Aug 1, 2026Kenny ShaoToken PruningSubword Tokenization

    Department of Computer Science Florida International University

  13. Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

    Aug 1, 2026Bohdan DidenkoLow-Resource Language ProcessingSubword Tokenization

    Lviv Polytechnic National University

  14. TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    Jul 31, 2026Zhenyu Zhang, Zhichao CaoLLM ServingLLM Inference Acceleration

    Arizona State University · Tempe, Arizona, USA

  15. The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

    Jul 27, 2026Priyansh SrivastavaLow-Resource Language ProcessingSubword Tokenization

    Sirena Ai India

  16. Joint Optimization for Greedy Longest-match Tokenization

    Jul 25, 2026Adhiraj Singh, Deepanshu Mody, Ghina Al Shdaifat +4Subword Tokenization

    Center for Data Science, New York University New York, NY, USA · Kensho Technologies Cambridge, MA, USA

  17. BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

    Jul 25, 2026Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran +1Low-Resource Language ProcessingSubword Tokenization

    Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi, New Delhi 110016, India · RSL Quantum, FITT, IIT Delhi, New Delhi 110016, India

  18. Tokenizing Crosslingual Homographs

    Jul 20, 2026Rotem Brillant, Yuval PinterMultilingual Language ModelsSubword Tokenization

    Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer Sheva, Israel

  19. Privacy Leakage in Federated Learning in Radiology Reports: A Comparative Evaluation of Tokenizer-Driven Privacy Risks

    Jul 15, 2026Santhosh Parampottupadam, Andres Martinez, Dimitrios Bounias +3Gradient Inversion AttacksAdversarial Attacks

    German Cancer Research Center (DKFZ), Division of Medical Image Computing, Heidelberg, Germany · Medical Faculty Heidelberg, Heidelberg University, Heidelberg, Germany · Department of Computer Engineering, Bilkent University, Universiteler 06800 Çankaya/Ankara, Türkiye +1

  20. BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech

    Jul 7, 2026Ho Lam Chung, Bo-Xuan Zheng, Cheng-Chieh Huang +8TTS SynthesisLanguage Modeling

    National Taiwan University

  21. Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

    Jul 6, 2026Hunter HeidenreichSubword TokenizationByte-Pair Encoding

    Independent Researcher

  22. How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation

    Jun 25, 2026Maria LevchenkoHistorical Document AnalysisLLM Evaluation

    University of Bologna

  23. MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment

    Jun 25, 2026Sander LandToken EfficiencySubword Tokenization

    Writer, Inc.

  24. QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages

    Jun 22, 2026Maria ContrerasLow-Resource Language ProcessingSubword Tokenization

    Universidad Peruana de Ciencias Aplicadas (UPC) Lima, Peru

  25. Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

    Jun 18, 2026Milan Miletić, Julie Kallini, Ekaterina ShutovaMultilingual Language ModelsLow-Resource Language Processing

    University of Amsterdam · Stanford University

  26. Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

    Jun 17, 2026Tolga ŞakarWord EmbeddingsSubword Tokenization

    Independent Researcher

  27. Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

    Jun 13, 2026Kieron Seven Jun Wei Lee, Muhammad Reza Qorib, Andrew Ivan Soegeng +1Multilingual Language ModelsSocial Bias in Language Models

    National University of Singapore · Carnegie Mellon University · SAP

  28. Inside the LLM Word Factory

    Jun 7, 2026Benzi Busigin, Yuval PinterLanguage Model DecodingLLM Interpretability

    Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer Sheva, Israel

  29. LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling

    Jun 3, 2026Daria Ledneva, Denis KuznetsovHierarchical Representation LearningGenomics

    Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia.

  30. Incremental BPE Tokenization

    May 29, 2026Shenghu Jiang, Ruihao GongSubword TokenizationByte-Pair Encoding

    1Beihang University · 2SenseTime Research.

  31. BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

    May 28, 2026Rohan ShravanToken EfficiencySubword Tokenization

    The School of AI India

  32. The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

    May 23, 2026Volodymyr OvcharovMultilingual Language Model EvaluationLow-Resource Language Processing

    LEX AI Platform, legal.org.ua · Kyiv, Ukraine

  33. Tokenisation via Convex Relaxations

    May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2Token EfficiencyLinear Programming

    ETH Zurich · Kensho Technologies

  34. Tokenization with Split Trees

    May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4Token EfficiencySubword Tokenization

    Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA

  35. Surface-Form Neural Sparse Retrieval: Robust Fuzzy Matching for Industrial Music Search

    May 18, 2026Paul Greyson, Zhichao Geng, Wei Zhang +1Learned Sparse RetrievalMusic Information Retrieval

    Amazon

  36. Effective Context in Transformers: An Analysis of Fragmentation and Tokenization

    May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenLanguage ModelingSubword Tokenization

    Department of Communications and Electronics Institut Polytechnique de Paris

  37. Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

    May 13, 2026Ruan Visser, Trienko Grobler, Marcel DunaiskiLanguage Model PretrainingLow-Resource Language Processing

    Department of Computer Science, Stellenbosch University, Stellenbosch, South Africa

  38. Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF

    May 5, 2026Mullosharaf K. ArabovLLM Fine-TuningLow-Resource Language Processing

    KAZAN (VOLGA REGION) FEDERAL UNIVERSITY · INSTITUTE OF COMPUTATIONAL MATHEMATICS AND INFORMATION TECHNOLOGIES

  39. A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language

    May 5, 2026Thibault Bañeras-Roux, Mickael Rouvier, Jane Wottawa +1ASR EvaluationAutomatic Speech Recognition

    Nantes University LS2N Nantes, France · Avignon University LIA Avignon, France · Le Mans University LIUM Le Mans, France

  40. Compute Optimal Tokenization

    May 2, 2026Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6Language Model Scaling LawsEfficient Language Model Training

    1FAIR at Meta · University of Washington

  41. Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

    Apr 29, 2026Théo Gigant, Bowen Peng, Jeffrey QuesnelleLanguage Model PretrainingEfficient Language Model Training

    Nous Research

  42. KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters

    Apr 27, 2026SungHo Kim, Juhyeong Park, Yeachan Kim +1Language ModelingSubword Tokenization

    Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea

  43. Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspective

    Apr 20, 2026Meifang Chen, Zhe Yang, Huang Nianchen +4Data LeakageMemorization in Language Models

    †The Chinese University of Hong Kong · ‡Nanyang Technological University · §University of Southern California +1

  44. How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

    Apr 18, 2026Disen Liao, Freda ShiLLM Fine-TuningLanguage Modeling

    University of Waterloo · Vector Institute

  45. Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion

    Apr 17, 2026Maitrey Mehta, Nishant Subramani, Zhichao Xu +2Multilingual Language ModelsSubword Tokenization

    Kahlert School of Computing, University of Utah · Language Technologies Institute, Carnegie Mellon University

  46. Stochasticity in Tokenisation Improves Robustness

    Apr 17, 2026Sophie Steger, Rui Li, Sofiane Ennadir +4Adversarial Attacks on LLMsNeural Network Robustness

    Institute of Signal Processing and Speech Communication, Graz University of Technology, Graz, Austria · ELLIS Institute Finland & Aalto University, Espoo, Finland · King AI Labs, Microsoft Gaming +2

  47. MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models

    Mar 17, 2026Chen-Hao Chao, Wei-Fang Sun, Junwei Quan +2Diffusion Language ModelsSubword Tokenization

    University of Toronto, Vector Institute · NVIDIA AI Technology Center · National Taiwan University

  48. Tokenization vs. Augmentation: A Systematic Study of Writer Variance in IMU-Based Online Handwriting Recognition

    Feb 25, 2026Jindong Li, Dario Zanca, Vincent Christlein +4Data AugmentationOnline Handwriting Recognition

    Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · STABILO International GmbH, Heroldsberg, Germany · Ludwig-Maximilians-Universität München, Munich, Germany +2

  49. You Can Learn Tokenization End-to-End with Reinforcement Learning

    Feb 15, 2026Sam Dauncey, Roger WattenhoferSubword TokenizationLLM Training

    Department of Electrical Engineering, ETH Zurich, Zurich, Switzerland.

  50. TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

    Dec 23, 2025Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4Subword Tokenization

    University of Toronto · Vector Institute · Google DeepMind +4

  51. Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

    Aug 6, 2025Negar Foroutan, Clara Meister, Debjit Paul +4Low-Resource Language ProcessingSubword Tokenization

    EPFL · University of Zurich

  52. Less Is More: Reducing Token Counts Without Compromising Performance

    Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference AccelerationToken Efficiency

    Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University

  53. Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

    Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9LLM InterpretabilityDistributional Semantics

    Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3