Tokenizer

Ask about this topic

Momentum

14 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 82

All topics
CardsList

  1. More Than Words: Compositional Tokenization for Efficient Language Models

    Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzTokenizerLanguage Modeling

    The Hebrew University of Jerusalem

  2. Q-SPT: Learnable Query-Based Compression for Low-Frame-Rate Speech Tokenization

    Oct 1, 2026Jeeyoung Yun, Seohwan Yun, Sungwoong KimNeural Speech CodecsAudio Tokenizers

    Korea University, Seoul, Republic of Korea

  3. ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

    Sep 30, 2026Jinsung Jeon, Seung-won HwangLLM Reasoning StrategiesBehavioral Embeddings

    KAIST InnoCORE LLM · Seoul National University · Korea University

  4. The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype

    Sep 30, 2026Thomas ServalTokenizerLanguage Pairs

    Baracoda AI Labs & NEOMA Business School

  5. NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generation

    Sep 29, 2026Jiawei Zhang, Shuhao Liu, Rong Huang +4Autoregressive Image GenerationVisual Tokenizers

    North China Electric Power University · University of Science and Technology of China · Stanford University

  6. Humanoid Loco-Manipulation With Discrete VLA Model

    Sep 28, 2026Wenxin Shao, Siqi Chai, Kun Li +4Humanoid Loco-ManipulationAction Space

    Horizon Robotics

  7. Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali

    Sep 27, 2026Kalash Shrestha, Nikhil PradhanByte-Pair EncodingTokenizer

    Kathmandu University · IOE Thapathali Campus

  8. ChunkRank: Model-Aware Text Chunking and Abstention-Aware Answer Selection for LLM Pipelines

    Sep 24, 2026Amit Nautiyal, Ayush Bhatt, Gaurav NautiyalChunkQuestion-Answering Benchmarks

  9. Objective vs. Search: Decomposing What Makes a Good Tokeniser

    Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelByte-Pair EncodingTokenizer

    ETH Zürich · EPFL

  10. Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

    Sep 15, 2026Adam Zachary Wasserman, David BeaucheminLegal Natural Language Processing BenchmarksMotion-Language Alignment

    1Open Honest Foundation · 2Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Québec, Canada

  11. To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

    Sep 14, 2026Franck Signe, Hippolyte Pilchen, François Yvon +1Multilingual Language ModelsTokenizer

    Kyutai, Paris · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble · Sorbonne Université, CNRS, Institute of Intelligent Systems and Robotics, Paris

  12. On the role of the tokenizer in ECG transformer models

    Sep 14, 2026Jiawei Li, Fabio Bonassi, Johan Sundström +2Transformer ArchitecturesTokenizer

    Uppsala University, Uppsala, Sweden · Scilifelab, Uppsala, Sweden

  13. BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models

    Sep 9, 2026Shivam Singh, Aditya Yadavalli, Catherine Arnett +1Whisper-Large-V3Tokenizer

    UC San Diego · EleutherAI

  14. Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

    Sep 8, 2026Siting Li, Zhengyang Wang, Simon Shaolei Du +2TokenizerLoss Function

    University of Washington · Amazon FAR

  15. Compressing Streaming Neural Audio Encoders via Latent-Space Distillation

    Sep 3, 2026Prasanth Yadla, Mohammad Samragh, Dongseong Hwang +7Neural Speech CodecsDataset Distillation

    Apple

  16. Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining

    Sep 1, 2026Francois MeyerSubword TokenizationLarge Language Model Pretraining

    Department of Computer Science University of Cape Town

  17. TokEval: A Tokenizer Evaluation Suite

    Aug 18, 2026Clara MeisterTokenizerDownstream Fine-Tuning

    EPFL

  18. Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

    Aug 13, 2026Aoxin NiMathematical Reasoning BenchmarksFundamental Limits

    Department of Computer Science University of Chinese Academy of Sciences

  19. Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

    Aug 12, 2026Peijie Chen, Zhuanling Zha, Zhipeng Nie +8Flow-Matching Text-To-SpeechAudio Tokenizers

    Didichuxing Co. Ltd

  20. MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models

    Aug 11, 2026Yuan Wang, Hualiang Wang, Yixin Chen +6Medical Vision-Language ModelsPerception

    Zhejiang University · Fudan University

  21. Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

    Aug 9, 2026Sofiia Riazhskykh, Nam Luu, Ondřej BojarLarge Language Model PretrainingPretraining

    Charles University, Faculty of Mathematics and Physics

  22. Tokenizer-Generator Coupling in Medical Image Generation

    Aug 7, 2026Liam ChalcroftMedical Image GenerationVisual Tokenizers

    University College London London, United Kingdom

  23. Bend the Basics: Degradation-Aware Deformable Tokenization for All-in-One Image Restoration

    Aug 7, 2026Zihao He, Yunfeng Wu, Xinchao Wang +1Image RestorationDegradation

    School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Beijing, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China +1

  24. KVAE: Family of Tokenizers for Multimodal Generative Models

    Aug 6, 2026Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov +11Visual TokenizersStar-Vae

    Kandinsky Lab

  25. DigitCode: Symbolic Tokenization of Hand Motion by Anatomical Units

    Aug 4, 2026Haoyu Gu, Haotian Lu, Jingrun Du +1HandLatent Motion

    School of Future Technology, South China University of Technology · Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University

  26. Token-Native Storage: Read and Write in your Agent's Language

    Aug 3, 2026Kumar ShivenduByte-Pair EncodingToken Embeddings

    Qdrant

  27. Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

    Aug 1, 2026Bohdan DidenkoByte-Pair EncodingTokenizer

    Lviv Polytechnic National University

  28. TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    Jul 31, 2026Zhenyu Zhang, Zhichao CaoTime-To-First-TokenLarge Language Model Serving

    Arizona State University · Tempe, Arizona, USA

  29. The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

    Jul 27, 2026Priyansh SrivastavaSubword TokenizationTokenizer

    Sirena Ai India

  30. Tokenizing Crosslingual Homographs

    Jul 20, 2026Rotem Brillant, Yuval PinterCross-Lingual ConsistencyTokenizer

    Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer Sheva, Israel

  31. When Close Enough Is Not Enough: Autoregressive Drift in Quantum Circuit Synthesis

    Jul 14, 2026Mehdi Saeedi, Eddie Richter, Paul HartkeQuantum CircuitsAutoregressive Model

    Advanced Micro Devices, Inc. (AMD)

  32. ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

    Jul 7, 2026Tianjiao Yu, Xinzhuo Li, Yifan Shen +43D Generative Models3D Representation

    University of Illinois Urbana-Champaign

  33. PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

    Jul 2, 2026Haofei Xu, Rundi Wu, Philipp Henzler +73D ReconstructionTokenizer

    Google · ETH Zurich · University of Tübingen, Tübingen AI Center +3

  34. GEAR: Guided End-to-End AutoRegression for Image Synthesis

    Jun 30, 2026Bin Lin, Zheyuan Liu, Chenguo Lin +8Autoregressive Image GenerationGenerative Models

    Peking University · Tencent Hunyuan

  35. SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance

    Jun 29, 2026Tengyue Jiang, Chunpu Xu, Jiayue Kang +1Discrete Action TokenizersGeneralizable Vision-Language-Action Policies

    2East China University of Science and Technology · 3Hong Kong Polytechnic University · 4Xi’an University of Electronic Science and Technology +1

  36. Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling

    Jun 26, 2026Qinfeng Zhu, Lei FanOmnidirectional ImagesPanoptic Segmentation

    University of Liverpool, United Kingdom. · Xi’an Jiaotong-Liverpool University, China.

  37. Beyond MoCap: Scaling Motion Tokenizers with Synthetic Human Motion for Generative Modeling

    Jun 25, 2026Yiwen Yan, Wanning He, Yu-Wing TaiHuman Motion GenerationHuman Motion

    Dartmouth College

  38. ForensicsTok: Forensics-Guided Tokenized Modeling for Image Tampering Localization

    Jun 23, 2026Lei Xu, Haowei Wang, Shen Chen +3Mask2FormerDigital Forensics

    1Shenzhen University · 2Tencent Youtu Lab

  39. LangMAP: A Language-Adaptive Approach to Tokenization

    Jun 22, 2026Clara Meister, Suchir Salhan, Andrzej Szablewski +3TokenizerMultilingual Language Models

    EPFL · University of Cambridge · ETH Zürich

  40. Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

    Jun 18, 2026Milan Miletić, Julie Kallini, Ekaterina ShutovaSubword TokenizationMultilingual Language Models

    University of Amsterdam · Stanford University

  41. Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers

    Jun 18, 2026Dong Hoon Lee, Seunghoon HongDiffusion TransformersVisual Tokenizers

    Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea

  42. Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

    Jun 17, 2026Tolga ŞakarTokenizerMorphology

    Independent Researcher

  43. MotionVLA: Vision-Language-Action Model for Humanoid Motion

    Jun 13, 2026Nonghai Zhang, Siyu Zhai, Yanjun Li +5Human Motion GenerationHumanml3D

    School of Computer Science, Peking University · AI2 Robotics

  44. Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

    Jun 13, 2026Kieron Seven Jun Wei Lee, Muhammad Reza Qorib, Andrew Ivan Soegeng +1Multilingual Language ModelsTokenizer

    National University of Singapore · Carnegie Mellon University · SAP

  45. Augmenting Molecular Language Models with Local nn-gram Memory

    Jun 10, 2026Xinni Zhang, Zijing Liu, He Cao +2Unigram TokenizerLanguage Modeling

    1The Chinese University of Hong Kong · 2International Digital Economy Academy

  46. SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations

    Jun 10, 2026Peijie Chen, Wenhao Guan, Weijie Wu +7Flow-Matching Text-To-SpeechTokenizer

    School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China

  47. What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks

    Jun 8, 2026Qin Yang, Lu Malloy, Joshua Lee +4Attacker Large Language ModelContent Moderation

    University of Connecticut · University of Tennessee · University of California, Santa Barbara +1

  48. Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

    Jun 8, 2026Yifan Niu, Han Xiao, Dongyi Liu +4Teacher-Student DistillationTokenizer

    The Hong Kong University of Science and Technology (Guangzhou) · Tencent · The Hong Kong University of Science and Technology

  49. X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining

    Jun 7, 2026Miracle Kang, Lights Shi, Lucy Liang +10Discrete Action TokenizersDiffusion-Based Vision-Language-Actions

    X SQUARE ROBOT · City University of Hong Kong · Tsinghua University

  50. AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens

    Jun 5, 2026Xiaocheng Lu, Yuxi Chen, Jie Zhang +5Visual TokenizersLong Visual-Token Sequences

    The Hong Kong University of Science and Technology · The Hong Kong Polytechnic University

  51. LLM-Conditioned Synthesis of Pathological Gaits via Structured Gait-Language Representations

    Jun 4, 2026Mritula Chandrasekaran, Sanket Kachole, Jarek Francik +1Gait AnalysisGait Dynamics

    School of Computer Science and Mathematics, Kingston University, London, United Kingdom

  52. Balancing Image Compression and Generation with Bootstrapped Tokenization

    Jun 4, 2026Haozhe Chi, Jinghan Li, Hao Jiang +4Visual TokenizersAutoregressive Image Generation

    1Peking University · 2Central Media Technology Institute, Huawei

  53. LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

    Jun 3, 2026Yuanrui Wang, Xingxuan Zhang, Han Yu +7Tabular Foundation ModelsTokenizer

    Stable AI · Tsinghua University

  54. TiWeaver: Unified Temporal Dynamics Modeling via Contextual Patching

    Jun 2, 2026Zhe Li, Jindong Tian, Hao Miao +3Multivariate Time Series ForecastingTemporal Dynamics

    East China Normal University Shanghai, China · Hong Kong Polytechnic University Hong Kong, China · Aalborg University Aalborg, Denmark

  55. Large Byte Model: Teaching Language Models About Compiled Code

    Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4MalwareTokenizer

    CrowdStrike · CrowdStrike U.K. · CrowdStrike Romania +1

  56. Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering

    Jun 1, 2026Dongxing Mao, Jinpeng Wang, Jiahao Tang +6Autoregressive Image GenerationVisual Tokenizers

    Central South University · University of Oxford · Microsoft Research

  57. Time Series as Language: A Universal Tokenizer for General-Purpose Time Series Foundation Models

    May 31, 2026Yunhao Zhang, Ruiying Qi, Jiale Zheng +3Time Series Foundation ModelsTime Series

    Shanghai Jiao Tong University · Huawei Noah’s Ark Lab

  58. HiTokSR: A Coarse-to-Fine Tokenizer with Hierarchical Codebooks for High-Fidelity Real-World Image Super-Resolution

    May 31, 2026Mingxi LiReal-World Image Super-ResolutionVisual Tokenizers

  59. Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals

    May 30, 2026Shenghao DingAudio TokenizersWavelets

    Yet Another AI

  60. Incremental BPE Tokenization

    May 29, 2026Shenghu Jiang, Ruihao GongByte-Pair EncodingTokenizer

    1Beihang University · 2SenseTime Research.