Byte-Pair Encoding

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 29

All topics

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities
Oct 1, 2026Hyunsik Kim, Youngmoon JungByte-Pair Encoding Unicode U+2014
Samsung Research
Component-Weighted Centroid Search for Exact Incremental BPE
Sep 30, 2026Harshit Verma, Rex YingByte-Pair Encoding Bounded-Memory
Yale University
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali
Sep 27, 2026Kalash Shrestha, Nikhil PradhanByte-Pair Encoding Tokenizer
Kathmandu University · IOE Thapathali Campus
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelByte-Pair Encoding Tokenizer
ETH Zürich · EPFL
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Sep 14, 2026Franck Signe, Hippolyte Pilchen, François Yvon +1Multilingual Language Models Tokenizer
Kyutai, Paris · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble · Sorbonne Université, CNRS, Institute of Intelligent Systems and Robotics, Paris
Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition
Sep 14, 2026Devansh SharmaByte-Pair Encoding Indian Languages
Independent researcher
Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer
Aug 30, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunByte-Pair Encoding Transformer Architectures
Karela Technologies Inc.
Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
Aug 11, 2026Qingjie Zhang, Ziqi Tang, Jie Zhang +7Text Corpora Byte-Pair Encoding
Tsinghua University · SiliconProspect AI · Nanyang Technological University +1
Disentangling Language Modeling and Boundaries
Aug 4, 2026Mykola HaltiukByte-Pair Encoding Language Modeling
Faculty of Computer Science AGH University of Krakow Krakow, Poland
Token-Native Storage: Read and Write in your Agent's Language
Aug 3, 2026Kumar ShivenduByte-Pair Encoding Token Embeddings
Qdrant
Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding
Aug 1, 2026Kenny ShaoByte-Pair Encoding Subword Tokenization
Department of Computer Science Florida International University
Writing-System-Level Tokenizer Adaptation for Byte-Level BPE
Aug 1, 2026Bohdan DidenkoByte-Pair Encoding Tokenizer
Lviv Polytechnic National University
Joint Optimization for Greedy Longest-match Tokenization
Jul 25, 2026Adhiraj Singh, Deepanshu Mody, Ghina Al Shdaifat +4Byte-Pair Encoding Token Compression
Center for Data Science, New York University New York, NY, USA · Kensho Technologies Cambridge, MA, USA
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
Jul 25, 2026Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran +1Subword Tokenization Indian Languages
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi, New Delhi 110016, India · RSL Quantum, FITT, IIT Delhi, New Delhi 110016, India
JPEG AIC2026: A large-scale dataset for fine-grained assessment of image coding
Jul 24, 2026Mohsen Jenadeleh, Jon Sneyers, João Ascenso +8Learned Image Compression Image Quality Assessment
University of Konstanz, Germany · Cloudinary, Belgium · IST-IT, Portugal +5
Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR
Jul 10, 2026Sanjid Hasan, Md. Abdur RahmanAutomatic Speech Recognition Bangla
Khulna University of Engineering & Technology, Bangladesh · Military Institute of Science and Technology, Bangladesh
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Jul 6, 2026Hunter HeidenreichSubword Tokenization Unigram Tokenizer
Independent Researcher
DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyGenerative Pretrained Transformers Transformer Architectures
CNRS, LORIA · Nancy, 54000, France · CNRS +1
PortBERT: Navigating the Depths of Portuguese Language Models
Jun 1, 2026Raphael Scheible-Schmitt, Henry He, Armando B. MendesBert-Based Models Portuguese
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · IS2E - Intelligent Systems, Science and Engineering, LIACC polo on Azores University, Ponta Delgada, Portugal
Incremental BPE Tokenization
May 29, 2026Shenghu Jiang, Ruihao GongByte-Pair Encoding Tokenizer
1Beihang University · 2SenseTime Research.
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
May 28, 2026Rohan ShravanByte-Pair Encoding Tokenizer
The School of AI India
Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
May 13, 2026Ruan Visser, Trienko Grobler, Marcel DunaiskiSubword Tokenization Pretraining
Department of Computer Science, Stellenbosch University, Stellenbosch, South Africa
From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
May 12, 2026Mingcheng Zhu, Zhiyao Luo, Yu Liu +1Multi-Token Prediction Electronic Health Records
Department of Engineering Science, University of Oxford, Oxford, United Kingdom.
MambaNetBurst: Direct Byte-level Network Traffic Classification without Tokenization or Pretraining
May 11, 2026Gayan K. Kulatilleke, Siamak Layeghy, Mahsa Baktashmotlagh +1Mamba-Based Models Malware
University of Queensland, Brisbane, Australia
Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment
May 1, 2026Tung-Ling Li, Hongliang Liu, Yuhao WuLarge Language Model Safety Large Language Model Alignment
Palo Alto Networks
Flashback: A Reversible Bilateral Run-Peeling Decomposition of Strings
Apr 29, 2026Thomas Konstantinovsky, Gur YaariByte-Pair Encoding Decomposition
Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel · Bar-Ilan Institute for Nanotechnology and Advanced Materials, Bar-Ilan University, Ramat Gan, Israel · Department of Pathology, Yale School of Medicine, New Haven, CT 06510, United States
Adapting TrOCR for Printed Tigrinya Text Recognition: Word-Aware Loss Weighting for Cross-Script Transfer Learning
Apr 22, 2026Yonatan Haile Medhanie, Yuanhua NiHandwritten Text Recognition Optical Character Recognition
College of Software Engineering, Nankai University, Tianjin, China · College of Artificial Intelligence, Nankai University, Tianjin, China
Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspective
Apr 20, 2026Meifang Chen, Zhe Yang, Huang Nianchen +4Data Leakage Token-Level Entropy
†The Chinese University of Hong Kong · ‡Nanyang Technological University · §University of Southern California +1
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
Aug 6, 2025Negar Foroutan, Clara Meister, Debjit Paul +4Byte-Pair Encoding Cross-Lingual Consistency
EPFL · University of Zurich

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

Hyunsik Kim, Youngmoon Jung

Samsung Research

Byte-Pair Encoding Unicode U+2014 Cross-Lingual Consistency

Open paper AskPDF arXiv

Byte-level byte-pair encoding (BBPE) tokenizers are attractive for multilingual large language models (LLMs) because they cover all Unicode text. In UTF-8-based BBPE, however, many scripts start from a higher fallback cost than English: when no learned merges can be applied, a multibyte character requires multiple byte-derived symbols. We call this worst-case pre-merge cost the encoding floor. A higher floor can increase token counts and per-request cost and shrink usable context. Changing the text encoding can reduce this gap, but a single global encoding can make already-efficient English spans more expensive in mixed-script text. We propose Universal Byte-Level Encoding (UBE), a dual-alphabet tokenizer that keeps 1-2-byte UTF-8 characters on the UTF-8 path while routing 3-4-byte UTF-8 characters through UTF-16. This lowers the encoding floor for 3-byte Basic Multilingual Plane (BMP) characters in scripts with high token premiums (token counts relative to English) without raising it for already-efficient spans in mixed-script text. UBE changes only the byte representation presented to byte-pair encoding (BPE); the merge rule remains standard, and exact decoding is preserved. UBE also composes with alternative boundary policies and morphology-based representations. In a Unicode 17 audit, UBE exactly round-trips all Unicode scalar values and all inputs in the official normalization, grapheme-break, and emoji test suites. Across intrinsic evaluations, UBE lowers dispersion in English-normalized token-count ratios, reducing cross-lingual token-budget disparity. In multilingual language model (LM) experiments, UBE matches BBPE's LM quality. In the main multilingual settings, UBE reduces token counts most for high-premium scripts and slightly lowers English token counts, yielding more usable context under fixed token budgets and faster prompt processing in content-matched benchmarks.

Momentum

Related topics

Latest papers 29