Unicode U+2014

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 14

All topics

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities
Oct 1, 2026Hyunsik Kim, Youngmoon JungByte-Pair Encoding Unicode U+2014
Samsung Research
DeBERTa-ConPara: Attack-Aware and Deployment-Realistic Detection of AI-Generated Text
Oct 1, 2026Mohamed Mady, Yupei Li, Johannes Reschke +1Machine-Generated Text Detection Bert-Based Models
CHI, Chair of Health Informatics, Technical University of Munich, Germany · Smart Embedded Systems Lab, OTH Regensburg, Germany · GLAM, Group on Language, Audio & Music, Imperial College London, UK
Yorùbá in Unicode: An Overview of a Problem
Sep 27, 2026Kólá TúbòsúnUnicode U+2014 African Languages
Yoruba Names Project
Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
Aug 27, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosMultilingual Language Models Language Pairs
George Mason University
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?
Aug 4, 2026Adnan Al Ali, Kathy Hämmerl, Jindřich Libovický +1Translation Quality Unicode U+2014
Charles University, Faculty of Mathematics and Physics, Czech Republic · Technical University of Munich, Germany · Munich Center for Machine Learning
Token-Native Storage: Read and Write in your Agent's Language
Aug 3, 2026Kumar ShivenduByte-Pair Encoding Token Embeddings
Qdrant
grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP
Jul 24, 2026Izzath Nisfer, Ashini Kavindya, Ovindu Atukorala +2Optical Character Recognition Unicode U+2014
Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka · Computer Science and Engineering, University of Nevada, Nevada, USA · Department of Information & Computing Sciences, Utrecht University, Utrecht, The Netherlands
Em-ergence of the em-dash: a population-level rise in em-dash frequency in medRxiv preprints at the dawn of the large-language-model era
Jun 28, 2026Przemysław CzumaPrevalence Scientific Discovery
Polish Association for Artificial Intelligence in Medicine
Stringalign: Moving beyond summary statistics with a transparent Unicode-aware tool for evaluating automatic transcription models
Jun 14, 2026Yngve Mardal Moe, Marie RoaldTranscription Optical Character Recognition
†Independent researcher · ‡The National Library of Norway
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Unicode U+2014 Language Modeling
Google LLC / Mountain View, CA, USA · Institute of Science Tokyo / Tokyo, Japan · Mohamed bin Zayed University of Artificial Intelligence / Abu Dhabi, United Arab Emirates
Invisible to humans, visible to machines: a preregistered audit of Unicode fidelity across four biomedical bibliographic APIs
Jun 8, 2026Przemysław CzumaUnicode U+2014 Openalex Metadata
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
May 28, 2026Rohan ShravanByte-Pair Encoding Tokenizer
The School of AI India
Entropy of Ukrainian
Apr 30, 2026Anton Lavreniuk, Mykyta Mudryi, Markiian ChakloshEntropy Ukrainian
Polish-Japanese Academy of Information Technology · University of the National Education Commission in Kraków
Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique
Oct 7, 2025Yanming Li, Cédric Eichler, Nicolas Anciaux +3Large Language Model Watermarks Machine-Generated Text Detection
PETSCRAFT · Petscraft, Inria, Palaiseau, France · INSA CVL, Bourges, France +6

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

Hyunsik Kim, Youngmoon Jung

Samsung Research

Byte-Pair Encoding Unicode U+2014 Cross-Lingual Consistency

Open paper AskPDF arXiv

Byte-level byte-pair encoding (BBPE) tokenizers are attractive for multilingual large language models (LLMs) because they cover all Unicode text. In UTF-8-based BBPE, however, many scripts start from a higher fallback cost than English: when no learned merges can be applied, a multibyte character requires multiple byte-derived symbols. We call this worst-case pre-merge cost the encoding floor. A higher floor can increase token counts and per-request cost and shrink usable context. Changing the text encoding can reduce this gap, but a single global encoding can make already-efficient English spans more expensive in mixed-script text. We propose Universal Byte-Level Encoding (UBE), a dual-alphabet tokenizer that keeps 1-2-byte UTF-8 characters on the UTF-8 path while routing 3-4-byte UTF-8 characters through UTF-16. This lowers the encoding floor for 3-byte Basic Multilingual Plane (BMP) characters in scripts with high token premiums (token counts relative to English) without raising it for already-efficient spans in mixed-script text. UBE changes only the byte representation presented to byte-pair encoding (BPE); the merge rule remains standard, and exact decoding is preserved. UBE also composes with alternative boundary policies and morphology-based representations. In a Unicode 17 audit, UBE exactly round-trips all Unicode scalar values and all inputs in the official normalization, grapheme-break, and emoji test suites. Across intrinsic evaluations, UBE lowers dispersion in English-normalized token-count ratios, reducing cross-lingual token-budget disparity. In multilingual language model (LM) experiments, UBE matches BBPE's LM quality. In the main multilingual settings, UBE reduces token counts most for high-premium scripts and slightly lowers English token counts, yielding more usable context under fixed token budgets and faster prompt processing in content-matched benchmarks.

Momentum

Related topics

Latest papers 14