Token Efficiency

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 22

All topics

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities
Oct 1, 2026Hyunsik Kim, Youngmoon JungMultilingual Language Models Token Efficiency
Samsung Research
The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype
Sep 30, 2026Thomas ServalMultilingual Language Models Token Efficiency
Baracoda AI Labs & NEOMA Business School
SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
Sep 17, 2026Haozhe Liu, Tian Ye, Sensen Gao +11Agent Harness Optimization LLM Agent Workflow Optimization
NVIDIA · NTU · MIT
Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition
Sep 14, 2026Devansh SharmaToken Efficiency Low-Resource Language Processing
Independent researcher
Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities
Aug 10, 2026Avijit Roy, Proma Roy, Hrishitva PatelAlgorithmic Fairness Token Efficiency
John Jay College of Criminal Justice, City University of New York, USA · The City College of New York, City University of New York, USA · University of Texas at San Antonio, USA
Explicit Boundary Markers for Subword Vocabularies
Aug 9, 2026Sander Land, Clara MeisterToken Efficiency Language Modeling
EPFL
The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
Jul 24, 2026Zixuan Wu, Carolyn Jane Anderson, Arjun GuhaAI Coding Agents Code Generation
Northeastern University · Wellesley College
Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images
Jul 23, 2026Ronak BhalgamiImage Tokenization Token Efficiency
Mach-Mind-4-Flash Technical Report
Jul 10, 2026Foundation Model TeamAgentic RL Token Efficiency
Foundation Model, Li Auto Inc.
The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI
Jul 8, 2026Muayad Sayed Ali, Aliaksandra Novik, Anji Boddupally +28LLM Inference Efficiency LLM Agent Orchestration
Writer, Inc.
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Jun 25, 2026Sander LandToken Efficiency Subword Tokenization
Writer, Inc.
Token Complexity Theory for AI-Augmented Computing
Jun 10, 2026Jie WangToken Efficiency
Richard Miner School of Computing and Information Sciences, University of Massachusetts, Lowell, MA 01854, USA.
AI Tokenomics: The Economics of Tokens, Computation, and Pricing in Foundation Models
Jun 10, 2026Quanyan ZhuCost-Aware Inference Token Efficiency
Department of Electrical and Computer Engineering, New York University Tandon School of Engineering, 370 Jay Street, Room 1004, Brooklyn, NY 11201, USA
Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems
May 28, 2026Lorenz Kutschka, Bernhard GeigerTool-Augmented Language Model Agents AI Agent Evaluation
Know Center Research GmbH / Sandgasse 34, A-8010 Graz · Signal Processing and Speech Communication, Graz University of Technology / Inffeldgasse 16c, A-8010 Graz · Graz Center for Machine Learning // A-8010 Graz
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
May 28, 2026Rohan ShravanToken Efficiency Subword Tokenization
The School of AI India
Token Optimization Strategies for LLM-Based Oracle-to-PostgreSQL Migration
May 27, 2026Oleg Grynets, Dmytro Babarytskyi, Vasyl LyashkevychToken Efficiency Prompt Compression
EPAM Systems McLean, Virginia, USA · EPAM Systems Kharkiv, Ukraine · EPAM Systems Lviv, Ukraine
Tokenisation via Convex Relaxations
May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2Token Efficiency Linear Programming
ETH Zurich · Kensho Technologies
Tokenization with Split Trees
May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4Token Efficiency Subword Tokenization
Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA
Token Economics for LLM Agents: A Dual-View Study from Computing and Economics
May 9, 2026Yuxi Chen, Junming Chen, Chenyu He +9Token Efficiency
College of Computer Science and Technology, Zhejiang University · School of Economics, Zhejiang University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University +1
AgentSlimming: Towards Efficient and Cost-Aware Multi-Agent Systems
May 9, 2026Yulang Chen, Haoxuan Peng, Jinyan Liu +3Multi-Agent LLM Systems Multi-Agent System Optimization
1Shanghai Jiao Tong University · 2Nanjing University · 3The University of Sydney +2
ONTO: A Token-Efficient Columnar Notation for LLM Input Optimization
Apr 19, 2026Harshavardhanan DeekeswarLLM Inference Efficiency Token Efficiency
Independent Researcher · Chennai, India
Less Is More: Reducing Token Counts Without Compromising Performance
Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference Acceleration Token Efficiency
Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

Hyunsik Kim, Youngmoon Jung

Samsung Research

Multilingual Language Models Token Efficiency Subword Tokenization Byte-Pair Encoding

Open paper AskPDF arXiv

Byte-level byte-pair encoding (BBPE) tokenizers are attractive for multilingual large language models (LLMs) because they cover all Unicode text. In UTF-8-based BBPE, however, many scripts start from a higher fallback cost than English: when no learned merges can be applied, a multibyte character requires multiple byte-derived symbols. We call this worst-case pre-merge cost the encoding floor. A higher floor can increase token counts and per-request cost and shrink usable context. Changing the text encoding can reduce this gap, but a single global encoding can make already-efficient English spans more expensive in mixed-script text. We propose Universal Byte-Level Encoding (UBE), a dual-alphabet tokenizer that keeps 1-2-byte UTF-8 characters on the UTF-8 path while routing 3-4-byte UTF-8 characters through UTF-16. This lowers the encoding floor for 3-byte Basic Multilingual Plane (BMP) characters in scripts with high token premiums (token counts relative to English) without raising it for already-efficient spans in mixed-script text. UBE changes only the byte representation presented to byte-pair encoding (BPE); the merge rule remains standard, and exact decoding is preserved. UBE also composes with alternative boundary policies and morphology-based representations. In a Unicode 17 audit, UBE exactly round-trips all Unicode scalar values and all inputs in the official normalization, grapheme-break, and emoji test suites. Across intrinsic evaluations, UBE lowers dispersion in English-normalized token-count ratios, reducing cross-lingual token-budget disparity. In multilingual language model (LM) experiments, UBE matches BBPE's LM quality. In the main multilingual settings, UBE reduces token counts most for high-premium scripts and slightly lowers English token counts, yielding more usable context under fixed token budgets and faster prompt processing in content-matched benchmarks.

Momentum

Related topics

Latest papers 22