Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Ben Gubler

cs.CLMar 25, 2026

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Authors: Ben Gubler

Abstract

Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.

Figures & tables

Tokenizer	Vocab Size	Year
GPT-2	50,257	2019
GPT-4	100,263	2023
gpt-oss	200,019	2025
Llama 3.1	128,256	2024
Gemma 3	262,145	2025
Qwen3	151,669	2025

Table 1: Tokenizers evaluated in Tokka-Bench .

Metric	Description
Bytes/Token	Average UTF-8 bytes per token ( $\frac{\text{total\_bytes}}{\text{total\_tokens}}$ ). Higher = more efficient.
Unique Tokens	Count of distinct token IDs used. Higher = better vocabulary coverage.
Subword Fertility	Tokens per linguistic unit ( $\frac{\text{total\_tokens}}{\text{total\_units}}$ ). Lower = closer to 1:1 mapping.
Word-Split %	Percentage of units requiring $>$ 1 token. Lower = better boundary alignment.
Vocab Composition	Script distribution across decoded vocabulary (aggregated).

Table 2: Per-language metrics in Tokka-Bench .

Figure 1: Tokenization efficiency (bytes per token) across 10 languages.

Figure 2: Unique token count when encoding 2 MB per language.

Figure 3: Subword fertility (tokens per linguistic unit) across 17 languages and 7 tokenizers. Values near 1.0 are optimal; red cells indicate high fragmentation. Units are language-dependent: words for alphabetic scripts, characters for CJK/Thai/Khmer.

Figure 4: Code tokenization efficiency. Modern tokenizers show near-identical performance despite differing natural-language profiles.

Figure 5: Vocabulary size vs. mean efficiency across natural languages.

Tokenization in Language Models

See all 46 papers

TokEval: A Tokenizer Evaluation Suite
Aug 18, 2026Clara MeisterLLM Evaluation Tokenization
EPFL
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Dec 23, 2025Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4Subword Tokenization Tokenization in Language Models
University of Toronto · Vector Institute · Google DeepMind +4
Less Is More: Reducing Token Counts Without Compromising Performance
Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference Acceleration Token Efficiency
Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University

Aug 18, 2026J/K move · Enter open · S save

TokEval: A Tokenizer Evaluation Suite

Clara Meister

EPFL

LLM Evaluation Tokenization Tokenization in Language Models

Open paper AskPDF arXiv

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers' training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.

Abstract

Tokenization in Language Models

Explore similar work

TokEval: A Tokenizer Evaluation Suite

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Less Is More: Reducing Token Counts Without Compromising Performance