Tokenizer Myths and Where To Find Them

Rylan Schaeffer; Sanmi Koyejo

cs.CLNov 3, 2026

Tokenizer Myths and Where To Find Them

Authors: Rylan Schaeffer, Sanmi Koyejo

Organizations: Stanford Computer Science

Abstract

Tokenization is a fundamental component of modern language models: before a model sees data, its tokenizer has already determined how the data appears. We identify and challenge three myths in the tokenization research literature. Myth #1: Bits-Per-Byte (BPB) is agnostic to the tokenizer and/or the vocabulary size: BPB is tokens per byte times bits per token, and the two factors do not cancel; using untrained models, we show that BPB prefers some tokenizers over others: the model has learned nothing, but the tokenizer--model pair has. Myth #2: lower BPB means a better model: On untrained models, BPB strongly prefers some vocabulary sizes, even though accuracy remains at chance, and on real models trained on the same data, BPB correlates weakly with downstream performance. Myth #3: different "tokenizer-agnostic" metrics yield consistent results: The "unigram-normalized loss" is not tokenizer-agnostic; its reported correlation of 0.989 with BPB comes from a confounder and falls to 0.057 once fixed; from the same data the two metrics select different vocabularies at each model size. We present a tokenizer trilemma: no metric can score only the true distribution best, give the same predictions the same score, and tie all untrained models, so choosing a metric is choosing a research question. We conclude with a caution for reading the tokenization literature and a conjecture for future work.

Figures & tables

Figure 1: Myth #1: Bits-Per-Byte Is Agnostic to the Tokenizer and/or Its Vocabulary Size. For untrained models that output uniform distributions, BPB on held-out test data prefers certain vocabulary sizes over others (right). For uniform distributions, BPB is a product of two functions: the number of bits per token, which increases with vocabulary size (left), and the number of tokens per byte, which decreases with vocabulary size (center). The product has a minimum (right). Corpus: FineWeb-Edu-dedup, English ( Penedo et al., 2024 ) .

Figure 2: Myth #1, Continued: Bits-Per-Byte Is Agnostic to the Tokenization Algorithm. The three panels of Figure 1 for SuperBPE ( Liu et al., 2025 ) (color) against standard BPE (gray, dashed), for untrained models on the same held-out corpus. Up to the transition point

t=262

K the two tokenizers coincide. Above

t

, SuperBPE merges across pre-token boundaries and scores lower at every vocabulary size (shaded). Beyond

1.05

million, where the BPE sweep ends, the dotted BPE curve holds tokens per byte fixed. BPB prefers one tokenization algorithm (SuperBPE) over another (BPE), even for untrained models. Corpus: FineWeb-Edu-dedup, English ( Penedo et al., 2024 ) .

Figure 3: Myth #2: Lower Bits-Per-Byte Means a Better Model (Option-Letter Scoring). Untrained models that output uniform distributions. Top:

\operatorname{BPB}

of the correct label drops one step at each merge that shortens a label (inset), then rises with

\log V

. Bottom: accuracy is constant, at the fraction of items whose correct answer is listed first. Red band, for reference: central

95\%

of a uniform random guesser; black dashed: always choosing the first option.

\operatorname{BPB}

strongly prefers a vocabulary of a few hundred tokens, whereas accuracy finds all models to be equally useless.

	Option-Letter Scoring	Answer-Text Scoring
Prompt	Question: Which piece of safety equipment is used to keep mold spores from entering the respiratory system? A. safety goggles B. breathing mask C. rubber gloves D. lead apron Answer:	Question: Which piece of safety equipment is used to keep mold spores from entering the respiratory system? Answer:
Scored strings	␣A , ␣B , ␣C , ␣D	␣safety goggles , ␣breathing mask , ␣rubber gloves , ␣lead apron
Bytes charged	$2$	$15$

Table 1: Two Standard Ways to Score Multiple-Choice Question-Answering Problems. Example problem from ARC-Easy ( Clark et al., 2018 ) . The correct option is bold, and ␣ marks a leading space. Accuracy compares the four scored strings;

\operatorname{BPB}

charges the correct one.

Figure 4: Myth #2: Lower Bits-Per-Byte Means a Better Model (Answer-Text Scoring). Untrained models that output uniform distributions, scored on answer text (WinoGrande: on a shared continuation). Top:

\operatorname{BPB}

of the correct answers has an interior minimum on every benchmark (star), between

8

K and

39

K. Bottom: accuracy on the same strings moves by at most a few points and has no common preferred vocabulary size. Red band, for reference: central

95\%

of a uniform random guesser; dashed: always choosing the first option.

\operatorname{BPB}

concludes that vocabulary sizes between

8

K and

39

K are strongly preferable, whereas accuracy finds all models to be equally useless.

Figure 5: Myth #2: Lower Bits-Per-Byte Means a Better Model (Trained Models). Each point is one model-tokenizer pair from Meister (2026) : every model has the same layers and saw the same documents in the same order; the tokenizer varies. The horizontal axis is

\operatorname{BPB}

on the benchmark’s own text; for BLiMP, the grammatical sentence of each pair, scored alone. Each panel gives the Pearson (

r

) and Spearman (

\rho

) correlations between

\operatorname{BPB}

and downstream performance. Dashed line and band: least-squares line with a

95\%

bootstrap band. On trained models, the correlation is relatively weak, with or without the outlier tokenizer (red cross, Appendix F ).

Figure 6: Myth #2, Continued: Lower Bits-Per-Byte Means a Better Model (Where the Differences Lie). Each marker is the least-squares slope of the decision term on

\operatorname{BPB}

across the tokenizers of one set, with its

95\%

bootstrap interval; the format term’s slope is one minus it. Most of the difference lies in the format term, which accuracy ignores, not in the decision term, which it reads.

Figure 7: Myth #3: Different “Tokenizer-Agnostic” Metrics Yield Consistent Results (Correlation). The

56

points of Figure 11 of Tao et al. (2024) , digitized and colored by model size, with least-squares fits and

95\%

confidence bands (bootstrap on the left and right; analytic in the center, where each model size has few points). Left: pooled,

\mathcal{L}_{u}

and BPC correlate at

r=0.989

. Center: within each model size, where the points differ in vocabulary size, the correlation ranges from

-0.61

+0.48

. Right: with each size’s mean removed from both metrics,

r=0.057

. The pooled correlation comes from differences between model sizes; within a size, little remains.

Figure 8: Myth #3: Different “Tokenizer-Agnostic” Metrics Yield Consistent Results (Minima). Left and center: the

56

digitized runs of Figure 11 of Tao et al. (2024) against vocabulary size; stars mark each model size’s minimum, and hollow diamonds show candidate vocabularies for five runs whose match is ambiguous. Right: the vocabulary at each minimum. BPC is always lowest at the largest vocabulary trained, and its optimum may lie beyond it (arrows);

\mathcal{L}_{u}

is lowest inside the grid. Unigram-normalized loss and BPC pick different vocabularies from the same data.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

		At the minimum		At the largest $V$
Corpus and tokenizer	Largest $V$	$V$	$\operatorname{BPB}$	$\operatorname{BPB}$
English, BPE	$2^{20}$	$24{,}114$	$3.2386$	$3.9320$
Code, BPE	$2^{20}$	$8{,}271$	$4.2467$	$5.3694$
French, BPE	$2^{20}$	$26{,}977$	$3.3195$	$3.9718$
Hindi, BPE	$2^{18}$	$504$	$2.5790$	$4.5190$
Russian, BPE	$2^{20}$	$117{,}360$	$1.8260$	$1.9632$

Appendix

Table 2: Myth #1: Bits-Per-Byte Is Agnostic to the Tokenizer and/or Its Vocabulary Size. The minimum of

\operatorname{BPB}

for untrained uniform models in each vocabulary sweep, on each corpus’s held-out split. Each minimum is the smallest observed value: at every integer vocabulary size for BPE, and on the grid of

67

sizes for SuperBPE. Chinese reaches its smallest value near the end of its sweep, where

\operatorname{BPB}

is flat to six decimal places, so its

V

is a lower bound and we count no minimum. Where

\operatorname{BPB}

is lowest for untrained models depends on the corpus and the tokenization algorithm.

Figure 9: Table 2 of Tao et al. (2024) , With Its Reported Error Bars. Each panel is one benchmark, and each point is one trained model. The bars are the reported

\pm

values, which are numerically consistent with the standard errors of the mean over test items.

Figure 10: Figure 5 Without the Outlier Tokenizer. Axes, markers, and statistics as in Figure 5 , now over 83 scored tokenizers (74 for the three generated tasks), without whitespace-balanced-unigram . Without the outlier, every correlation stays weak.

$\operatorname{BPB}$ measured on	BLiMP	GSM8K	MBPP	HumanEval
Held-out mixture	$+0.11$ $[-0.13,+0.34]$	$-0.65$ $[-0.78,-0.47]$	$-0.16$ $[-0.39,+0.09]$	$-0.26$ $[-0.45,-0.05]$
Code	$+0.17$ $[-0.06,+0.38]$	$-0.59$ $[-0.74,-0.39]$	$-0.18$ $[-0.40,+0.06]$	$-0.31$ $[-0.48,-0.11]$
Multilingual	$+0.31$ $[+0.10,+0.48]$	$-0.14$ $[-0.37,+0.09]$	$+0.36$ $[+0.15,+0.54]$	$-0.04$ $[-0.26,+0.18]$

Appendix

Table 3: The Sign of a Correlation Between a Code Length and a Score Depends on the Text the Code Length Is Measured On. Spearman correlation across TokEval’s tokenizers between a released

\operatorname{BPB}

and a released score, with

95\%

bootstrap intervals;

n=84

for BLiMP and

74

for the other three. A negative sign means that the shorter code goes with the higher score. The held-out mixture is TokEval’s own validation text; the code and multilingual texts are its released code and FLORES evaluations.

		TokSuite’s Rule (Used)	Harness’s Rule
Benchmark	Scored	Slope $[95\%$ Interval $]$	Slope $[95\%$ Interval $]$
TokSuite, 14 tokenizers
MMLU	Letter	$-0.01$ $[-0.11,+0.04]$	$+0.04$ $[-0.10,+0.04]$
MMLU	Text	$+0.33$ $[+0.25,+0.49]$	$+0.19$ $[+0.15,+0.49]$
ARC-Easy	Letter	$+0.04$ $[+0.01,+0.08]$	$+0.04$ $[+0.02,+0.07]$
ARC-Easy	Text	$+0.04$ $[-0.12,+0.17]$	$+0.06$ $[-0.08,+0.17]$

Appendix

Table 4: The Decision Term’s Slope in All Sixteen Conditions, Under Both Scoring Rules. Slope:

\operatorname{Cov}(D,B)/\operatorname{Var}(B)

across tokenizers, with its

95\%

bootstrap interval. Figure 6 draws all sixteen under TokSuite’s rule; the harness’s rule changes only TokSuite’s numbers.

Figure 11: TokSuite: The Format and Decision Terms of Every Tokenizer. One bar per tokenizer on the answer text of each benchmark, sorted by the code length of the correct answer: the format term, then the decision term, with the accuracy at the end of the bar. The box gives the Spearman correlation of the code length with accuracy across tokenizers and its

95\%

bootstrap interval.

Figure 12: TokEval: The Format and Decision Terms of Every Tokenizer. As Figure 11 , for the 24 TokEval tokenizers, which saw the same documents in the same order; runs of one tokenizer are averaged.

Tokenization in Language Models

See all 54 papers

Sep 16, 2026

Objective vs. Search: Decomposing What Makes a Good Tokeniser

Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. We train language models with tokenisers produced by each algorithm, varying: model size, vocabulary sizes, and domain (English-only vs. multilingual). Evaluating models on bits-per-byte, we find that the search procedure -- not the objective -- is the dominant factor: bottom-up tokenisers consistently achieve lower bits-per-byte in most settings. Evaluating models on the BLiMP task, however, shows no consistent relationship between design choice and performance. Overall, our results disentangle the effect of tokeniser design choices on language modelling performance, offering concrete guidance for their more principled construction.

Ahmetcan Yavuz, Clara Meister, Tiago Pimentel

ETH Zürich · EPFL

Subword Tokenization Byte-Pair Encoding Tokenization in Language Models Tokenization

Ask PDF

Oct 8, 2026

Language-Specific Effects of Tokenizer Choice in Multilingual Language Models

Tokenizer choice affects multilingual language modeling, but vocabulary capacity is finite and vocabulary size is often constrained: improving representation for some languages often comes at the expense of others. We therefore ask whether tokenizer choice matters equally across languages, a question that the current literature leave unanswered. To this end, we train 123 language models spanning 54 tokenizers. In the main comparison, architecture, training corpus, training-token budget, and optimization are held fixed, so the models differ only in their tokenizer. We find that tokenizer choice matters more for languages with less language-model training data: across the 54 tokenizers, the standard deviation of a language's bits-per-byte (BPB) increases as its model training-data share decreases (Spearman rho = -0.52 over the 31 trained languages and -0.69 over the 28 written with word boundaries). Leaving a language out of tokenizer training raises its BPB in every language we study, and the penalty tends to be larger for languages with less language-model training data. Giving lower-resource languages a larger share of tokenizer-training data, however, does not unconditionally help those languages: both equal weighting and an allocation inverting the shares with respect to the language model training data increase their BPB, particularly when language-model training repeats data. Finally, which intrinsic tokenizer properties are associated with better BPB differs across languages, providing further evidence that what makes a good tokenizer depends on the language. We find that the metrics quantifying these properties can be successfully used to predict downstream models' pairwise BPB rankings, suggesting a practical strategy for screening tokenizer candidates before training language models.

Clara Meister, Gül Sena Altıntaş, Antoine Bosselut

EPFL · University of Toronto, Vector Institute

Multilingual Language Models Multilingual Language Model Evaluation Low-Resource Language Processing

Ask PDF

Oct 15, 2026

No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance

Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run. In this paper, we analyze approximately 200 language models trained on nine different languages. We evaluate both models and their tokenizers and test whether there is a relationship between any proposed intrinsic tokenizer evaluations and model performance, as measured by a wide variety of benchmarks. Despite the quantity and variety of proposed intrinsic tokenizer metrics, we find that none consistently predicts downstream performance across model families, tasks, and languages. This motivates the need for novel tokenizer evaluation metrics and approaches. We propose novel linguistically grounded metric introduced in this work shows that new metrics can capture aspects of tokenizer quality that existing metrics miss, though their predictive power is likely to vary across model families, languages, and tasks.

Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava +3

EleutherAI · University of Basel · National Taiwan University of Science and Technology +2

Tokenization in Language Models LLM Evaluation

Ask PDF

Abstract

Tokenization in Language Models

Explore similar work

Objective vs. Search: Decomposing What Makes a Good Tokeniser

Language-Specific Effects of Tokenizer Choice in Multilingual Language Models

No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance