cs.CLNov 3, 2026
SaveTokenizer Myths and Where To Find Them
Organizations: Stanford Computer Science
Abstract
Tokenization is a fundamental component of modern language models: before a model sees data, its tokenizer has already determined how the data appears. We identify and challenge three myths in the tokenization research literature. Myth #1: Bits-Per-Byte (BPB) is agnostic to the tokenizer and/or the vocabulary size: BPB is tokens per byte times bits per token, and the two factors do not cancel; using untrained models, we show that BPB prefers some tokenizers over others: the model has learned nothing, but the tokenizer--model pair has. Myth #2: lower BPB means a better model: On untrained models, BPB strongly prefers some vocabulary sizes, even though accuracy remains at chance, and on real models trained on the same data, BPB correlates weakly with downstream performance. Myth #3: different "tokenizer-agnostic" metrics yield consistent results: The "unigram-normalized loss" is not tokenizer-agnostic; its reported correlation of 0.989 with BPB comes from a confounder and falls to 0.057 once fixed; from the same data the two metrics select different vocabularies at each model size. We present a tokenizer trilemma: no metric can score only the true distribution best, give the same predictions the same score, and tie all untrained models, so choosing a metric is choosing a research question. We conclude with a caution for reading the tokenization literature and a conjecture for future work.
Figures & tables
Figure 1: Myth #1: Bits-Per-Byte Is Agnostic to the Tokenizer and/or Its Vocabulary Size. For untrained models that output uniform distributions, BPB on held-out test data prefers certain vocabulary sizes over others (right). For uniform distributions, BPB is a product of two functions: the number of bits per token, which increases with vocabulary size (left), and the number of tokens per byte, which decreases with vocabulary size (center). The product has a minimum (right). Corpus: FineWeb-Edu-dedup, English ( Penedo et al., 2024 ) .
Figure 2: Myth #1, Continued: Bits-Per-Byte Is Agnostic to the Tokenization Algorithm. The three panels of Figure 1 for SuperBPE ( Liu et al., 2025 ) (color) against standard BPE (gray, dashed), for untrained models on the same held-out corpus. Up to the transition point K the two tokenizers coincide. Above , SuperBPE merges across pre-token boundaries and scores lower at every vocabulary size (shaded). Beyond million, where the BPE sweep ends, the dotted BPE curve holds tokens per byte fixed. BPB prefers one tokenization algorithm (SuperBPE) over another (BPE), even for untrained models. Corpus: FineWeb-Edu-dedup, English ( Penedo et al., 2024 ) .
Figure 3: Myth #2: Lower Bits-Per-Byte Means a Better Model (Option-Letter Scoring). Untrained models that output uniform distributions. Top: of the correct label drops one step at each merge that shortens a label (inset), then rises with . Bottom: accuracy is constant, at the fraction of items whose correct answer is listed first. Red band, for reference: central of a uniform random guesser; black dashed: always choosing the first option. strongly prefers a vocabulary of a few hundred tokens, whereas accuracy finds all models to be equally useless.
| Option-Letter Scoring | Answer-Text Scoring | |
|---|---|---|
| Prompt | Question: Which piece of safety equipment is used to keep mold spores from entering the respiratory system? A. safety goggles B. breathing mask C. rubber gloves D. lead apron Answer: | Question: Which piece of safety equipment is used to keep mold spores from entering the respiratory system? Answer: |
| Scored strings | ␣A , ␣B , ␣C , ␣D | ␣safety goggles , ␣breathing mask , ␣rubber gloves , ␣lead apron |
| Bytes charged |
Table 1: Two Standard Ways to Score Multiple-Choice Question-Answering Problems. Example problem from ARC-Easy ( Clark et al., 2018 ) . The correct option is bold, and ␣ marks a leading space. Accuracy compares the four scored strings; charges the correct one.
Figure 4: Myth #2: Lower Bits-Per-Byte Means a Better Model (Answer-Text Scoring). Untrained models that output uniform distributions, scored on answer text (WinoGrande: on a shared continuation). Top: of the correct answers has an interior minimum on every benchmark (star), between K and K. Bottom: accuracy on the same strings moves by at most a few points and has no common preferred vocabulary size. Red band, for reference: central of a uniform random guesser; dashed: always choosing the first option. concludes that vocabulary sizes between K and K are strongly preferable, whereas accuracy finds all models to be equally useless.
Figure 5: Myth #2: Lower Bits-Per-Byte Means a Better Model (Trained Models). Each point is one model-tokenizer pair from Meister (2026) : every model has the same layers and saw the same documents in the same order; the tokenizer varies. The horizontal axis is on the benchmark’s own text; for BLiMP, the grammatical sentence of each pair, scored alone. Each panel gives the Pearson ( ) and Spearman ( ) correlations between and downstream performance. Dashed line and band: least-squares line with a bootstrap band. On trained models, the correlation is relatively weak, with or without the outlier tokenizer (red cross, Appendix F ).
Figure 6: Myth #2, Continued: Lower Bits-Per-Byte Means a Better Model (Where the Differences Lie). Each marker is the least-squares slope of the decision term on across the tokenizers of one set, with its bootstrap interval; the format term’s slope is one minus it. Most of the difference lies in the format term, which accuracy ignores, not in the decision term, which it reads.
Figure 7: Myth #3: Different “Tokenizer-Agnostic” Metrics Yield Consistent Results (Correlation). The points of Figure 11 of Tao et al. (2024) , digitized and colored by model size, with least-squares fits and confidence bands (bootstrap on the left and right; analytic in the center, where each model size has few points). Left: pooled, and BPC correlate at . Center: within each model size, where the points differ in vocabulary size, the correlation ranges from to . Right: with each size’s mean removed from both metrics, . The pooled correlation comes from differences between model sizes; within a size, little remains.
Figure 8: Myth #3: Different “Tokenizer-Agnostic” Metrics Yield Consistent Results (Minima). Left and center: the digitized runs of Figure 11 of Tao et al. (2024) against vocabulary size; stars mark each model size’s minimum, and hollow diamonds show candidate vocabularies for five runs whose match is ambiguous. Right: the vocabulary at each minimum. BPC is always lowest at the largest vocabulary trained, and its optimum may lie beyond it (arrows); is lowest inside the grid. Unigram-normalized loss and BPC pick different vocabularies from the same data.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| At the minimum | At the largest | |||
|---|---|---|---|---|
| Corpus and tokenizer | Largest | |||
| English, BPE | ||||
| Code, BPE | ||||
| French, BPE | ||||
| Hindi, BPE | ||||
| Russian, BPE | ||||
Table 2: Myth #1: Bits-Per-Byte Is Agnostic to the Tokenizer and/or Its Vocabulary Size. The minimum of for untrained uniform models in each vocabulary sweep, on each corpus’s held-out split. Each minimum is the smallest observed value: at every integer vocabulary size for BPE, and on the grid of sizes for SuperBPE. Chinese reaches its smallest value near the end of its sweep, where is flat to six decimal places, so its is a lower bound and we count no minimum. Where is lowest for untrained models depends on the corpus and the tokenization algorithm.
Figure 9: Table 2 of Tao et al. (2024) , With Its Reported Error Bars. Each panel is one benchmark, and each point is one trained model. The bars are the reported values, which are numerically consistent with the standard errors of the mean over test items.
Figure 10: Figure 5 Without the Outlier Tokenizer. Axes, markers, and statistics as in Figure 5 , now over 83 scored tokenizers (74 for the three generated tasks), without whitespace-balanced-unigram . Without the outlier, every correlation stays weak.
| measured on | BLiMP | GSM8K | MBPP | HumanEval |
|---|---|---|---|---|
| Held-out mixture | ||||
| Code | ||||
| Multilingual |
Table 3: The Sign of a Correlation Between a Code Length and a Score Depends on the Text the Code Length Is Measured On. Spearman correlation across TokEval’s tokenizers between a released and a released score, with bootstrap intervals; for BLiMP and for the other three. A negative sign means that the shorter code goes with the higher score. The held-out mixture is TokEval’s own validation text; the code and multilingual texts are its released code and FLORES evaluations.
| TokSuite’s Rule (Used) | Harness’s Rule | ||
| Benchmark | Scored | Slope Interval | Slope Interval |
| TokSuite, 14 tokenizers | |||
| MMLU | Letter | ||
| MMLU | Text | ||
| ARC-Easy | Letter | ||
| ARC-Easy | Text | ||
Table 4: The Decision Term’s Slope in All Sixteen Conditions, Under Both Scoring Rules. Slope: across tokenizers, with its bootstrap interval. Figure 6 draws all sixteen under TokSuite’s rule; the harness’s rule changes only TokSuite’s numbers.
Figure 11: TokSuite: The Format and Decision Terms of Every Tokenizer. One bar per tokenizer on the answer text of each benchmark, sorted by the code length of the correct answer: the format term, then the decision term, with the accuracy at the end of the bar. The box gives the Spearman correlation of the code length with accuracy across tokenizers and its bootstrap interval.
Figure 12: TokEval: The Format and Decision Terms of Every Tokenizer. As Figure 11 , for the 24 TokEval tokenizers, which saw the same documents in the same order; runs of one tokenizer are averaged.