Early-exit language models decide for each token how much computation to spend, usually by stopping once the prediction is confident. Yet a token is not a fixed unit: tokenizers cut words into different numbers of pieces in different languages. We ask where inside a word such a policy spends its budget. In Qwen3, along two adaptive axes (the timesteps of a spiking conversion and the layers of the original Transformer), confidence stopping gives the continuation pieces of a word less computation than its first piece in English, German, and Turkish, but more in Bangla. Segmentation drives this. Re-segmenting the same parallel passages with BPE-dropout shifts the gap by up to eight exits on both axes, small models trained under two tokenizers shift Bangla's gap by six exits, and BLOOM and mGPT, which split Bangla less finely, give it the English pattern. Characters per token do not predict the gap across languages; within-word surprisal explains 75% of its variance. A learned stopping head beats confidence stopping by 7 to 8 points, but gains less than one point over the best mix of fixed exits at equal cost, and sometimes loses. All results come from offline analysis, not measured speedups.
Figures & tables
Figure 1: Experimental design, and where each part is reported. (1) We change segmentation in three ways: across the tokenizers of pretrained models, with BPE-dropout on a fixed model, and by training small models under two tokenizers. (2) Each model gives a prediction at every exit along one of two adaptive axes. (3) Stopping policies choose one exit per token at the same average budget. (4) We compare the exits that word starts and continuations receive, split that gap into what tokens need and what the policy adds, and compare policies at equal cost. Icons are schematic.
Figure 2: How we measure allocation. This is a schematic: the segmentation and values are illustrative, not model output. (a) Confidence stopping gives each token its own exit. Pieces that begin with the tokenizer’s space marker are word starts ( I ), the rest are continuations ( C ), and the allocation gap G is the difference between their mean exits. An exit is a spiking timestep on one axis and a layer read through a tuned lens on the other. (b) For one token, the exit ti is where confidence first reaches the threshold τ , and the demand di is where the loss first comes within ε of its final value. Service si=ti−di is positive when the policy keeps computing after the token is done.
Figure 3: Re-segmenting the same 60 parallel passages with BPE-dropout, at a budget of four exits. Every panel uses the same realized tokens. Points are means over three segmentation draws, and error bars show one standard deviation. (a) Spiking timesteps: English, German, and Turkish change sign, while Bangla starts positive and stays positive. (b) Layers of the unconverted model, read through one tuned lens per language that is fitted once and then frozen. Solid lines use a lens fitted on both canonical and fragmented text, dotted lines a lens fitted on canonical text only, and hollow markers two further lens seeds. (c) The timestep curves against characters per token. At equal granularity the languages sit several exits apart.
Figure 4: Allocation gap against surprisal contrast for all 24 canonical corpus–model pairs: two Qwen3 sizes on both axes, and BLOOM-1.1B and mGPT-1.3B on depth. Filled markers are 0.6B or the other families, hollow markers 1.7B; blue marks English corpora and pink Bangla ones. Every pair whose gap differs from zero lies in the upper-right or lower-left quadrant. Where Bangla falls depends on how finely its tokenizer splits it, and not monotonically.
Setting
Corpus
Gdem
G
Gsvc
Time / 0.6B
WikiText
−1.89
−1.42
+0.47∗
Belebele-en
−1.43
−1.76
−0.33∗
Belebele-bn
+2.51
+3.19
+0.68∗
Sangraha-bn
+2.52
+3.27
+0.75∗
Time / 1.7B
WikiText
−1.56
−1.60
−0.05
Belebele-en
−1.86
−1.93
−0.07
Table 1: Allocation gap G split into demand Gdem and service Gsvc , in exits, at a budget of four and canonical segmentation. Asterisks mark service gaps whose cluster-bootstrap interval excludes zero. Demand has the sign of G in eleven of twelve settings and makes up roughly half of it or more in each of them. The exception, WikiText on depth, disappears under the stable target of Appendix F .
Axis / size
Corpus
Conf.
Fixed@ tˉH
Head
Δconf
Δfix
tˉH
Time / 0.6B
WikiText
28.39
35.51
35.78
+7.39
+0.27 ∗
3.68
Belebele-en
27.14
34.16
34.03
+6.89
-0.14
3.95
Belebele-bn
46.05
53.48
54.20
+8.15
+0.71 ∗
3.78
Sangraha-bn
46.56
53.91
54.42
+7.87
+0.52 ∗
4.02
Time / 1.7B
WikiText
40.20
42.77
43.03
+2.83
+0.26
3.78
Belebele-en
36.24
39.38
39.58
+3.34
+0.20
3.78
Table 2: Next-token accuracy (%) at a development budget of four exits. Conf. is confidence stopping and Head the learned head, whose realized cost on the reporting split is tˉH . Fixed@ tˉH is the best accuracy a mix of two fixed exits reaches at that cost (the upper concave envelope of the fixed-exit curve on the same split). Δconf and Δfix are the head’s margins over these two references in points, and asterisks mark Δfix intervals that exclude zero. Depth cost is a sampled-exit index.
Figure 5: Demand gap on the same rungs and tokens as Figure 3 . On timesteps (a), fragmentation increases how much more computation continuation pieces need than word starts. On depth (b), through the pooled lens, it decreases it. Allocation rises on both axes, so outside Bangla the policy follows demand on timesteps and moves against it on depth.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Allocation, demand, and service gaps of the 0.6B model at budget four under six threshold-calibration draws (the main run and five re-conversions on different WikiText blocks), evaluation blocks held fixed. Each mark is one draw; the six draws are almost superimposed, and no gap changes sign.
Figure 7: Confidence allocation signs recur across two converted model sizes and the non-spiking tuned-lens probe. Depth values are sampled-exit indices; their magnitudes are not directly comparable hardware costs. WT: WikiText; B: Belebele; S: Sangraha.
Figure 8: Learned-head gains over confidence stopping at development budget four. Bars show archived 95% paired block-bootstrap intervals, conditional on the fitted head and threshold. Gains reproduce across SNN sizes but not uniformly on depth. WT: WikiText; B-en/B-bn: Belebele English/Bangla; S-bn: Sangraha Bangla.
Figure 9: Why the fixed-exit reference must be the envelope. Reporting-split accuracy at each fixed exit (dots), the upper concave envelope reachable by mixing two fixed exits (line), and the learned head at its realized cost (star), Belebele-English at 0.6B. (a) The tuned-lens depth curve is strongly convex over the first eight exits, so the envelope at the head’s cost lies a full point above the adjacent-exit interpolation and the head falls below both. (b) The timestep curve is concave, so the two references coincide.
Language
p=0
.1
.2
.3
.5
.7
.85
English
−1.44
−0.45
+0.44
+1.60
+3.35
+3.42
+3.37
German
−4.00
−3.93
−3.66
−2.97
−0.04
+2.58
+2.96
Turkish
−5.03
−4.73
−4.37
−3.56
−1.42
+1.38
+2.56
Bangla
+3.15
+3.03
+2.97
+2.77
+2.20
+1.80
+1.99
Appendix
Table 3: Mean allocation gap across segmentation draws at each dropout level. Only nonzero rungs have three draws. The near-zero German .5 rung is the only one with draw-level sign disagreement.
Language
p=0
.1
.2
.3
.5
.7
.85
r
G
English
−1.49
−0.66
+0.30
+0.89
+2.45
+3.46
+3.44
+0.96
German
−4.38
−4.39
−4.21
−3.81
−1.13
+2.46
+3.11
+0.97
Turkish
−5.35
−5.17
−5.08
−4.32
−2.11
+1.40
+2.69
+0.98
Bangla
+3.26
+3.10
+2.97
+2.81
+2.42
+2.48
+2.79
−0.71
Gdem
English
−1.94
−1.09
−0.28
+0.23
+1.38
+2.63
+4.28
+0.96
German
−1.66
−1.32
−0.68
−0.03
+1.15
+2.82
+4.87
+0.97
Appendix
Table 4: Passage-aligned intervention: allocation gap G and demand gap Gdem at each dropout level, with the Pearson correlation r between them across the seven rungs. The 60 scored Belebele passages are the same link identifiers in all four languages, and the evaluated character total is identical at every rung of a language. Demand rises with fragmentation everywhere; allocation follows it except in Bangla.
Language
p=0
.1
.2
.3
.5
.7
.85
r
G canonical lens
English
−0.88
−0.47
−0.18
+0.05
+0.45
+0.60
+0.51
−0.86
German
−3.97
−4.01
−3.99
−3.78
−3.13
−1.88
−0.53
−0.96
Turkish
−5.45
−5.48
−5.48
−5.42
−5.01
−3.82
−2.20
−0.97
Bangla
+3.13
+2.95
+2.72
+2.54
+2.27
+2.17
+2.22
+0.98
G pooled lens
English
−0.54
+0.92
+1.93
+2.73
+3.51
+3.60
+3.48
−0.55
German
−3.45
−3.51
−3.38
−2.90
−0.14
+3.02
+3.32
−0.90
Appendix
Table 5: Depth-axis ladder on the same tokens as Table 4 , through two readouts, each fitted once and frozen across all rungs: a lens fitted on canonical text only, and one fitted on the pooled support of the ladder. G is the allocation gap at budget four of 16 exits and Gdem the first-hit demand gap; r is their correlation across the seven rungs. Under the pooled lens the allocation curves match the timestep ladder; under both, the demand gap moves the other way.
Figure 10: Full-compute surprisal contrast predicts allocation across the 15 whitespace-delimited languages. Both axes are derived from the same model, so the relationship describes model behavior rather than establishing an independent linguistic cause.
Figure 11: Descriptive interpolation over the overlapping ranges of English, German, and Turkish. The spread decreases by 5.54-fold. Bangla is excluded by range support, and interpolation adds no new observations.
Corpus
ε
mean d
Gdem
Gsvc
WikiText
.02
7.17
−1.91
+0.49
.05
6.25
−1.89
+0.47
.10
5.31
−1.67
+0.25
.20
4.27
−1.29
−0.13
Belebele-en
.02
6.97
−1.32
−0.44
.05
6.07
−1.43
−0.33
Appendix
Table 6: Demand and service gaps for the 0.6B timestep model as the tolerance ε of Equation 3 varies. The allocation gap G does not depend on ε . Mean demand moves by roughly three exits; the demand gap moves far less and never changes sign. The service gap changes sign only on WikiText.
first hit
stable
Setting
Corpus
Gdem
Gsvc
Gdem
Gsvc
Time / 0.6B
WT
−1.89
+0.47∗
−2.33
+0.91∗
B-en
−1.43
−0.33∗
−1.71
−0.05
B-bn
+2.51
+0.68∗
+3.87
−0.68∗
S-bn
+2.52
+0.75∗
+3.94
−0.67∗
Time / 1.7B
WT
−1.56
−0.05
−2.59
+0.98∗
Appendix
Table 7: The decomposition under both demand targets, ε=0.05 , budget four. Asterisks mark cluster-bootstrap intervals excluding zero. The demand gap matches the allocation sign in all twelve settings under the stable target, including the depth WikiText case that is the exception under first hit. Four service gaps change sign between targets.
B − A by seed
0
1
2
mean (sd)
Bangla ΔG
−6.00
−6.27
−6.06
−6.11 ( 0.14 )
English ΔG
+0.27
+0.15
−0.48
−0.02 ( 0.40 )
Bangla ΔG , eq. tok.
−6.11
–
–
–
Bangla ΔGdems
−0.39
−0.49
−0.57
−0.48 ( 0.09 )
Bangla ΔGsvcs
−5.62
−5.78
−5.49
−5.63 ( 0.14 )
Appendix
Table 8: From-scratch 49.1M models under two tokenizers differing only in Bangla exposure, budget 6.5 of 13 exits, 300 parallel Belebele passages. Seeds pair document order and initialization across arms; superscript s marks the stable demand target. Three seeds are descriptive; no interval over seeds is reported. Within-seed passage-cluster intervals on G have half-widths of 0.06 – 0.15 .
Figure 12: Allocation gap of each from-scratch model, with the two arms of a seed joined by a line. The three Bangla lines fall by six exits in parallel; the three English lines are flat and cross.
Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.
Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short English phrase such as "On the table." is usually produced as four separate predictions for the preposition (On), article (the), noun (table), and punctuation (.), where each consumes a sequence position and adds inference cost. We introduce CoBPE, a compositional tokenization approach that represents such phrases as a lexical base token (table) attached with a small set of reusable surface modifiers, composed in embedding space at input and predicted jointly at output. In controlled pretraining from scratch at 780M and 1.3B scales, CoBPE shortens sequences by 30% and improves average downstream performance by 1.2 points relative to standard BPE under matched training compute. Our results suggest that part of what is now expressed through token sequences can instead be modeled through structured representations, opening a broad design space for more token-efficient and capable language models.
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. We train language models with tokenisers produced by each algorithm, varying: model size, vocabulary sizes, and domain (English-only vs. multilingual). Evaluating models on bits-per-byte, we find that the search procedure -- not the objective -- is the dominant factor: bottom-up tokenisers consistently achieve lower bits-per-byte in most settings. Evaluating models on the BLiMP task, however, shows no consistent relationship between design choice and performance. Overall, our results disentangle the effect of tokeniser design choices on language modelling performance, offering concrete guidance for their more principled construction.