A Token Is Not a Unit of Difficulty: Segmentation Controls Where Adaptive Computation Goes Inside a Word

Dipto Sumit; Mahima Mahzabin Sandria; Farig Sadeque; Md. Golam Rabiul Alam

cs.CLOct 31, 2026

A Token Is Not a Unit of Difficulty: Segmentation Controls Where Adaptive Computation Goes Inside a Word

Authors: Dipto Sumit, Mahima Mahzabin Sandria, Farig Sadeque, Md. Golam Rabiul Alam

Organizations: BRAC University, Dhaka, Bangladesh

Abstract

Early-exit language models decide for each token how much computation to spend, usually by stopping once the prediction is confident. Yet a token is not a fixed unit: tokenizers cut words into different numbers of pieces in different languages. We ask where inside a word such a policy spends its budget. In Qwen3, along two adaptive axes (the timesteps of a spiking conversion and the layers of the original Transformer), confidence stopping gives the continuation pieces of a word less computation than its first piece in English, German, and Turkish, but more in Bangla. Segmentation drives this. Re-segmenting the same parallel passages with BPE-dropout shifts the gap by up to eight exits on both axes, small models trained under two tokenizers shift Bangla's gap by six exits, and BLOOM and mGPT, which split Bangla less finely, give it the English pattern. Characters per token do not predict the gap across languages; within-word surprisal explains 75% of its variance. A learned stopping head beats confidence stopping by 7 to 8 points, but gains less than one point over the best mix of fixed exits at equal cost, and sometimes loses. All results come from offline analysis, not measured speedups.

Figures & tables

Panel of Figure 1 — Figure 1: Experimental design, and where each part is reported. (1) We change segmentation in three ways: across the tokenizers of pretrained models, with BPE-dropout on a fixed model, and by training small models under two tokenizers. (2) Each model gives a prediction at every exit along one of two adaptive axes. (3) Stopping policies choose one exit per token at the same average budget. (4) We compare the exits that word starts and continuations receive, split that gap into what tokens need and what the policy adds, and compare policies at equal cost. Icons are schematic.

Panel of Figure 2 — Figure 2: How we measure allocation. This is a schematic: the segmentation and values are illustrative, not model output. (a) Confidence stopping gives each token its own exit. Pieces that begin with the tokenizer’s space marker are word starts ( $I$ ), the rest are continuations ( $C$ ), and the allocation gap $G$ is the difference between their mean exits. An exit is a spiking timestep on one axis and a layer read through a tuned lens on the other. (b) For one token, the exit $t_{i}$ is where confidence first reaches the threshold $\tau$ , and the demand $d_{i}$ is where the loss first comes within $\varepsilon$ of its final value. Service $s_{i}=t_{i}-d_{i}$ is positive when the policy keeps computing after the token is done.

Figure 3: Re-segmenting the same 60 parallel passages with BPE-dropout, at a budget of four exits. Every panel uses the same realized tokens. Points are means over three segmentation draws, and error bars show one standard deviation. (a) Spiking timesteps: English, German, and Turkish change sign, while Bangla starts positive and stays positive. (b) Layers of the unconverted model, read through one tuned lens per language that is fitted once and then frozen. Solid lines use a lens fitted on both canonical and fragmented text, dotted lines a lens fitted on canonical text only, and hollow markers two further lens seeds. (c) The timestep curves against characters per token. At equal granularity the languages sit several exits apart.

Figure 4: Allocation gap against surprisal contrast for all 24 canonical corpus–model pairs: two Qwen3 sizes on both axes, and BLOOM-1.1B and mGPT-1.3B on depth. Filled markers are 0.6B or the other families, hollow markers 1.7B; blue marks English corpora and pink Bangla ones. Every pair whose gap differs from zero lies in the upper-right or lower-left quadrant. Where Bangla falls depends on how finely its tokenizer splits it, and not monotonically.

Setting	Corpus	$G_{\text{dem}}$	$G$	$G_{\text{svc}}$
Time / 0.6B	WikiText	$-1.89$	$-1.42$	$+0.47^{*}$
	Belebele-en	$-1.43$	$-1.76$	$-0.33^{*}$
	Belebele-bn	$+2.51$	$+3.19$	$+0.68^{*}$
	Sangraha-bn	$+2.52$	$+3.27$	$+0.75^{*}$
Time / 1.7B	WikiText	$-1.56$	$-1.60$	$-0.05$
	Belebele-en	$-1.86$	$-1.93$	$-0.07$

Table 1: Allocation gap

G

split into demand

G_{\text{dem}}

and service

G_{\text{svc}}

, in exits, at a budget of four and canonical segmentation. Asterisks mark service gaps whose cluster-bootstrap interval excludes zero. Demand has the sign of

G

in eleven of twelve settings and makes up roughly half of it or more in each of them. The exception, WikiText on depth, disappears under the stable target of Appendix F .

Axis / size	Corpus	Conf.	Fixed@ $\bar{t}_{H}$	Head	$\Delta_{\mathrm{conf}}$	$\Delta_{\mathrm{fix}}$	$\bar{t}_{H}$
Time / 0.6B	WikiText	28.39	35.51	35.78	+7.39	+0.27 ∗	3.68
	Belebele-en	27.14	34.16	34.03	+6.89	-0.14	3.95
	Belebele-bn	46.05	53.48	54.20	+8.15	+0.71 ∗	3.78
	Sangraha-bn	46.56	53.91	54.42	+7.87	+0.52 ∗	4.02
Time / 1.7B	WikiText	40.20	42.77	43.03	+2.83	+0.26	3.78
	Belebele-en	36.24	39.38	39.58	+3.34	+0.20	3.78

Table 2: Next-token accuracy (%) at a development budget of four exits. Conf. is confidence stopping and Head the learned head, whose realized cost on the reporting split is

\bar{t}_{H}

. Fixed@

\bar{t}_{H}

is the best accuracy a mix of two fixed exits reaches at that cost (the upper concave envelope of the fixed-exit curve on the same split).

\Delta_{\mathrm{conf}}

and

\Delta_{\mathrm{fix}}

are the head’s margins over these two references in points, and asterisks mark

\Delta_{\mathrm{fix}}

intervals that exclude zero. Depth cost is a sampled-exit index.

Figure 5: Demand gap on the same rungs and tokens as Figure 3 . On timesteps (a), fragmentation increases how much more computation continuation pieces need than word starts. On depth (b), through the pooled lens, it decreases it. Allocation rises on both axes, so outside Bangla the policy follows demand on timesteps and moves against it on depth.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Allocation, demand, and service gaps of the 0.6B model at budget four under six threshold-calibration draws (the main run and five re-conversions on different WikiText blocks), evaluation blocks held fixed. Each mark is one draw; the six draws are almost superimposed, and no gap changes sign.

Figure 7: Confidence allocation signs recur across two converted model sizes and the non-spiking tuned-lens probe. Depth values are sampled-exit indices; their magnitudes are not directly comparable hardware costs. WT: WikiText; B: Belebele; S: Sangraha.

Figure 8: Learned-head gains over confidence stopping at development budget four. Bars show archived 95% paired block-bootstrap intervals, conditional on the fitted head and threshold. Gains reproduce across SNN sizes but not uniformly on depth. WT: WikiText; B-en/B-bn: Belebele English/Bangla; S-bn: Sangraha Bangla.

Figure 9: Why the fixed-exit reference must be the envelope. Reporting-split accuracy at each fixed exit (dots), the upper concave envelope reachable by mixing two fixed exits (line), and the learned head at its realized cost (star), Belebele-English at 0.6B. (a) The tuned-lens depth curve is strongly convex over the first eight exits, so the envelope at the head’s cost lies a full point above the adjacent-exit interpolation and the head falls below both. (b) The timestep curve is concave, so the two references coincide.

Language	$p=0$	$.1$	$.2$	$.3$	$.5$	$.7$	$.85$
English	$-1.44$	$-0.45$	$+0.44$	$+1.60$	$+3.35$	$+3.42$	$+3.37$
German	$-4.00$	$-3.93$	$-3.66$	$-2.97$	$-0.04$	$+2.58$	$+2.96$
Turkish	$-5.03$	$-4.73$	$-4.37$	$-3.56$	$-1.42$	$+1.38$	$+2.56$
Bangla	$+3.15$	$+3.03$	$+2.97$	$+2.77$	$+2.20$	$+1.80$	$+1.99$

Appendix

Table 3: Mean allocation gap across segmentation draws at each dropout level. Only nonzero rungs have three draws. The near-zero German

.5

rung is the only one with draw-level sign disagreement.

	Language	$p=0$	$.1$	$.2$	$.3$	$.5$	$.7$	$.85$	$r$
$G$	English	$-1.49$	$-0.66$	$+0.30$	$+0.89$	$+2.45$	$+3.46$	$+3.44$	$+0.96$
	German	$-4.38$	$-4.39$	$-4.21$	$-3.81$	$-1.13$	$+2.46$	$+3.11$	$+0.97$
	Turkish	$-5.35$	$-5.17$	$-5.08$	$-4.32$	$-2.11$	$+1.40$	$+2.69$	$+0.98$
	Bangla	$+3.26$	$+3.10$	$+2.97$	$+2.81$	$+2.42$	$+2.48$	$+2.79$	$-0.71$
$G_{\text{dem}}$	English	$-1.94$	$-1.09$	$-0.28$	$+0.23$	$+1.38$	$+2.63$	$+4.28$	$+0.96$
	German	$-1.66$	$-1.32$	$-0.68$	$-0.03$	$+1.15$	$+2.82$	$+4.87$	$+0.97$

Appendix

Table 4: Passage-aligned intervention: allocation gap

G

and demand gap

G_{\text{dem}}

at each dropout level, with the Pearson correlation

r

between them across the seven rungs. The 60 scored Belebele passages are the same link identifiers in all four languages, and the evaluated character total is identical at every rung of a language. Demand rises with fragmentation everywhere; allocation follows it except in Bangla.

	Language	$p=0$	$.1$	$.2$	$.3$	$.5$	$.7$	$.85$	$r$
$G$ canonical lens	English	$-0.88$	$-0.47$	$-0.18$	$+0.05$	$+0.45$	$+0.60$	$+0.51$	$-0.86$
	German	$-3.97$	$-4.01$	$-3.99$	$-3.78$	$-3.13$	$-1.88$	$-0.53$	$-0.96$
	Turkish	$-5.45$	$-5.48$	$-5.48$	$-5.42$	$-5.01$	$-3.82$	$-2.20$	$-0.97$
	Bangla	$+3.13$	$+2.95$	$+2.72$	$+2.54$	$+2.27$	$+2.17$	$+2.22$	$+0.98$
$G$ pooled lens	English	$-0.54$	$+0.92$	$+1.93$	$+2.73$	$+3.51$	$+3.60$	$+3.48$	$-0.55$
	German	$-3.45$	$-3.51$	$-3.38$	$-2.90$	$-0.14$	$+3.02$	$+3.32$	$-0.90$

Appendix

Table 5: Depth-axis ladder on the same tokens as Table 4 , through two readouts, each fitted once and frozen across all rungs: a lens fitted on canonical text only, and one fitted on the pooled support of the ladder.

G

is the allocation gap at budget four of 16 exits and

G_{\text{dem}}

the first-hit demand gap;

r

is their correlation across the seven rungs. Under the pooled lens the allocation curves match the timestep ladder; under both, the demand gap moves the other way.

Figure 10: Full-compute surprisal contrast predicts allocation across the 15 whitespace-delimited languages. Both axes are derived from the same model, so the relationship describes model behavior rather than establishing an independent linguistic cause.

Figure 11: Descriptive interpolation over the overlapping ranges of English, German, and Turkish. The spread decreases by 5.54-fold. Bangla is excluded by range support, and interpolation adds no new observations.

Corpus	$\varepsilon$	mean $d$	$G_{\text{dem}}$	$G_{\text{svc}}$
WikiText	$.02$	7.17	$-1.91$	$+0.49$
	$.05$	6.25	$-1.89$	$+0.47$
	$.10$	5.31	$-1.67$	$+0.25$
	$.20$	4.27	$-1.29$	$-0.13$
Belebele-en	$.02$	6.97	$-1.32$	$-0.44$
	$.05$	6.07	$-1.43$	$-0.33$

Appendix

Table 6: Demand and service gaps for the 0.6B timestep model as the tolerance

\varepsilon

of Equation 3 varies. The allocation gap

G

does not depend on

\varepsilon

. Mean demand moves by roughly three exits; the demand gap moves far less and never changes sign. The service gap changes sign only on WikiText.

		first hit		stable
Setting	Corpus	$G_{\text{dem}}$	$G_{\text{svc}}$	$G_{\text{dem}}$	$G_{\text{svc}}$
Time / 0.6B	WT	$-1.89$	$+0.47^{*}$	$-2.33$	$+0.91^{*}$
	B-en	$-1.43$	$-0.33^{*}$	$-1.71$	$-0.05$
	B-bn	$+2.51$	$+0.68^{*}$	$+3.87$	$-0.68^{*}$
	S-bn	$+2.52$	$+0.75^{*}$	$+3.94$	$-0.67^{*}$
Time / 1.7B	WT	$-1.56$	$-0.05$	$-2.59$	$+0.98^{*}$

Appendix

Table 7: The decomposition under both demand targets,

\varepsilon=0.05

, budget four. Asterisks mark cluster-bootstrap intervals excluding zero. The demand gap matches the allocation sign in all twelve settings under the stable target, including the depth WikiText case that is the exception under first hit. Four service gaps change sign between targets.

	B $-$ A by seed
	0	1	2	mean (sd)
Bangla $\Delta G$	$-6.00$	$-6.27$	$-6.06$	$-6.11$ ( $0.14$ )
English $\Delta G$	$+0.27$	$+0.15$	$-0.48$	$-0.02$ ( $0.40$ )
Bangla $\Delta G$ , eq. tok.	$-6.11$	–	–	–
Bangla $\Delta G_{\text{dem}}^{\text{s}}$	$-0.39$	$-0.49$	$-0.57$	$-0.48$ ( $0.09$ )
Bangla $\Delta G_{\text{svc}}^{\text{s}}$	$-5.62$	$-5.78$	$-5.49$	$-5.63$ ( $0.14$ )

Appendix

Table 8: From-scratch 49.1M models under two tokenizers differing only in Bangla exposure, budget 6.5 of 13 exits, 300 parallel Belebele passages. Seeds pair document order and initialization across arms; superscript s marks the stable demand target. Three seeds are descriptive; no interval over seeds is reported. Within-seed passage-cluster intervals on

G

have half-widths of

0.06

–

0.15

Figure 12: Allocation gap of each from-scratch model, with the two arms of a seed joined by a line. The three Bangla lines fall by six exits in parallel; the three English lines are flat and cross.

Adaptive Computation

Adaptive computation allocates model depth, tokens, attention, or sensing effort according to input difficulty and resource constraints.

See all 109 papers

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
Mar 25, 2026Ben GublerTokenization in Language Models Tokenization
More Than Words: Compositional Tokenization for Efficient Language Models
Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model Pretraining Efficient Language Model Training
The Hebrew University of Jerusalem
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword Tokenization Byte-Pair Encoding
ETH Zürich · EPFL

Mar 25, 2026J/K move · Enter open · S save

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Ben Gubler

Tokenization in Language Models Tokenization Subword Tokenization

Open paper AskPDF arXiv

Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.

Abstract

Adaptive Computation

Explore similar work

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

More Than Words: Compositional Tokenization for Efficient Language Models

Objective vs. Search: Decomposing What Makes a Good Tokeniser