No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance

Antonio Lopardo; Luis Frentzen Salim; Shikhar Srivastava; Kevin Jiang; Mridul Sharma; Catherine Arnett

cs.CLOct 15, 2026

No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance

Authors: Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava, Kevin Jiang, Mridul Sharma, Catherine Arnett

Organizations: EleutherAI · University of Basel · National Taiwan University of Science and Technology · Academia Sinica ·

Abstract

Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run. In this paper, we analyze approximately 200 language models trained on nine different languages. We evaluate both models and their tokenizers and test whether there is a relationship between any proposed intrinsic tokenizer evaluations and model performance, as measured by a wide variety of benchmarks. Despite the quantity and variety of proposed intrinsic tokenizer metrics, we find that none consistently predicts downstream performance across model families, tasks, and languages. This motivates the need for novel tokenizer evaluation metrics and approaches. We propose novel linguistically grounded metric introduced in this work shows that new metrics can capture aspects of tokenizer quality that existing metrics miss, though their predictive power is likely to vary across model families, languages, and tasks.

Figures & tables

Figure 1: Each row represents the distribution of correlation coefficients (x-axis) between a tokenizer metric (y-axis) and individual tasks. The color of the box plot represents whether the overall correlation between the tokenizer metric and aggregate performance is significant.

Panel of Figure 2 — Figure 2: Correlations between intrinsic metric and English model performance for two model suites, including aggregated performance (Avg.). The color of each cell corresponds to direction of correlation and the shade corresponds to the strength. Results for other languages are reported in Appendix E .

Figure 3: Correlation between each tokenizer metric and the Italian downstream evaluations, individually and in aggregate (Avg.) for the TokSuite models.

Language	$\text{PMI}_{\min}$ (std)	BE (std)	$\bar{u}$
English	$-$ 1.06 (0.22)	0.77 (0.30)	2.44
Italian	$-$ 1.42 (0.20)	0.81 (0.26)	2.42
Turkish	$-$ 1.70 (0.21)	0.84 (0.30)	2.35
Farsi	3.09 (0.34)	0.76 (0.41)	6.31
Chinese	1.93 (0.09)	1.11 (0.50)	6.51

Table 1: U-statistic components by language for the TokSuite models (mean and std across the 13 TokSuite tokenizers, excluding ByT5, whose byte-level vocabulary has no character statistics).

\text{PMI}_{\min}

shows minimal cross-tokenizer variation, especially for Chinese (std 0.09), confirming it reflects script properties rather than tokenizer quality.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Metric		Description
Information Theoretic	renyi_efficiency_a_*	$\uparrow$	Rényi entropy ( $\alpha\in\{1,2,2.5\}$ )
	unigram_entropy	$\uparrow$	Shannon entropy of token distribution
	compression_ratio	$\uparrow$	Normalization units per token
	bigrams_entropy	$\uparrow$	Entropy over token bigrams
	fertility	$\downarrow$	Tokens per normalization unit
	encoding_length †	$\downarrow$	Total tokens to encode corpus

Appendix

Table 2: Summary of intrinsic tokenizer metrics computed in this study. Metrics marked with

\dagger

were implemented in our fork of tokenizer-intrinsic-evals . Arrows indicate the direction proposed in the literature to correlate with better downstream performance; blank cells indicate metrics without a clear proposed direction.

Panel of Figure 4 — Figure 4: Token frequency vs. frequency rank in log-log scale. Dashed lines (slope $-1$ ) reference Zipfian distributions. Figure from Lotz et al. (2025) .

Vocab Constr	Init Voc	Pre-tok	Segment
PathPieceL	BPE	FirstSpace	PathPieceL
	Unigram	FirstSpace
	$n$ -gram	FirstSpDigit
	$n$ -gram	FirstSpace
Unigram		FirstSpace	Likelihood
			Greedy

Appendix

Table 3: Summary of 350M parameter model vocabulary construction methods, showing initialization, pre-tokenization, and segmentation strategies. All models have been trained with different vocab sizes, specifically 32,768, 40,960 and 49,152 tokens.

	English	Thai	Urdu	Amharic †	Vietnamese
Byte Premium	1.00	2.74	1.71	1.72	1.35
English reference (7B tokens)	31.11 GiB
BP-scaled target	31.11 GiB	85.25 GiB	53.20 GiB	53.52 GiB	42.00 GiB
FineWeb2 Raw Available	–	321.88 GiB	22.48 GiB	2.69 GiB	402.69 GiB
Tokenized Data Size	31.11 GiB	$>$ 85 GiB	12.54 GiB	–	38.27 GiB
Coverage	100%	$>$ 100%	$\sim$ 24%	$<$ 5%	$\sim$ 91%

Appendix

Table 4: FineWeb-2 data budget for VISTA 350M models (for the BPE, vocab = 32,768 case). The BP-scaled target is the Chinchilla-optimal tokenized data size scaled by each language’s byte premium. Tokenized Data Size is the actual tokenized data after processing downloaded FineWeb-2 text using the pre-trained language specific tokenizers.

	VISTA-130M	VISTA-350M
Languages	English, Thai, Urdu, Amharic, Vietnamese	English, Thai, Urdu, Vietnamese
Tokenizers	BPE, Unigram	BPE, Unigram
Vocabulary sizes	8192, 16384, 32768, 49152, 65536, 81920, 98304, 114688	8192, 98304
Total runs	$5\times 2\times 8=\mathbf{80}$	$4\times 2\times 2=\mathbf{16}$

Appendix

Table 5: VISTA Pre-training Variations.

Language (tag)	BP-scaled GB	Raw GB
English (eng_latn)	1	1.00
Thai (tha_thai)	1	2.74
Urdu (urd_arab)	1	1.71
Amharic (amh_ethi)	1	1.72
Vietnamese (vie_latn)	1	1.35

Appendix

Table 6: Monolingual corpora and byte-premium (BP) statistics used in our experiments. The left column shows the BP-scaled data size (in GB), and the right column shows the raw UTF-8 byte size. BP multipliers are derived from Arnett et al. (2024a) ; BP scaling ensures languages with less information-dense scripts are allocated proportionately more raw text data.

Model Architecture
Configuration	VISTA-130M	VISTA-350M
Model Type	LLaMA 1
Hidden Size ( $d_{\text{model}}$ )	768	1024
Intermediate Size ( $d_{\text{ff}}$ )	2048	2736
Number of Hidden Layers	12	24
Number of Attention Heads	12	16

Appendix

Table 7: Model architecture and training hyperparameters for VISTA-130M and VISTA-350M. Where the two scales differ, both values are shown; shared values appear once spanning both columns.

Task Type	Benchmark	Language(s)
Linguistic	CoLA ( Warstadt et al., 2019 )	English
	BLiMP ( Warstadt et al., 2020 )	English
	ZhoBLiMP ( Liu et al., 2024 )	Chinese
	TurBLiMP ( Başar et al., 2025 )	Turkish
	MultiBLiMP ( Jumelet et al., 2026 )	English, Italian, Farsi, Turkish
NLU	SuperGLUE ( Wang et al., 2019 )	English

Appendix

Table 8: List of language model evaluations used for TokSuite models. Tasks marked with

\dagger

were implemented in our fork of the LM evaluation harness.

Task Type	Benchmark
Linguistic	SuperGLUE ( Wang et al., 2019 )
	Winogrande ( Sakaguchi et al., 2020 )
	BLiMP ( Warstadt et al., 2020 )
	MultiBLiMP ( Jumelet et al., 2026 )
World Knowledge/ Semantics	COMPS $\dagger$ ( Misra et al., 2023 )
	ANALOGICAL $\dagger$ ( Wijesiriwardene et al., 2023 )

Appendix

Table 9: List of English language model evaluations used for “Tokenization is More Than Compression” models. Tasks marked with

\dagger

were implemented in our fork of the LM evaluation harness.

Task Type	Benchmark	Language(s)
Perplexity	FLORES $\dagger$ NLLB Team et al. (2022)	Amharic, English, Thai, Urdu, Vietnamese
Linguistic	BLiMP Warstadt et al. (2020)	English
Linguistic	MultiBLiMP Jumelet et al. (2026)	Amharic, English, Urdu
NLU	SuperGLUE Wang et al. (2019)	English
	Winograd_th $\dagger$	Thai
	Winogrande Sakaguchi et al. (2020)	English

Appendix

Table 10: List of language model evaluations used for VISTA. Tasks marked with

\dagger

were implemented in our fork of LM evaluation harness.

Figure 5: Downstream performance for TokSuite models aggregated by language and macro average across language. Each cell value shows mean performance. The color of each cell corresponds to the z-scored performance, where the darker the green the better the performance.

Tokenizer	Aya	Qwen	BLOOM	Comma	XGLM	BERT	ByT5	Gemma	GPT-2	Llama	Phi	Tekken	GPT-4o	TokenMonster	Chance
GLUE	0.448	0.446	0.420	0.430	0.397	0.402	0.394	0.459	0.418	0.394	0.414	0.451	0.429	0.418	0.45
Winogrande	0.584	0.579	0.586	0.560	0.577	0.525	0.553	0.568	0.542	0.593	0.586	0.576	0.582	0.539	0.50
BLiMP	0.835	0.836	0.833	0.843	0.835	0.833	0.838	0.838	0.827	0.844	0.836	0.837	0.844	0.787	0.50
MultiBLiMP	0.990	0.992	0.992	0.984	0.988	0.994	0.990	0.988	0.990	0.988	0.994	0.992	0.990	0.982	0.50
EWoK	0.629	0.623	0.622	0.608	0.615	0.563	0.619	0.627	0.615	0.632	0.614	0.614	0.623	0.600	0.50
PIQA	0.729	0.724	0.726	0.722	0.726	0.669	0.696	0.739	0.716	0.724	0.711	0.712	0.721	0.702	0.50

Appendix

Table 11: Evaluation scores for TokSuite models on English tasks.

Tokenizer	Aya	Qwen	BLOOM	Comma	XGLM	BERT	ByT5	Gemma	GPT-2	Llama	Phi	Tekken	GPT-4o	TokenMonster	Chance
ZhoBLiMP	0.872	0.872	0.876	0.879	0.871	0.871	0.860	0.876	0.879	0.879	0.882	0.874	0.871	0.891	0.50
XWinograd	0.724	0.708	0.690	0.738	0.712	0.687	0.716	0.732	0.730	0.722	0.732	0.690	0.744	0.702	0.50
XCOMPS	0.667	0.675	0.703	0.670	0.668	0.702	0.686	0.635	0.686	0.665	0.690	0.676	0.667	0.690	0.50
XCOPA	0.642	0.634	0.630	0.620	0.624	0.610	0.596	0.644	0.616	0.648	0.634	0.644	0.642	0.624	0.50
XStoryCloze	0.592	0.592	0.595	0.578	0.600	0.567	0.561	0.592	0.567	0.587	0.572	0.590	0.584	0.568	0.50
XQuAD	0.057	0.007	0.034	0.006	0.000	0.000	0.051	0.013	0.035	0.020	0.011	0.015	0.000	0.003	0.00

Appendix

Table 12: Evaluation scores for TokSuite models on Chinese tasks.

Tokenizer	Aya	Qwen	BLOOM	Comma	XGLM	BERT	ByT5	Gemma	GPT-2	Llama	Phi	Tekken	GPT-4o	TokenMonster	Chance
Winogrande	0.533	0.519	0.547	0.488	0.509	0.483	0.516	0.511	0.506	0.500	0.516	0.501	0.501	0.502	0.50
MultiBLiMP	0.959	0.950	0.958	0.964	0.962	0.961	0.960	0.954	0.952	0.943	0.945	0.947	0.949	0.959	0.50
TurBLiMP	0.831	0.840	0.827	0.840	0.836	0.843	0.831	0.835	0.841	0.835	0.833	0.835	0.829	0.805	0.50
XCOMPS	0.656	0.662	0.651	0.654	0.680	0.675	0.655	0.644	0.662	0.666	0.641	0.662	0.660	0.612	0.50
XCOPA	0.632	0.642	0.610	0.614	0.638	0.596	0.616	0.622	0.636	0.642	0.610	0.630	0.626	0.622	0.50
Global PIQA	0.510	0.550	0.530	0.510	0.480	0.500	0.520	0.530	0.530	0.470	0.520	0.520	0.530	0.510	0.50

Appendix

Table 13: Evaluation scores for TokSuite models on Turkish tasks.

Tokenizer	Aya	Qwen	BLOOM	Comma	XGLM	BERT	ByT5	Gemma	GPT-2	Llama	Phi	Tekken	GPT-4o	TokenMonster	Chance
MultiBLiMP	0.930	0.941	0.929	0.948	0.974	0.968	0.926	0.876	0.944	0.899	0.942	0.908	0.900	0.946	0.50
XCOMPS	0.659	0.665	0.628	0.643	0.677	0.674	0.660	0.653	0.663	0.656	0.655	0.655	0.655	0.673	0.50
FarsTail	0.338	0.336	0.331	0.347	0.348	0.332	0.337	0.325	0.333	0.336	0.325	0.362	0.328	0.332	0.33
MultiLoKo	0.136	0.128	0.064	0.076	0.024	0.000	0.008	0.040	0.080	0.096	0.108	0.124	0.008	0.004	0.00
INCLUDE	0.223	0.276	0.217	0.268	0.241	0.221	0.230	0.276	0.276	0.246	0.261	0.228	0.276	0.241	0.25
SynTran	0.018	0.013	0.007	0.007	0.000	0.000	0.008	0.006	0.006	0.007	0.010	0.012	0.007	0.002	0.00

Appendix

Table 14: Evaluation scores for TokSuite models on Farsi tasks.

Tokenizer	Aya	Qwen	BLOOM	Comma	XGLM	BERT	ByT5	Gemma	GPT-2	Llama	Phi	Tekken	GPT-4o	TokenMonster	Chance
ITA-Bench	0.329	0.318	0.327	0.323	0.338	0.307	0.308	0.338	0.322	0.334	0.318	0.316	0.324	0.316	0.35
ARC Challenge	0.283	0.277	0.284	0.292	0.284	0.245	0.257	0.289	0.266	0.285	0.285	0.297	0.297	0.260	0.25
ARC Easy	0.560	0.544	0.547	0.542	0.564	0.385	0.538	0.564	0.538	0.556	0.509	0.554	0.567	0.523	0.25
HellaSwag	0.395	0.390	0.393	0.378	0.396	0.375	0.373	0.399	0.382	0.393	0.381	0.394	0.394	0.357	0.25
PIQA	0.654	0.652	0.639	0.637	0.654	0.633	0.619	0.642	0.642	0.640	0.631	0.657	0.650	0.633	0.50
Winogrande	0.610	0.604	0.609	0.599	0.606	0.553	0.613	0.612	0.607	0.605	0.606	0.593	0.601	0.581	0.50

Appendix

Table 15: Evaluation scores for TokSuite models on Italian tasks.

Figure 6: Correlation between vocab size and model performance for the TokSuite models. Facets represent evaluation language.

	BPE	PPL-	PPL-	PPL-	PPR-	PPR-	PPR-	SAGE-	SAGE-	SAGE-	SAGE-	UNI-	UNI-	UNI-	WP	Chance
		ng-fs	ng-fsd	uni-fs	fsd	none	sd	bpe	ng-fs	ng-fsd	uni	gr	lk	ppl
GLUE	0.478	0.452	0.475	0.463	0.457	0.468	0.480	0.461	0.482	0.458	0.451	0.471	0.462	0.463	0.466	0.45
Winogrande	0.505	0.507	0.507	0.507	0.529	0.507	0.515	0.522	0.530	0.541	0.515	0.507	0.524	0.522	0.521	0.50
BLiMP	0.824	0.711	0.730	0.757	0.741	0.758	0.790	0.817	0.811	0.799	0.814	0.822	0.823	0.739	0.829	0.50
MultiBLiMP	0.983	0.916	0.919	0.899	0.917	0.951	0.983	0.988	0.986	0.981	0.975	0.987	0.981	0.887	0.979	0.50
EWoK	0.569	0.518	0.519	0.538	0.515	0.525	0.556	0.570	0.565	0.564	0.569	0.564	0.568	0.535	0.568	0.50

Appendix

Table 16: Evaluation scores for TIMTC models (vocab size 32,768) on English tasks.

	BPE	PPL-	PPL-	PPL-	PPR-	PPR-	PPR-	SAGE-	SAGE-	SAGE-	SAGE-	UNI-	UNI-	UNI-	WP	Chance
		ng-fs	ng-fsd	uni-fs	fsd	none	sd	bpe	ng-fs	ng-fsd	uni	gr	lk	ppl
GLUE	0.474	0.456	0.458	0.456	0.466	0.458	0.479	0.462	0.478	0.469	0.459	0.455	0.470	0.455	0.460	0.45
Winogrande	0.536	0.507	0.497	0.507	0.527	0.513	0.488	0.522	0.503	0.507	0.517	0.524	0.496	0.496	0.520	0.50
BLiMP	0.825	0.750	0.732	0.748	0.736	0.748	0.738	0.819	0.802	0.793	0.814	0.821	0.822	0.735	0.822	0.50
MultiBLiMP	0.977	0.919	0.922	0.865	0.918	0.951	0.927	0.990	0.984	0.974	0.984	0.983	0.982	0.869	0.982	0.50
EWoK	0.570	0.518	0.524	0.533	0.527	0.536	0.530	0.575	0.574	0.557	0.560	0.572	0.571	0.530	0.577	0.50

Appendix

Table 17: Evaluation scores for TIMTC models (vocab size 40,960) on English tasks.

	BPE	PPL-	PPL-	PPL-	PPR-	PPR-	PPR-	SAGE-	SAGE-	SAGE-	SAGE-	UNI-	UNI-	UNI-	WP	Chance
		ng-fs	ng-fsd	uni-fs	fsd	none	sd	bpe	ng-fs	ng-fsd	uni	gr	lk	ppl
GLUE	0.474	0.455	0.455	0.463	0.461	0.459	0.456	0.451	0.460	0.436	0.459	0.444	0.462	0.467	0.464	0.45
Winogrande	0.522	0.501	0.510	0.503	0.508	0.525	0.520	0.515	0.512	0.519	0.508	0.500	0.512	0.505	0.510	0.50
BLiMP	0.810	0.727	0.738	0.746	0.748	0.749	0.802	0.814	0.809	0.779	0.821	0.817	0.822	0.738	0.826	0.50
MultiBLiMP	0.971	0.919	0.910	0.856	0.925	0.927	0.975	0.987	0.982	0.969	0.987	0.978	0.983	0.869	0.982	0.50
EWoK	0.566	0.518	0.518	0.537	0.522	0.527	0.556	0.566	0.574	0.544	0.575	0.583	0.557	0.543	0.563	0.50

Appendix

Table 18: Evaluation scores for TIMTC models (vocab size 49,152) on English tasks.

Tokenizer Type	Vocab Size	AfriMMLU	Belebele	Global MMLU	MultiBLiMP	Global PIQA	Avg.
BPE	8192	0.222	0.253	0.236	0.946	0.520	0.402
	16384	0.232	0.254	0.249	0.955	0.470	0.402
	32768	0.230	0.268	0.251	0.893	0.500	0.399
	49152	0.240	0.253	0.253	1.000	0.450	0.408
	65536	0.228	0.260	0.260	0.973	0.480	0.410
	81920	0.248	0.257	0.248	0.973	0.470	0.407

Appendix

Table 19: Evaluation scores for Amharic VISTA models on Amharic tasks.

Tokenizer	Vocab	ARC-C	ARC-E	Belebele	BLiMP	COPA	Global	Hella	Multi	Winogrande	XWinograd	Global	Super	Avg.
Type	Size						MMLU	Swag	BLiMP			PIQA	GLUE
BPE	8192	0.191	0.277	0.273	0.586	0.580	0.232	0.266	0.818	0.488	0.523	0.520	0.463	0.419
	16384	0.182	0.269	0.252	0.600	0.580	0.230	0.259	0.783	0.479	0.514	0.450	0.434	0.405
	32768	0.188	0.280	0.258	0.613	0.520	0.229	0.255	0.832	0.511	0.505	0.550	0.439	0.416
	49152	0.181	0.270	0.248	0.654	0.550	0.229	0.255	0.810	0.503	0.508	0.480	0.435	0.412
	65536	0.177	0.275	0.251	0.640	0.540	0.230	0.255	0.838	0.505	0.497	0.470	0.447	0.412

Appendix

Table 20: Evaluation scores for English VISTA models on English tasks. ARC Challenge and ARC Easy are abbreviated to ARC-C and ARC-E.

Tokenizer	Vocab	GLUE	Wino-	BLiMP	Multi	EWoK	PIQA	Hella	PROST	MMLU	BBH	XQuAD	Bele-	COMPS	ANA-	Avg.
Type	Size		grande		BLiMP			Swag					bele		LOGICAL
BPE	8192	0.469	0.499	0.593	0.861	0.525	0.612	0.289	0.269	0.233	0.059	0.000	0.288	0.540	0.398	0.402
	32768	0.456	0.501	0.624	0.825	0.530	0.595	0.294	0.217	0.231	0.144	0.005	0.301	0.526	0.368	0.401
	65536	0.466	0.516	0.579	0.792	0.535	0.606	0.292	0.252	0.229	0.163	0.008	0.321	0.551	0.377	0.406
	98304	0.469	0.497	0.629	0.855	0.540	0.630	0.301	0.272	0.230	0.181	0.001	0.293	0.575	0.448	0.423
Unigram	8192	0.413	0.490	0.582	0.852	0.512	0.582	0.283	0.244	0.230	0.090	0.000	0.298	0.512	0.362	0.389

Appendix

Table 21: Evaluation scores for English VISTA 350M models on English tasks.

Tokenizer	Vocab	ARC	Belebele	Global	HellaSwag	INCLUDE	ViQuAD	VMLU	XCOPA	XNLI	Global	Avg.
Type	Size			MMLU							PIQA
bpe	8192	0.209	0.256	0.245	0.266	0.255	0.006	0.249	0.544	0.343	0.520	0.285
	16384	0.206	0.228	0.248	0.260	0.253	0.016	0.270	0.540	0.341	0.560	0.288
	32768	0.206	0.239	0.246	0.262	0.260	0.031	0.277	0.532	0.356	0.580	0.294
	49152	0.220	0.239	0.249	0.259	0.253	0.021	0.263	0.530	0.337	0.600	0.293
	65536	0.214	0.246	0.241	0.259	0.273	0.001	0.290	0.534	0.341	0.550	0.290

Appendix

Table 22: Evaluation scores for Vietnamese VISTA models on Vietnamese tasks.

Tokenizer	Vocab	Belebele	Winograd	XCOPA	MMLU	Global PIQA	Avg.
Type	Size
BPE	8192	0.259	0.470	0.536	0.082	0.540	0.377
	16384	0.261	0.540	0.556	0.083	0.500	0.388
	32768	0.251	0.530	0.544	0.080	0.540	0.389
	49152	0.246	0.540	0.540	0.084	0.530	0.388
	65536	0.254	0.516	0.562	0.082	0.510	0.385

Appendix

Table 23: Evaluation scores for Thai VISTA models on Thai tasks.

Tokenizer Type	Vocab Size	Belebele	MMLU ProX	Global PIQA	Avg.
BPE	8192	0.249	0.085	0.590	0.308
	16384	0.259	0.083	0.620	0.321
	32768	0.254	0.086	0.620	0.320
	49152	0.254	0.086	0.610	0.317
	65536	0.262	0.086	0.610	0.319
	81920	0.244	0.087	0.580	0.304

Appendix

Table 24: Evaluation scores for Urdu VISTA models on Urdu tasks.

Figure 7: Correlation between vocabulary size (x-axis) and aggregated (z-scored) model performance by language and macro-averaged over language (bottom right facet). Each facet also reports correlation coefficient and p-value.

	All Tasks				With Exclusions
Metric	$r$	$p$	$p_{\text{corr}}$	sig	$r$	$p$	$p_{\text{corr}}$	sig
AUC_zipf	0.03	0.17	0.26		$-0.10$	0.23	0.44
avg_chars_per_token	$-0.00$	0.83	0.98		$-0.24$	0.00	0.02
avg_token_rank	0.03	0.16	0.26		$-0.03$	0.70	0.94
avg_tokens_per_line	0.02	0.36	0.45		0.05	0.53	0.84
bigrams_entropy	0.00	0.89	0.98		$-0.10$	0.23	0.44

Appendix

Table 25: Correlation between tokenizer metrics and model performance, with and without task exclusions (for tasks where performance was near chance). Significance markers (*) indicate

p_{\text{corrected}}<0.05

Panel of Figure 8 — Figure 8: Correlations between intrinsic metric and model performance for TokSuite models for each of the five languages covered. Each column represents an individual task, with the final column representing aggregated performance. The color of each cell corresponds to direction of correlation and the shade corresponds to the strength.

Figure 9: Correlations between intrinsic metric and model performance for the TIMTC models for English. Each column represents an individual task, with the final column representing aggregated performance. The color of each cell corresponds to direction of correlation and the shade corresponds to the strength.

Panel of Figure 10 — Figure 10: Correlations between intrinsic metric and model performance for VISTA models for each of the five languages. Each column represents an individual task, with the final column representing aggregated performance. The color of each cell corresponds to direction of correlation and the shade corresponds to the strength.

Figure 11: Performance for each of the TokSuite models for the English orthographic awareness tasks. Color corresponds to the z-scored performance, where darker green indicates better performance.

Task Group	Language	$\rho$	p-value
LMentry	English	-0.179	0.5408
TokSuite	English	0.341	0.0292
StringBench	English	0.163	0.4806
TokSuite	Chinese	0.114	0.6990
EXECUTE	Chinese	0.267	0.0663
TokSuite	Farsi	0.433	0.1224

Appendix

Table 26: Correlation between orthographic tasks and downstream tasks.

	Metric 1	Metric 2	adj_r2_m1	adj_r2_m2	adj_r2_joint	r2_gain	p-value
1	leading_space_tokens_count	u_mean_log_weighted	0.158	$-$ 0.000	0.170	0.012	$<0.001$
2	final_tokens_count	used_in_merges	0.053	0.110	0.168	0.058	$<0.001$
3	leading_space_tokens_count	u_mean	0.158	0.002	0.167	0.009	$<0.001$
4	leading_space_tokens_count	u_median	0.158	0.003	0.165	0.007	$<0.001$
5	leading_space_tokens_count	u_mean_weighted	0.158	0.001	0.165	0.007	$<0.001$
6	leading_space_tokens_count	AUC_zipf	0.124	0.011	0.157	0.033	$<0.001$

Appendix

Table 27: Variance explained by combinations of two metrics, shown for the ten best combinations.

Figure 12: The boxes highlight comparisons between metrics that index similar things. Yellow highlights correlation between MorphScore and compression metrics.

	Metric 1	Metric 2	adj_r2_m1	adj_r2_m2	adj_r2_joint	r2_gain	p-value
1	leading_space_tokens_count	u_mean_log_weighted	0.074	$-$ 0.000	0.085	0.012	$<0.001$
2	leading_space_tokens_count	u_mean_weighted	0.074	0.000	0.084	0.010	$<0.001$
3	leading_space_tokens_count	u_mean	0.074	0.002	0.083	0.009	$<0.001$
4	leading_space_tokens_count	u_median	0.074	0.003	0.082	0.008	$<0.001$
5	leading_space_tokens_count	compression_ratio	0.072	0.000	0.073	0.000	$<0.001$
6	leading_space_tokens_count	encoding_length	0.072	0.000	0.072	0.000	$<0.001$

Appendix

Table 28: Variance explained by combinations of two metrics, shown for the ten best combinations. These results exclude any of the tasks where the small models performed near chance.

Tokenization in Language Models

See all 43 papers

Aug 18, 2026

TokEval: A Tokenizer Evaluation Suite

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers' training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.

Clara Meister

EPFL

LLM Evaluation Tokenization Tokenization in Language Models

Ask PDF

Dec 23, 2025

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.

Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4

University of Toronto · Vector Institute · Google DeepMind +4

Subword Tokenization Tokenization in Language Models

Ask PDF

Mar 25, 2026

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.

Ben Gubler

Tokenization in Language Models Tokenization Subword Tokenization

Ask PDF

Abstract

Tokenization in Language Models

Explore similar work

TokEval: A Tokenizer Evaluation Suite

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages