From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution

Pavel Chizhov; Egor Bogomolov; Ivan P. Yamshchikov

cs.CLApr 15, 2026

From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution

Authors: Pavel Chizhov, Egor Bogomolov, Ivan P. Yamshchikov

Organizations: CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · JetBrains Research · TU Delft

Abstract

Efficiency and safety of Large Language Models (LLMs), among other factors, rely on the quality of tokenization. A good tokenizer not only improves inference speed and language understanding but also provides extra defense against jailbreak attacks and lowers the risk of hallucinations. In this work, we investigate the efficiency of code tokenization, in particular from the perspective of data source diversity. We demonstrate that code tokenizers are prone to producing unused, and thus under-trained, tokens due to the imbalance in repository and language diversity in the training data, as well as the dominance of source-specific, repetitive tokens that are often unusable in future inference. By modifying the BPE objective and introducing merge skipping, we implement different techniques under the name Source-Attributed BPE (SA-BPE) to regularize BPE training and minimize overfitting, thereby substantially reducing the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. This provides an effective tool suitable for production use.

Figures & tables

Panel of Figure 1 — Figure 1: Under-trained token example from StarCoder2, token number 48263. We show (a) the last steps of the merge tree and (b) the under-trained token indicators for StarCoder2 3B computed with the magikarp library. The tokens closer to $(0,0)$ are similar to known under-trained tokens and thus are more likely to be under-trained. Such under-trained tokens that are used to build token number 48263 are highlighted in red in both figures.

	Qwen2.5 Coder 3B	CodeGemma 7B	StarCoder2 3B
Category	2% of 151k tokens	2% of 262k tokens	2% of 49k tokens
Special tokens	00 18 0 (0.6%)	0 101 0 (2.0%)	00 0 0 (0.0%)
Digits and numbers	000 2 0 (0.1%)	00 77 0 (1.5%)	00 0 0 (0.0%)
Punctuation	0 124 0 (4.1%)	0 521 (10.2%)	125 (12.2%)
Full or partial variable or function names	000 4 0 (0.1%)	00 64 0 (1.2%)	451 (44.0%)
ALL–CAPS Latin words	000 0 0 (0.0%)	00 60 0 (1.2%)	0 88 0 (8.6%)

Table 1: Under-trained token classification for coding language models. For each under-trained token category, we present the number of tokens and their percentage out of the total studied token set (2% of each vocabulary).

Panel of Figure 2 — Figure 2: BPE merge visualization. We show (a) BPE merge sequences with counts of repositories and languages, and (b) correspondence between the two counts, highlighting the distances from $(0,0)$ in the under-trained token indicator space. We also show correlations between these distances and (c) repository and (d) language counts.

Tokenizer	CR	Used	MTL	Steps
BPE	3.90	19585	6.75	32508
Skip $\mathcal{R}<2$	3.90	19862	6.57	36263
Skip $\mathcal{R}<5$	3.91	20117	6.51	40096
Skip $\mathcal{R}<10$	3.91	20343	6.50	43575
Skip $\mathcal{R}<20$	3.92	20741	6.50	51093
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	3.91	20491	6.54	32508

Table 2: Evaluation of Java (top) and Python (bottom) SA-BPE tokenizers compared to BPE: compression rate (CR) and number of used tokens in the evaluation set, mean token length (MTL), and number of training steps.

Panel of Figure 3 — Figure 3: CamelCase and snake_case token lengths (in name parts) in basic BPE and our modifications.

Priority Criterion	Merge Skip	Compression	Coverage	MTL	# 3-digit numbers	# Under-trained
$\mathcal{F}$ (BPE)	— (BPE)	3.623	29240	6.23	517	640
$\mathcal{F}$	Skip $\mathcal{L}<4$	3.623	31416	5.89	655	52
$\mathcal{F}\cdot\mathcal{L}$	—	3.637	31267	5.85	772	151
$\mathcal{F}\cdot\mathcal{L}$	Skip $\mathcal{L}=1$	3.635	31357	5.84	779	75
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	—	3.641	30876	6.02	595	202
$\mathcal{F}\cdot\log\mathcal{R}$	—	3.643	31066	5.99	607	72

Table 3: Evaluation of multilingual SA-BPE tokenizers compared to BPE: compression rate and number of total used tokens in the evaluation set, mean token length (MTL), three-digit numbers count, and number of verified under-trained tokens out of 646 tested. The first row represents the basic BPE algorithm.

Panel of Figure 4 — Figure 4: Compression rate for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values, here distance from $(0,0)$ in indicator space.

Panel of Figure 5 — Figure 5: Under-trained token analysis in comparison for the model trained with basic BPE and the model trained with priority criterion $\mathcal{F}\cdot\log\mathcal{R}\cdot\mathcal{L}$ . Each plot is accompanied by 1D histograms for better visual comparison.

Figure 6: Kernel density estimation of the token probabilities calculated on the model training data.

Tokenizer	CR	Used	MTL	Time
BPE	3.62	29240	6.23	0:12
PickyBPE	3.61	28673	6.22	27:04 ∗
BoundlessBPE	3.56	29375	6.70	28:55 ∗
SA-BPE (ours)	3.64	31567	5.86	0:12

Table 4: Evaluation of multilingual tokenizers: compression rate (CR) and number of used tokens in the evaluation set, mean token length (MTL), and total inference time. “SA-BPE” denotes our best model trained with priority criterion

\mathcal{F}\cdot\log\mathcal{R}\cdot\mathcal{L}

. ∗ Inference time is implementation-dependent.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Model Parameter	Value
Hidden size	768
Intermediate size	2048
Number of hidden layers	12
Number of attention heads	12
Number of key-value heads	4
Max position embeddings	2048

Appendix

Table 5: Main model configuration parameters

Training Parameter	Value
Optimizer	adamW
Weight decay	0.01
Gradient clipping	1.0
Adam beta1	0.9
Adam beta2	0.95
Adam epsilon	$1.0\times 10^{-8}$

Appendix

Table 6: Main training configuration parameters

Language	# Documents	Size (in GB)
Java	2,741,833	11.15
JavaScript	2,757,358	8.40
Python	1,770,216	7.90
PHP	2,006,486	7.02
C++	990,971	6.67
C#	1,468,542	5.70

Appendix

Table 7: Per-language dataset statistics

Language	Extensions
Java	java
C#	cs
C++	cpp , hpp
Python	py
Haskell	hs
Dart	dart

Appendix

Table 8: File extensions chosen for each language for the evaluation set.

Figure 7: Models with thresholds for the minimum number of languages evaluated for compression rate and coverage. Minimum

\mathcal{L}=1

is regular BPE. Tokenizers with

\mathcal{L}>7

resulted in exhausting the merge queue.

Min $\mathcal{L}$	Gini	Compression
1 (BPE)	0.0576	3.5693
2	0.0576	3.5743
3	0.0570	3.5719
4	0.0568	3.5710
5	0.0567	3.5651
6	0.0566	3.5588

Appendix

Table 9: Gini index and compression rate for tokenizers regularized with language-based merge skip criteria.

Figure 8: Token frequencies distribution in the evaluation corpus for basic BPE, Unigram, and Wordpiece tokenizers. Total coverage for BPE is 89.23%, for Unigram 75.79%, and for Wordpiece 74.92%.

Priority Criterion	Merge Skip	Java	JavaScript	Python	PHP	C++	C#	Go	Rust	Ruby
$\mathcal{F}$ (BPE)	— (BPE)	3.875	3.481	3.634	3.842	3.225	4.265	3.030	3.499	3.683
$\mathcal{F}$	Skip $\mathcal{L}<4$	3.867	3.499	3.629	3.822	3.224	4.244	3.026	3.492	3.686
$\mathcal{F}\cdot\mathcal{L}$	—	3.887	3.507	3.644	3.848	3.238	4.267	3.037	3.507	3.692
$\mathcal{F}\cdot\mathcal{L}$	Skip $\mathcal{L}=1$	3.880	3.508	3.643	3.837	3.235	4.263	3.030	3.506	3.693
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	—	3.901	3.505	3.650	3.859	3.239	4.284	3.040	3.513	3.702
$\mathcal{F}\cdot\log\mathcal{R}$	—	3.903	3.507	3.652	3.860	3.240	4.286	3.041	3.514	3.703

Appendix

Table 10: Text compression by language for the nine higher-resource languages.

Priority Criterion	Merge Skip	Kotlin	Scala	Swift	Vue	Dart	Lua	Haskell	Julia	OCaml
$\mathcal{F}$ (BPE)	— (BPE)	3.968	3.562	4.140	3.165	3.894	3.264	3.340	3.041	3.305
$\mathcal{F}$	Skip $\mathcal{L}<4$	3.941	3.559	4.162	3.178	3.911	3.294	3.340	3.049	3.319
$\mathcal{F}\cdot\mathcal{L}$	—	3.980	3.567	4.167	3.186	3.912	3.296	3.350	3.055	3.334
$\mathcal{F}\cdot\mathcal{L}$	Skip $\mathcal{L}=1$	3.980	3.564	4.166	3.187	3.912	3.296	3.342	3.056	3.335
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	—	3.985	3.574	4.162	3.181	3.920	3.284	3.356	3.052	3.318
$\mathcal{F}\cdot\log\mathcal{R}$	—	3.987	3.576	4.166	3.183	3.921	3.285	3.357	3.055	3.316

Appendix

Table 11: Text compression by language for the nine lower-resource languages.

Priority Criterion	Merge Skip	Java	JavaScript	Python	PHP	C++	C#	Go	Rust	Ruby
$\mathcal{F}$ (BPE)	— (BPE)	17935	13643	15693	15541	15755	15370	16019	15701	12618
$\mathcal{F}$	Skip $\mathcal{L}<4$	18763	14211	16315	16106	16437	15993	16767	16376	13041
$\mathcal{F}\cdot\mathcal{L}$	—	18816	14359	16541	16266	16549	16163	16993	16594	13251
$\mathcal{F}\cdot\mathcal{L}$	Skip $\mathcal{L}=1$	18850	14377	16561	16278	16566	16188	17007	16625	13267
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	—	18688	14130	16247	16113	16279	15890	16574	16180	12965
$\mathcal{F}\cdot\log\mathcal{R}$	—	18756	14184	16289	16160	16314	15926	16627	16231	13000

Appendix

Table 12: Language coverage for the nine higher-resource languages.

Priority Criterion	Merge Skip	Kotlin	Scala	Swift	Vue	Dart	Lua	Haskell	Julia	OCaml
$\mathcal{F}$ (BPE)	— (BPE)	16898	7375	14528	11475	15626	13189	5565	6214	5325
$\mathcal{F}$	Skip $\mathcal{L}<4$	17693	7475	15191	12006	16443	13726	5653	6296	5405
$\mathcal{F}\cdot\mathcal{L}$	—	17834	7484	15353	12262	16649	13875	5650	6354	5431
$\mathcal{F}\cdot\mathcal{L}$	Skip $\mathcal{L}=1$	17871	7483	15380	12295	16681	13908	5654	6360	5434
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	—	17561	7460	15005	11924	16229	13568	5610	6293	5381
$\mathcal{F}\cdot\log\mathcal{R}$	—	17627	7473	15053	11975	16291	13608	5617	6298	5387

Appendix

Table 13: Language coverage for the nine lower-resource languages.

	Priority Criterion	Merge Skip	1	2	3	4	5	6	7	8	9	10
1	$\mathcal{F}$ (BPE)	— (BPE)	0	8304	9168	9386	5848	6362	9580	10004	10088	10228
2	$\mathcal{F}$	$\mathcal{L}<4$	8304	0	5572	5392	6132	5878	3292	6434	6472	6392
3	$\mathcal{F}\cdot\mathcal{L}$	—	9168	5572	0	372	6366	6316	5006	2666	2826	2894
4	$\mathcal{F}\cdot\mathcal{L}$	$\mathcal{L}=1$	9386	5392	372	0	6562	6346	4842	2614	2764	2678
5	$\mathcal{F}\cdot\log(\mathcal{R}+1)$	—	5848	6132	6366	6562	0	576	4834	5892	5956	6086
6	$\mathcal{F}\cdot\log\mathcal{R}$	—	6362	5878	6316	6346	576	0	4454	5590	5618	5728

Appendix

Table 14: Counts of different tokens in the trained tokenizers. The columns with numbers represent the models in the same order, enumerated by identifiers in the first column.

Priority Criterion	Merge Skip	Under-trained token examples
$\mathcal{F}$ (BPE)	— (BPE)	0123351271903 , _AtaSmartAttributeDisplayType , minCuSize , DISKinematics
$\mathcal{F}$	Skip $\mathcal{L}<4$	IMARY , ientation , vanced , pendencies , ventory
$\mathcal{F}\cdot\mathcal{L}$	—	OfSurgeServices , AceHighFlush , PeriodoDeclara , OXWSMTGS
$\mathcal{F}\cdot\mathcal{L}$	Skip $\mathcal{L}=1$	arency , ICENSE , ployment , ereum , avascript
$\mathcal{F}\cdot\log\left(\mathcal{R}+1\right)$	—	gridBagConstraintsPeriodoDeclara , HighFlush , ablytyped
$\mathcal{F}\cdot\log\mathcal{R}$	—	IntoConstraints , scalably , ilibj , VisualStyleBackColor , arency

Appendix

Table 15: Examples of verified under-trained tokens in multilingual models.

Panel of Figure 9 — Figure 9: Compression rate for Java for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 10 — Figure 10: Compression rate for JavaScript for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 11 — Figure 11: Compression rate for Python for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 12 — Figure 12: Compression rate for PHP for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 13 — Figure 13: Compression rate for C++ for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 14 — Figure 14: Compression rate for C# for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 15 — Figure 15: Compression rate for Go for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 16 — Figure 16: Compression rate for Rust for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 17 — Figure 17: Compression rate for Ruby for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 18 — Figure 18: Compression rate for Kotlin for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 19 — Figure 19: Compression rate for Scala for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 20 — Figure 20: Compression rate for Swift for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 21 — Figure 21: Compression rate for Vue for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 22 — Figure 22: Compression rate for Dart for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 23 — Figure 23: Compression rate for Lua for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 24 — Figure 24: Compression rate for Haskell for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 25 — Figure 25: Compression rate for Julia for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Panel of Figure 26 — Figure 26: Compression rate for OCaml for tokenizers with applied pruning (a) in the reverse order of token ids and (b) starting from the tokens with the lowest under-trained indicator values (distance from (0, 0) in indicator space).

Tokenization

See all 44 papers

Mar 25, 2026

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.

Ben Gubler

Tokenization in Language Models Tokenization Subword Tokenization

Ask PDF

Jun 18, 2025

Less Is More: Reducing Token Counts Without Compromising Performance

Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost. Although longer, multi-word tokens can reduce fertility, naively adding them often degrades language model performance. We propose Thunder-Tok, a subword tokenizer that reduces fertility while preserving downstream performance. Thunder-Tok first constructs a large seed vocabulary from corpus substrings and filters structurally incomplete candidates, including invalid Unicode byte fragments and word-boundary violations. It then prunes the seed vocabulary using a likelihood-based token score derived from a uniform Jensen lower bound of the training-data probability. Experiments show that Thunder-Tok reduces fertility by approximately 25% in English and 9% in Korean compared with the standard BPE tokenizer while maintaining competitive performance.

Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1

Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University

LLM Inference Acceleration Token Efficiency Subword Tokenization Tokenization in Language Models Tokenization

Ask PDF

Oct 4, 2026

More Than Words: Compositional Tokenization for Efficient Language Models

Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short English phrase such as "On the table." is usually produced as four separate predictions for the preposition (On), article (the), noun (table), and punctuation (.), where each consumes a sequence position and adds inference cost. We introduce CoBPE, a compositional tokenization approach that represents such phrases as a lexical base token (table) attached with a small set of reusable surface modifiers, composed in embedding space at input and predicted jointly at output. In controlled pretraining from scratch at 780M and 1.3B scales, CoBPE shortens sequences by 30% and improves average downstream performance by 1.2 points relative to standard BPE under matched training compute. Our results suggest that part of what is now expressed through token sequences can instead be modeled through structured representations, opening a broad design space for more token-efficient and capable language models.

Yuval Reif, Guy Kaplan, Roy Schwartz

The Hebrew University of Jerusalem

Language Model Pretraining Efficient Language Model Training Efficient Language Model Inference Subword Tokenization Tokenization in Language Models Tokenization

Ask PDF

Abstract

Tokenization

Explore similar work

Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Less Is More: Reducing Token Counts Without Compromising Performance

More Than Words: Compositional Tokenization for Efficient Language Models