Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

Sajal Regmi; Siddhartha Pudasaini; Chetan Phakami Pun

cs.CLAug 26, 2026

Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

Authors: Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun

Organizations: Karela Technologies Inc.

Abstract

Byte-level BPE tokenizers that use the HuggingFace ByteLevel pre-tokenizer inherit GPT-2's word regex, where a word is defined as \p{L}+, one or more Unicode letters. In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign. Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition. We formalise this effect as a training-free lower bound on fertility. Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai), whereas Latin, Cyrillic, Hangul, and Han show exactly 1.00x. For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali. When the Nepali share of the training corpus is swept from 5% to 95%, the broken tokenizer barely shifts at all (1.7%) while the fixed one shifts 33.9%, which separates a structural ceiling from a data shortage without needing to inspect any code. We train three 268M models that differ only in their tokenizer; the fixed variant achieves 4.43% lower held-out Nepali bits per byte at equal compute, and it still leads when given the same bytes with 1.59x the compute. A census of 3,479 HuggingFace repositories finds the letters-only word class present in 63.3% of the most-downloaded text-generation models, accounting for 72.5% of their downloads. GPT-4o's o200k pattern already uses a mark-aware word class, making the repair itself prior art. We quantify its value, show how to recognise its absence from symptoms alone, map which scripts it reaches, measure how widely it is deployed, and release a 65,536-entry Nepali-English tokenizer with a harness that regenerates every number here from public data on a laptop.

Figures & tables

Panel of Figure 1 — Figure 1: Nepali fertility against the Nepali share of the tokenizer training corpus, 7 points per curve, each representing a tokenizer trained from scratch. The control remains on its pre-tokenization bound across the entire range. The treatment declines as Nepali content rises. The two curves differ by one character class.

Language	Script	\p{L}+	[\p{L}\p{M}]+	\p{L}+	[\p{L}\p{M}]+	$\Delta$ bound	Ratio	English
		Bound (tok/word)		Measured (tok/word)
Hindi	Devanagari	3.30	1.12	3.36	1.32	+1.8%	2.54 $\times$	+1.6%
Bengali	Bengali	4.65	1.15	4.73	1.50	+1.8%	3.16 $\times$	+2.2%
Nepali	Devanagari	4.67	1.14	4.78	1.58	+2.2%	3.03 $\times$	+2.1%
Tamil	Tamil	6.76	1.19	6.85	1.77	+1.4%	3.86 $\times$	+2.2%
Malayalam	Malayalam	7.34	1.26	7.48	2.11	+1.9%	3.54 $\times$	+2.4%

Table 1: Five matched pairs on FLORES-200 devtest, one row per target language, each 2,000 MB at 50% target language by bytes with a 65,536 vocabulary. “Bound” denotes the training-free floor of § 3 ; “

\Delta

bound” denotes how far the trained control arm settles above its own floor; “Ratio” is measured control fertility over measured treatment fertility; “English” is the change in English fertility from control to treatment. Nepali is the headline pair of § 5 ; Table 9 in Appendix D provides its bytes-per-token figures.

Tokenizer	Family	$\|V\|$	tok/word	B/tok	tok/word	B/tok	Lossless
			Nepali		English
Llama-3	English-centric	128,256	3.76	4.84	1.24	4.90	yes
DeepSeek-V3	English-centric	128,815	4.29	4.24	1.23	4.92	yes
Qwen2.5	English-centric	151,665	6.57	2.77	1.26	4.82	yes
cl100k (GPT-3.5/4)	English-centric	100,277	6.98	2.61	1.24	4.90	yes
GPT-2	English-centric	50,257	10.97	1.66	1.28	4.74	yes

Table 2: Fertility and bytes per token on FLORES-200 devtest. “Lossless” indicates whether decode(encode(

x

)) preserves every character. Bold marks the best lossless Nepali fertility.

		Letters-only, as % of…
Population	Classified	repos	distinct tok.	downloads
Text generation, top- $N$	1345	63.3%	52.9%	72.5%
of those, affected language tag	78	67.9%	56.8%	60.8%
All repositories, top- $N$	1126	45.2%	42.2%	30.3%
of those, affected language tag	149	50.3%	40.5%	18.9%

Table 3: Share of repositories using a letters-only pre-tokenizer, under three denominators: repositories, distinct tokenizer.json files (deduplicated by content hash, so a base model with many fine-tunes counts once; repositories that ship no such file have nothing to deduplicate and are omitted from that column alone), and 30-day downloads. Percentages are of the classified repositories in each row; repositories with no tokenizer at all, 797 of the overall top-2,000, comprising vision, audio, and diffusion models and weights-only redistributions, as well as gated ones, are excluded from every denominator.

		Bound, tok/word
Language	Shatter	\p{L}+	[\p{L}\p{M}]+
Thai	9.02 $\times$	–	–
Burmese	7.58 $\times$	–	–
Lao	5.95 $\times$	–	–
Malayalam	5.82 $\times$	7.34	1.26
Tamil	5.66 $\times$	6.76	1.19

Table 4: The training-free pre-tokenization bound across FLORES-200 devtest. Shatter ratio is pre-tokens under \p{L}+ over pre-tokens under [\p{L}\p{M}]+ ;

1.00\times

means the two word classes are indistinguishable, and those languages are grouped by family. Dashes mark the languages that do not separate words with spaces, where tok/word is undefined; Table 6 in Appendix A reports those in tokens per 100 characters and provides the full per-language grid. The corpus is parallel, so every row describes the same content.

Condition	Budget	Nepali	English	Train
		Held-out bits/byte
A broken \p{L}+	4.00B	0.4080	1.1706	0.7125
B fixed [\p{L}\p{M}]+	4.00B	0.3899	1.1490	0.6720
C broken \p{L}+	6.36B	0.3925	1.1342	0.6060

Table 5: Held-out bits per byte; lower is better. C receives

1.59\times

B’s compute by construction.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

			Bound, tok/word		Bound, tok/100 char
Language	Script	Shatter	$\backslash$ p{L}+	mark	$\backslash$ p{L}+	mark	Type
Thai	Thai	9.02 $\times$	–	–	37.4	4.1	abugida
Burmese	Myanmar	7.58 $\times$	–	–	63.9	8.4	abugida
Lao	Lao	5.95 $\times$	–	–	39.7	6.7	abugida
Malayalam	Malayalam	5.82 $\times$	7.34	1.26	72.9	12.5	abugida
Tamil	Tamil	5.66 $\times$	6.76	1.19	73.0	12.9	abugida

Appendix

Table 6: The training-free pre-tokenization bound across FLORES-200 devtest, in full. Languages at exactly

1.00\times

have identical bounds under both word classes and are grouped by family; their per-language values are in the harness’s table_bounds.csv .

Code point	Character	Category	Under \p{L}+
U+0928	na	Lo	kept
U+0947	-e	Mn	cut
U+092A	pa	Lo	kept
U+093E	-ā	Mc	cut
U+0932	la	Lo	kept
U+0940	-ī	Mc	cut

Appendix

Table 7: The word nepālī (“Nepali”) by code point. Three of its six characters are marks, so \p{L}+ produces six single-character pre-tokens where [\p{L}\p{M}]+ produces one.

Sample	Marks	Unvocalised	Vocalised
Arabic	33	1.00 $\times$	6.50 $\times$
Hebrew	23	1.00 $\times$	6.14 $\times$
Devanagari	22	1.00 $\times$	5.22 $\times$

Appendix

Table 8: Shatter ratio on the same sentence with its combining marks stripped (“unvocalised”) and intact (“vocalised”).

	Nepali		English
Word class	tok/word	B/tok	tok/word	B/tok
Control	4.78	3.81	1.28	4.76
Treatment	1.58	11.53	1.30	4.67
Relative change	$-$ 66.9%	$+$ 202.5%	+2.12%	-2.08%

Appendix

Table 9: The controlled comparison on FLORES-200 devtest. Corpus 2000 MB at 50% Nepali by bytes, vocabulary 65,536, identical in every other respect.

Nepali share of corpus	5%	20%	35%	50%	65%	80%	95%
Control	4.84	4.79	4.78	4.78	4.77	4.76	4.76
relative spread	1.7%
Treatment	1.97	1.70	1.63	1.58	1.54	1.51	1.47
relative spread	33.9%
Pre-tokenization bound	4.67 vs 1.14

Appendix

Table 10: The mixture sweep. Nepali fertility on FLORES-200 devtest, by Nepali share of the tokenizer training corpus. Control is \p{L}+ , treatment is [\p{L}\p{M}]+ ; the two arms are identical in every other respect. The last row is the training-free bound of § 3 for each arm.

Repos	Word class	Pattern (truncated)
664	not letter-based	no split stage that can cut inside a word e.g. sentence-transformers/all-MiniLM-L6-v2
510	letters only	(?i:’s\|’t\|’re\|’ve\|’m\|’ll\|’d)\|[^\r\n\p{L}\p{N}]?\p{L}+\|\p{N}\|?[^… e.g. Qwen/Qwen3-0.6B
325	letters only	’s\|’t\|’re\|’ve\|’m\|’ll\|’d\|?\p{L}+\|?\p{N}+\|?[^\s\p{L}\p{N}]+\|\s+… e.g. openai/clip-vit-base-patch32
246	letters only	(?i:’s\|’t\|’re\|’ve\|’m\|’ll\|’d)\|[^\r\n\p{L}\p{N}]?\p{L}+\|\p{N}{1,3}… e.g. zai-org/GLM-OCR
108	mark-aware	(?i:’s\|’t\|’re\|’ve\|’m\|’ll\|’d)\|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+\|\p{… e.g. Qwen/Qwen3.5-9B
60	mark-aware	[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\… e.g. nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4

Appendix

Table 11: The most common distinct pre-tokenization patterns in the census, ranked by number of repositories. Patterns are truncated for width; the full text of each, along with an example repository, is in table_census_pat.csv .

Tokenizer	Pre-tokenization word class	$\|V\|$	Nepali tok/word
Llama-3	letters-only	128,256	3.76
Control (\p{L}+)	letters-only	65,536	4.78
Qwen2.5	letters-only	151,665	6.57
cl100k (GPT-3.5/4)	letters-only	100,277	6.98
GPT-2	letters-only	50,257	10.97
Treatment ([\p{L}\p{M}]+)	mark-aware	65,536	1.58

Appendix

Table 12: Word class and measured Nepali fertility for each tokenizer in Table 2 , sorted by class and then by fertility.

Panel of Figure 2 — Figure 2: Each arm’s pre-tokenization bound against the fertility it reaches on FLORES-200 devtest, two points per matched pair. The dashed line is $y=x$ , the bound. Proposition 1 guarantees no point falls below it. Control points lie on the line; treatment points lie well above their own floor.

Tokenization in Language Models

See all 43 papers

Mar 26, 2026

Separate Before You Compress: The WWHO Tokenization Architecture

Current Large Language Models (LLMs) mostly use BPE (Byte Pair Encoding) based tokenizers, which are very effective for simple structured Latin scripts such as English. However, standard BPE tokenizers struggle to process complex Abugida scripts due to their structural complexity. The problem is that these tokenizers break complex conjuncts, which are multi-codepoint grapheme clusters, into meaningless sub-character units. This degrades the LLM's reasoning efficiency by forcing it to learn basic orthographic structures at inference time and raises inference costs, resulting in a significant "Token Tax" for the Global South. We propose a new three-layer architecture, the WWHO (Where-What-How Often), and an algorithm named SGPE (Syllable-aware Grapheme Pair Encoding) that separates the linguistic rules of the script from the statistical compression process while enabling seamless multilingual tokenization. Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set. For Sinhala, SGPE achieves a Token to Word Ratio (TWR) of 1.274 with 4.83 characters per token, representing a 61.7 percent reduction in tokens compared to OpenAI's o200k base. For Hindi, it achieves a TWR of 1.181 (27.0 percent reduction vs o200k). On the mixed-script (Sinhala, Devanagari, and English) dataset, SGPE achieves an overall TWR of 1.240, representing token reductions of 36.7 percent, 39.6 percent, and 60.2 percent relative to o200k base, Llama 4 Scout, and DeepSeek V3, respectively. This effectively extends the usable context window by up to 4.38 times for these Abugida languages while ensuring a Linguistic Zero-Breakage Guarantee, which ensures that no valid syllable is ever split across multiple tokens.

Kusal Darshana

Remeinium Research

Tokenization Subword Tokenization

Ask PDF

Aug 1, 2026

Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

Pretrained byte-level BPE tokenizers can segment underrepresented languages inefficiently. Replacing a tokenizer changes the meaning of nearly every token ID, while vocabulary expansion enlarges the model's embedding and output matrices. We study post-hoc adaptation that keeps the model-vocabulary size fixed and preserves most existing token-to-ID assignments as a construction-time compatibility property. Directly transferring tokens from a language-specific tokenizer does not guarantee derivability through the target BPE merge graph: an inserted entry can conflict with the target's greedy merge ranks. We formalize this failure as the merge ordering problem and introduce BPE-guided insertion, which builds each transferred token through a target-reachable decomposition. Our pipeline uses script-aware row selection to limit collateral fragmentation, reconstructs target-script byte-level prerequisites, and applies guided insertion to maintain merge-graph reachability. On Ukrainian adaptations of Nemotron and GPT-OSS, it reduces token counts by 33.5% and 36.6%, keeps changes on English and the evaluated four-language European aggregate within 0.05%, and retains 78.5%/77.3% of original model-vocabulary rows at the same IDs. Constraint-matched global and frequency-based removal achieve similar Ukrainian compression but increase English/European token counts by 0.7-2.2%; fresh same-size retraining compresses Ukrainian slightly more but retains effectively no same-ID rows and increases English token counts by 7.6-8.6%. The reallocation increases token counts on the evaluated three-language Cyrillic micro-aggregate by 6.7%/10.1%. Structural audits find all 28,134/45,398 inserted BPE nodes reachable under ordinary rank-ordered merging and no retained same-ID model-vocabulary entry newly broken. We release all tokenizers and code.

Bohdan Didenko

Lviv Polytechnic National University

Low-Resource Language Processing Subword Tokenization Tokenizer Adaptation Byte-Pair Encoding Adaptive Tokenization

Ask PDF

Mar 6, 2026

VerChol -- Grammar-First Tokenization for Agglutinative Languages

Tokenization is the foundational step in all large language model (LLM) pipelines, yet the dominant approach Byte Pair Encoding (BPE) and its variants is inherently script agnostic and optimized for English like morphology. For agglutinative languages a typological class encompassing the Dravidian family (Tamil, Kannada, Telugu, Malayalam), Turkic languages (Turkish, Azerbaijani, Uzbek), Uralic languages (Finnish, Hungarian, Estonian), Korean, Japanese, Swahili, Basque, and others, a single word may encode root, tense, aspect, person, number, gender agreement, case, and postpositions into one orthographic unit. Statistical tokenizers fragment these words into byte pair chunks that sever morpheme boundaries and inflate token counts.

Prabhu Raja

Bengaluru, India

Tokenization in Language Models Tokenization Subword Tokenization

Ask PDF

Abstract

Tokenization in Language Models

Explore similar work

Separate Before You Compress: The WWHO Tokenization Architecture

Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

VerChol -- Grammar-First Tokenization for Agglutinative Languages