Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

Abdul Rehman Antall; Baidya Nath Saha

cs.CLOct 14, 2026

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

Authors: Abdul Rehman Antall, Baidya Nath Saha

Organizations: National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Pakistan · Concordia University of Edmonton, Alberta, Canada

Abstract

Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys. Studies routinely establish that it helps while assuming the boundaries must be linguistically correct. We separate them. Tagalog marks voice with infixes (s-um-ulat 'wrote', s-in-ulat 'was written'), so the root is not contiguous and no sequence of BPE merges can recover it. We build MorphAustro-Tok, a tokenizer that emits the root as a contiguous unit followed by a positional infix token, and compare it against a corpus-matched BPE baseline and, at two of three fertility levels, against a scrambled-morpheme control that preserves morph count, invertibility and per-type consistency while randomising the pseudo-infix. Across 33 runs, MorphAustro-Tok improves Tagalog machine translation over matched BPE by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points. Token count does not explain the gain, since it grows as segmentation coarsens, and neither does capacity: at the coarsest level the scrambled arm carries more parameters and scores lower. Scrambling the analysis moves the gain by less than between-seed dispersion at both controlled levels, so we detect no contribution from morphological correctness at this scale.

Figures & tables

Level	Morph	BPE	$\Delta$ [95% CI]
F0 (0.319)	11.35	10.44	+0.844 [0.682, 1.011]
	$\pm$ 2.18	$\pm$ 2.27
F1 (0.379)	10.48	0 9.29	+1.488 [1.291, 1.690]
	$\pm$ 2.53	$\pm$ 2.37
F2 (0.438)	10.93	0 7.48	+3.396 [3.213, 3.580]
	$\pm$ 1.34	$\pm$ 1.58

Table 1: chrF++ on FLORES+ devtest, mean

\pm

sd over three seeds, with the paired-bootstrap difference and its 95% interval. All three reach the bootstrap resolution floor of § 3.7 .

Level	Evaluation set	Morph $-$ BPE	Scrambled $-$ BPE	Morph $-$ Scrambled	$p_{\text{Holm}}$	Seed sd
F1	FLORES+ devtest	+1.488	+1.543	$-$ 0.055	ns	3.040
F1	In-domain split	+1.631	+1.828	$-$ 0.196	ns	2.528
F2	FLORES+ devtest	+3.396	+2.735	+0.661	at floor	2.006
F2	In-domain split	+3.933	+3.294	+0.639	at floor	1.566

Table 2: Mean per-sentence chrF++ differences over the matched evaluation set. The last column is the pooled between-seed standard deviation for the Morph versus Scrambled contrast.

Level	Cond.	Vocab	Params	chrF++
F0	BPE	5,842	1,673,984	10.44
F0	Morph	7,780	1,922,048	11.35
F1	BPE	2,451	1,239,936	0 9.29
	Morph	2,936	1,302,016	10.48
	Scr.	3,567	1,382,784	10.62
F2	BPE	1,483	1,116,032	0 7.48

Table 3: Vocabulary size, inference-path parameters and chrF++ for every Tagalog cell of the sweep.

Level	Condition	P	R	F1
F0	BPE	$.599\pm.031$	$.700\pm.027$	$.646\pm.030$
F0	Morph	$.601\pm.023$	$.703\pm.009$	$.648\pm.017$
F1	BPE	$.597\pm.030$	$.705\pm.018$	$.646\pm.025$
	Morph	$.618\pm.018$	$.720\pm.008$	$.665\pm.013$
	Scrambled	$.587\pm.008$	$.694\pm.010$	$.636\pm.009$
F2	BPE	$.602\pm.012$	$.704\pm.012$	$.649\pm.012$

Table 4: WikiANN Tagalog entity-level precision, recall and F1 ( seqeval ), mean

\pm

sd over three seeds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture (identical in every condition)
Encoder / decoder layers	2 / 2
$d_{\text{model}}$	128
Attention heads	4
FFN dimension	512
Dropout	0.1
Normalisation	pre-LN, GELU

Appendix

Table 5: Training configuration.

	Tagalog	Indonesian
ParaCrawl pairs parsed	248,689	400,000
retained after filtering	223,832	283,018
retention rate	90.0%	70.8%
MT train	30,000	30,000
In-domain held-out pool	2,000	2,000
scored	300	n/a

Appendix

Table 6: Corpus sizes. WikiANN rows are the Tagalog splits.

Level	Condition	Vocab	Achieved
F0 (target 0.31867)	BPE	5,842	0.31579
F0 (target 0.31867)	Morph	7,780	0.31732
F1 (target 0.37858)	BPE	2,451	0.38177
	Morph	2,936	0.38335
	Scrambled	3,420	0.37490
F2 (target 0.43849)	BPE	1,483	0.43582

Appendix

Table 7: Achieved tokens per character and the vocabulary size each condition required to reach it.

Figure 1: Gain over matched BPE against fertility, with the shaded region marking differences smaller than the between-seed standard deviation.

Subword Tokenization

Subword tokenization divides text into reusable units smaller than words to support efficient and flexible language modeling.

See all 81 papers

QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages
Jun 22, 2026Maria ContrerasLow-Resource Language Processing Subword Tokenization
Universidad Peruana de Ciencias Aplicadas (UPC) Lima, Peru
VerChol -- Grammar-First Tokenization for Agglutinative Languages
Mar 6, 2026Prabhu RajaTokenization in Language Models Tokenization
Bengaluru, India
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher

Jun 22, 2026J/K move · Enter open · S save

QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages

Maria Contreras

Universidad Peruana de Ciencias Aplicadas (UPC) Lima, Peru

Low-Resource Language Processing Subword Tokenization

Open paper AskPDF arXiv

Tokenization is a foundational step in NLP pipelines, yet standard evaluation metrics such as fertility rate fail to capture morphological correctness for agglutinative languages. We present QuechuaTok, a systematic benchmark comparing four tokenization strategies - BPE, Unigram LM, WordPiece, and a morphology-aware PRPE tokenizer - for Southern Quechua (quz), a low-resource agglutinative language spoken by 8-10 million people in South America. Using a 200k-sentence corpus and the SQUOIA finite-state morphological analyzer (Rios, 2016) as silver standard, we evaluate three metrics: fertility rate, OOV rate, and morphological boundary accuracy (MorphAcc). Our results show that BPE achieves the lowest fertility rate (1.636 at 16k vocab) by memorizing surface word forms, while achieving only 6.67% MorphAcc. PRPE achieves 83.33% MorphAcc - the highest of all systems - demonstrating that fertility rate alone is insufficient to evaluate tokenizers for agglutinative languages. All code and models are publicly available at kaggle.com/code/macmaky/quechuatok

Abstract

Subword Tokenization

Explore similar work

QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages

VerChol -- Grammar-First Tokenization for Agglutinative Languages

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish