HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval

Senol Gulgonul

cs.CLApr 12, 2026

HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval

Authors: Senol Gulgonul

Organizations: Electrical and Electronics Engineering, Ostim Technical University, Ankara Turkey

Abstract

HeceTokenizer is a syllable-based tokenizer for Turkish that exploits the deterministic six-pattern phonological structure of the language to construct a closed, out-of-vocabulary (OOV)-free vocabulary of approximately 8,000 unique syllable types. A BERT-tiny encoder (1.5M parameters) is trained from scratch on a subset of Turkish Wikipedia using a masked language modeling objective and evaluated on the TQuAD retrieval benchmark using Recall@5. Combined with a fine-grained chunk-based retrieval strategy, HeceTokenizer achieves 50.3% Recall@5, surpassing the 46.92% reported by a morphology-driven baseline that uses a 200 times larger model. These results suggest that the phonological regularity of Turkish syllables provides a strong and resource-light inductive bias for retrieval tasks.

Tokenization

See all 45 papers

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
Dec 28, 2025Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör +1Language Model Pretraining Long-Context Language Modeling
Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
Sep 18, 2026Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le +4Tokenization Language Model Pretraining
University of Information Technology, Vietnam National University, Ho Chi Minh city, Viet Nam · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Abu Dhabi, UEA

Jun 17, 2026J/K move · Enter open · S save

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

Tolga Şakar

Independent Researcher

Word Embeddings Subword Tokenization Tokenization in Language Models Tokenization Morphological Analysis

Open paper AskPDF arXiv

Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents Morpheus, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so

\mathrm{decode}(\mathrm{encode}(w)) = w

holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character (

1.425

), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1

0.61

vs.\

{\sim}0.32

), and uses

{\sim}19\%

less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP

0.85

) and same-root verification (ROC-AUC

1.00

), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.

Abstract

Tokenization

Explore similar work

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining