Morphology-Aware Tokenization

Latest papers

12 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog
Oct 14, 2026Abdul Rehman Antall, Baidya Nath SahaMorphology-Aware Tokenization Subword Tokenization
National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Pakistan · Concordia University of Edmonton, Alberta, Canada
Latent Core Tokenizer: Compress, but Meaningfully
Oct 8, 2026Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie +3Morphology-Aware Tokenization Tokenization in Language Models
Microsoft Research Africa · Microsoft Research Accelerator · Microsoft Research India
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali
Sep 27, 2026Kalash Shrestha, Nikhil PradhanLanguage Model Pretraining Low-Resource Language Processing
Kathmandu University · IOE Thapathali Campus
MoirfEolas and CríochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology
Sep 4, 2026Jane Adkins, Abigail Walsh, Brian Davis +1Morphology-Aware Tokenization Subword Tokenization
ADAPT Centre, Dublin City University · Trinity College Dublin
Morphology Aware Reversible Semantic Tokenization and Hierarchical Word Composition for Tamil Language Models
Aug 2, 2026Anand MuruganSubword Tokenization Morphology-Aware Tokenization
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
Jul 25, 2026Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran +1Low-Resource Language Processing Subword Tokenization
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi, New Delhi 110016, India · RSL Quantum, FITT, IIT Delhi, New Delhi 110016, India
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Jun 25, 2026Sander LandToken Efficiency Subword Tokenization
Writer, Inc.
QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages
Jun 22, 2026Maria ContrerasLow-Resource Language Processing Subword Tokenization
Universidad Peruana de Ciencias Aplicadas (UPC) Lima, Peru
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
Jun 17, 2026Tolga ŞakarWord Embeddings Subword Tokenization
Independent Researcher
Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs
Mar 16, 2026Yara Alakeel, Chatrine Qwaider, Hanan Aldarmaki +1Arabic NLP Arabic Language Models
Saudi Data & AI Authority (SDAIA) · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Princess Nourah Bint Abdulrahman University (PNU)
VerChol -- Grammar-First Tokenization for Agglutinative Languages
Mar 6, 2026Prabhu RajaMorphology-Aware Tokenization Tokenization in Language Models
Bengaluru, India
MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version
Sep 10, 2025Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang NejdlMorphology-Aware Tokenization Subword Tokenization
L3S Research Center, Leibniz University Hannover, Germany

Oct 14, 2026J/K move · Enter open · S save

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

Abdul Rehman Antall, Baidya Nath Saha

National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Pakistan · Concordia University of Edmonton, Alberta, Canada

Morphology-Aware Tokenization Subword Tokenization Morphological Analysis Machine Translation

Open paper AskPDF arXiv

Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys. Studies routinely establish that it helps while assuming the boundaries must be linguistically correct. We separate them. Tagalog marks voice with infixes (s-um-ulat 'wrote', s-in-ulat 'was written'), so the root is not contiguous and no sequence of BPE merges can recover it. We build MorphAustro-Tok, a tokenizer that emits the root as a contiguous unit followed by a positional infix token, and compare it against a corpus-matched BPE baseline and, at two of three fertility levels, against a scrambled-morpheme control that preserves morph count, invertibility and per-type consistency while randomising the pseudo-infix. Across 33 runs, MorphAustro-Tok improves Tagalog machine translation over matched BPE by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points. Token count does not explain the gain, since it grows as segmentation coarsens, and neither does capacity: at the coarsest level the scrambled arm carries more parameters and scores lower. Scrambling the analysis moves the gain by less than between-seed dispersion at both controlled levels, so we detect no contribution from morphological correctness at this scale.