Morphology-Aware Tokenization

Latest papers

CardsList

15 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and Hausa

    Nov 10, 2026Abdul Rehman Antall, Baidya Nath SahaTokenization in Language ModelsMorphology-Aware Tokenization

  2. khammoon: From a Linguistic Guideline to a Thai Morpheme Segmenter

    Nov 9, 2026Warit Sirichotedumrong, Adisai Na-Thalang, Chanakan Wittayasakpan +2Morphology-Aware TokenizationMorphological Analysis

  3. MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling

    Nov 8, 2026Gül Sena Altıntaş, Colin RaffelTokenization in Language ModelsMultilingual Language Models

  4. Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

    Oct 14, 2026Abdul Rehman Antall, Baidya Nath SahaMorphology-Aware TokenizationSubword Tokenization

  5. Latent Core Tokenizer: Compress, but Meaningfully

    Oct 8, 2026Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie +3Morphology-Aware TokenizationTokenization in Language Models

  6. Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali

    Sep 27, 2026Kalash Shrestha, Nikhil PradhanLanguage Model PretrainingLow-Resource Language Processing

  7. MoirfEolas and CríochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology

    Sep 4, 2026Jane Adkins, Abigail Walsh, Brian Davis +1Morphology-Aware TokenizationSubword Tokenization

  8. BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

    Jul 25, 2026Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran +1Low-Resource Language ProcessingSubword Tokenization

  9. Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

    Mar 16, 2026Yara Alakeel, Chatrine Qwaider, Hanan Aldarmaki +1Arabic NLPArabic Language Models

  10. MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version

    Sep 10, 2025Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang NejdlMorphology-Aware TokenizationSubword Tokenization