cs.CLSep 10, 2025

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version

Authors: Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl

Organizations: L3S Research Center, Leibniz University Hannover, Germany

Abstract

Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Ge'ez script. To address this, we present Morpheme-aware Subword Vocabulary Construction (MoVoC) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we release morpheme-annotated data for four Ge'ez script languages (169,806 records) and a morpheme-aware vocabulary for two of them. Scoring every comparison on words held out from the morpheme resource that constrains the MoVoC-Tok applied to that language, MoVoC-Tok has the highest MorphScore in three of four languages and the highest morpheme boundary precision in two of four, with its largest and most clearly resolved gains on Tigre, a language it was not trained on and to which it is applied cross-lingually. The machine translation results reported in the published version are withdrawn. Data, tokenizers and evaluation code are publicly available.

Explore similar work

CardsList
  1. Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

    Jul 16, 2026Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang NejdlLanguage Model PretrainingMultilingual Language Models