Morphological Analysis

Follow this topic Ask about this topic

Latest papers

20 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Oct 14, 2026

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog

Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys. Studies routinely establish that it helps while assuming the boundaries must be linguistically correct. We separate them. Tagalog marks voice with infixes (s-um-ulat 'wrote', s-in-ulat 'was written'), so the root is not contiguous and no sequence of BPE merges can recover it. We build MorphAustro-Tok, a tokenizer that emits the root as a contiguous unit followed by a positional infix token, and compare it against a corpus-matched BPE baseline and, at two of three fertility levels, against a scrambled-morpheme control that preserves morph count, invertibility and per-type consistency while randomising the pseudo-infix. Across 33 runs, MorphAustro-Tok improves Tagalog machine translation over matched BPE by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points. Token count does not explain the gain, since it grows as segmentation coarsens, and neither does capacity: at the coarsest level the scrambled arm carries more parameters and scores lower. Scrambling the analysis moves the gain by less than between-seed dispersion at both controlled levels, so we detect no contribution from morphological correctness at this scale.

Abdul Rehman Antall, Baidya Nath Saha

National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Pakistan · Concordia University of Edmonton, Alberta, Canada

Subword Tokenization Morphological Analysis Machine Translation

Sep 18, 2026

Not All Irregularity Is Equal: Causally Isolating a Rare Failure Mode in Japanese Morphological Inflection

Neural morphological generation systems often achieve high aggregate accuracy on benchmark datasets, yet such performance can conceal systematic errors clustered in rare morphological subclasses. We present an orthography-aware diagnosis of Japanese past-tense verb inflection, treating hiragana not merely as a transcriptional medium but as a representational system that encodes morphophonological structure. Using two character-level Transformer architectures evaluated across five random seeds, we show that although both systems exceed 97% aggregate accuracy, a single structurally specific irregular subtype, verbs whose stems end in /e/ and require gemination before the past-tense suffix and make up fewer than 1% of the data, accounts for a disproportionate 30-43% share of residual errors and contributes roughly 34-48x its prevalence to total errors. We then move from diagnosis to causal isolation: controlled ablation experiments show that removing this subtype alone produces larger accuracy gains than removing all irregular verbs combined. These findings indicate that error concentration in neural morphological learning is not driven by irregularity per se, but by the interaction between extreme low-frequency morphological patterns and specific orthographic processes. We argue that morphological evaluation should incorporate fine-grained subclass analysis, and discuss implications for data-efficient, developmentally plausible language model pretraining.

Wen Zhang

Morphological Analysis

Sep 7, 2026

An LLM-Associated Register Shift in Korean Journal Abstracts: A Morphology-Aware Excess-Vocabulary Study, 2018-2026

Excess vocabulary, a word's frequency above its pre-2023 trend, is how the change in scholarly English after 2022 has been measured. We adapt it to Korean with morphological units on 398,296 KCI abstracts (2018-August 2026), with 47,165 Vietnamese abstracts for comparison. Placebo floors are 0.1-2.2 points for the single-word statistic and at most 2.9 for the re-selected split-half set statistic. Korean abstracts show nothing in 2023, onset in late 2024, a rise through 2025 flattening in mid-2026: sisahada "suggest" appears in 21.4% of 2026 abstracts against 5.3% expected; plain verbs like araboda "look into" fall to a quarter of trend. Under stated assumptions the single-word conditional lower bound on LLM-processed abstracts is 3.5%, 10.5% and 16.1% for 2024-2026 and a split-half set bound 7.8%, 20.6% and 33.0%. Holzwarth et al.'s estimator under the same discipline gives 41.9% and 72.1% for 2025-2026. Subject-matter controls reduce but do not remove it: restricting the set to lemmas three language-model annotators all call style leaves 14.7 of the 33.0 points, and pairing each 2026 abstract with its journal's closest base-period abstract leaves 34.1. Tested translation routes do not explain it: the surface marks of translated Korean fall as the markers rise. In the same articles' English abstracts the excess appears a year earlier; where the English side carries none, the Korean shift persists at 30 to 66% of the rate where it does. Control abstracts from three providers reproduce the rising words, with marker turnover consistent with model generations; implied prevalences are scenario-dependent.

Aron Lee

INTFRAME Research

AI-Generated Text Detection Scientific Communication Morphological Analysis

Aug 4, 2026

Probing Character-level Transformers for the Spanish L-shaped Morphome

When a transformer learns an irregular morphological pattern, what has it learned? Our test case is the Spanish L-shaped morphome, a complex irregular pattern in which the verb's stem alternates in exactly the first-person singular indicative and all subjunctive forms, and whose membership no phonological, semantic, or syntactic feature predicts. Prior studies have shown that character-level transformers can reproduce this pattern, but that evidence describes what models produce, not what they represent. Probing five architectures, twelve trained models each, under lemma-disjoint cross-validation with controls and surface baselines, we show that the models encode the L-shaped class itself, not just its visible alternations. It is decodable above every surface baseline, survives instances in which every form shows the same stem, and probes trained on alternating instances still classify non-alternating ones. The encoding is localized where the stem choice is made, at the stem-final consonant position of the middle decoder, before the alternant is read. And it is item-specific: which verbs a model learned matters far more than which architecture it is. The models store the morphome as an item-specific lexical abstraction, sufficient to reproduce the pattern but not to generalize it as humans do.

Akhilesh Kakolu Ramarao, Kevin Tang, Wiebke Petersen +1

Department of English Language and Linguistics, Heinrich Heine University Düsseldorf · Department of Linguistics, College of Liberal Arts and Sciences, University of Florida · Institute of Linguistics and Information Science, Heinrich Heine University Düsseldorf +1

Transformer Interpretability Transformer Representation Probing Morphological Analysis

Jul 30, 2026

The Morphological Core of Dungan: A Two-Dialect Finite-State Model and a Multi-Genre Evaluation

Dungan, a Sinitic language of Central Asia written in a Cyrillic-based script, is described in detail in the grammatical literature, yet the quantitative properties of its morphology in actual usage have, to the best of our knowledge, never been measured systematically. This paper uses a finite-state morphological analyzer as a measuring instrument. Implemented with HFST and covering both dialect groups (the Gansu variety, which is the literary standard, and the Shaanxi variety), the model offers no new grammatical description; it formalises the knowledge accumulated in Dungan studies and makes it measurable on corpora of three genres. Three results follow. Overt inflection is rare and limited: only 9.3% of recognized tokens in the encyclopaedic register have an overt marker, the system has just ten categories, and degree marking is almost absent. Ambiguity is genuine but sharply localized: 78.1% of tokens receive a single analysis, and the residue sits almost entirely on two clitics, -di (genitive/progressive) and -ni (locative/prospective). And the grammatical core proves effectively closed, the claim the instrument is really needed for: between 78% and 95% of the tokens the analyzer fails on, depending on register, are simply absent from the lexicon, and the phenomena the model deliberately declines to implement account for at most 4.5% of those failures. Held-out coverage (80 to 85%) is no lower than development coverage (73%), while a stem list with no morphology already reaches 67.4%, so the morphology is worth 5.2 points. The open frontier of Dungan is lexical. The analyzer, its sources and every evaluation script are released openly.

Anton M. Alekseev, Sergey I. Nikolenko

St. Petersburg Department of the Steklov Mathematical Institute, RAS · St. Petersburg State University · Kyrgyz State Technical University named after I. Razzakov

Morphological Analysis

Jul 6, 2026

Ossetic-COT: Designing a morphologically annotated corpus and morphological analyzer for Ossetic

In this work we present the first morphologically annotated corpus for Iron Ossetic that conforms to the Universal Dependencies schema. The corpus includes 5454 manually annotated sentences from the Iron Ossetic Corpus of Oral Texts, containing 74032 tokens. We use this corpus to train a BERT-based morphological analyzer. The analyzer achieves tag accuracy of 95.60%.

Anna Shatskikh, Alexey Sorokin

Lomonosov Moscow State University · Yandex

Morphosyntactic Tagging Morphological Analysis Low-Resource Language Processing

Jun 23, 2026

MorfFlex: Handling Rich Morphology

We present MorfFlex, a morphological dictionary architecture suitable for languages with extensive regularity in both inflection and derivation. As the primary example of MorfFlex in use we introduce MorfFlex CZ, a morphological dictionary of Czech. It is distributed as a simple, unstructured list of <wordform, lemma, tag> triplets, however, its manually maintained, unpublished source files and conversion scripts encode a sophisticated system of inflectional and derivational patterns. These patterns dramatically reduce the otherwise enormous size of the dictionary, which currently contains over 100 million wordforms and more than 1 million lemmas. The MorfFlex CZ dictionary serves as an essential resource for ensuring the consistency of manual morphological annotation in the Prague Dependency Treebanks and underpins state-of-the-art automatic tools such as MorphoDiTa. In this paper, we focus on: (i) presenting an effective method for managing the rich morphological system within the dictionary, and (ii) demonstrating the utility of such a language resource for maintaining annotation consistency in corpora and supporting the development of advanced NLP applications.

Jaroslava Hlaváčová, Marie Mikulová, Barbora Štěpánková +2

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Prague 1, Czech Republic

Morphological Analysis

Jun 17, 2026

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents Morpheus, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so

\mathrm{decode}(\mathrm{encode}(w)) = w

holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character (

1.425

), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1

0.61

vs.\

{\sim}0.32

), and uses

{\sim}19\%

less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP

0.85

) and same-root verification (ROC-AUC

1.00

), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.

Tolga Şakar

Independent Researcher

Word Embeddings Subword Tokenization Tokenization in Language Models Tokenization Morphological Analysis

Jun 13, 2026

PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino

Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation. This limitation is most acute for languages with non-concatenative morphology, where standard tokenizers systematically misalign token boundaries with morpheme boundaries. We introduce PACUTE, a diagnostic benchmark of 4,600 tasks designed to evaluate morphological understanding in Filipino, a language characterized by productive infixation, reduplication, and diacritic-driven lexical distinctions that are typically absent from written text. PACUTE includes a hierarchical diagnostic framework of six compositional levels that localizes where morphological understanding breaks down. Evaluating open-weight LLMs and frontier commercial models, we find that open-weight models perform near chance on morpheme decomposition regardless of scale. Frontier models perform much better, often recovering individual affixes under contains-match scoring, but remain far below their character-level ceilings on compositional tasks of morpheme transformations and syllabification. These results identify productive morphological composition, rather than character access alone, as the persistent bottleneck for Filipino word-structure understanding.

Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan +3

AI Singapore · Nanyang Technological University · UK AI Security Institute +2

LLM Evaluation Language Modeling Low-Resource Language Processing Morphological Analysis

May 27, 2026

Comonadic Morphophonology: A Compositional Framework for Context-Dependent Morphological Rules in Finnish

Composing finite-state transducers (FSTs) for context-dependent morphophonological rules -- consonant gradation, vowel harmony, possessive suffix assimilation -- leads to multiplicative state explosion; neural models sidestep the problem but provide no formal account of the rules themselves. We present the first framework where each morphophonological rule is a function from a focused local context to a single output segment -- the type of a local rule familiar from cellular automata -- and where length-changing rules compose as coKleisli arrows of a comonad. Our central contribution is the Writer comonad (DeletionSet x Zipper), a new algebraic construction that restores strict coKleisli compositionality for such rules: each rule is a coKleisli arrow, extend lifts it to a global transformation, and deletions accumulate as a monoid action rather than requiring intermediate materialization. As supporting evidence, thirteen coKleisli arrows provide an alternative formulation expressing the same morphophonological behaviors that Omorfi encodes via 874 continuation classes (67:1 reduction at the rule-representation level), and the same abstraction enables bidirectional morphology -- a MorphGenerator reuses the analysis arrows for generation. On UD Finnish-TDT, the system achieves 83.92% UPOS accuracy with rule-only disambiguation (94.66% with an external suffix tagger), validating the framework as a practical morphological engine.

Yongseok Jang

Independent Researcher

Morphological Analysis Category Theory

May 26, 2026

Formalization of Malagasy conjugation

This paper reports the core linguistic work performed to construct a dictionary-based morphological analyser for Malagasy simple verbs. It uses the Unitex platform and comprised the contruction of an electronic dictionary for Malagasy simple verbs. The data is encoded on the basis of morphological features. The morphological variations of verb stems and their combination with inflectional affixes are formalized in finite-state transducers represented by editable graphs. 78 transducers allow Unitex to generate a dictionary of allomorphs of stems. 271 other transducers are used by the morphological analyser of Unitex to recognize the stem and the affixes in conjugated verbs. The design of the dictionary and transducers prioritizes readability, so that they can be extended and updated by linguists.

Joro Ny Aina Ranaivoarison, Eric Laporte, Baholisoa Simone Ralalaoherivony

University of Antananarivo, Centre Interdisciplinaire de Recherche Appliquée au Malgache, Madagascar · Université Paris-Est, Laboratoire d'informatique Gaspard-Monge CNRS UMR 8049, F77454 Marne-la-Vallée, France

Morphological Analysis Low-Resource Language Processing

May 21, 2026

Pattern-and-root inflectional morphology: the Arabic broken plural

We present a substantially implemented model of description of the inflectional morphology of Arabic nouns, with special attention to the management of dictionaries and other language resources by Arabic-speaking linguists. The breakthrough lies in the reversal of the traditional root-and-pattern Semitic model into pattern-and-root, giving precedence to patterns over roots. Our model includes broken plurals (BPs), i.e. plurals formed by modifying the stem. It is based on the traditional notions of root and pattern of Semitic morphology. However, as compared to traditional Arabic morphology, it keeps the formal description of inflection separate from that of derivation and semantics. As traditional Arabic dictionaries, the updatable dictionary is structured in lexical entries for lemmas, and the reference spelling is fully diacritized. In our model, morphological analysis of Arabic text is performed directly with a dictionary of words and without morphophonological rules. Our taxonomy for noun inflection is simple, orderly and detailed. We simplify the taxonomy of singular patterns by specifying vowel quantity as v or vv, and ignoring vowel quality. Root alternations and orthographical variations are encoded independently from patterns and in a factual way, without deep roots or morphophonological or orthographical rules. Nouns with a triliteral BP are classified according to 22 patterns subdivided into 90 classes, and nouns with a quadriliteral BP according to 3 patterns subdivided into 70 classes. These 160 classes become 300 inflectional classes when we take into account inflectional variations that affect only the singular. We provide a straightforward encoding scheme that we applied to 3 200 entries of BP nouns.

Alexis Amid Neme, Eric Laporte

LIGM, Université Paris-Est · DLL, Universidade Federal do Espírito Santo

Morphological Analysis

May 19, 2026

When Irregularity Helps: A Subclass Analysis of Inductive Bias in Neural Morphology

Neural morphological generation systems often achieve high aggregate accuracy on benchmark datasets, yet such performance can conceal systematic errors concentrated in rare morphological subclasses. We examine Japanese past-tense verb inflection and show that a very small, structurally specific irregular subtype (<1% of data) accounts for a disproportionate share of model errors. Controlled ablation experiments demonstrate that removing this subtype yields larger improvements in generalization than removing all irregular verbs, indicating that not all irregularity contributes equally to model instability. These findings suggest that error concentration is driven by the interaction between extreme low-frequency morphological patterns and specific morphophonological processes, particularly gemination. We argue that morphological evaluation should incorporate finer-grained subclass analysis beyond standard conjugation categories.

Wen Zhang

Neural Network Generalization Long-Tail Learning Morphological Analysis

May 13, 2026

Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agent

Today's agents are highly effective at implementing well-scoped software design plans, but user intent is often vague and admits multiple equally valid solutions. In this paper, we introduce SpecBench, a new benchmark for evaluating an agent's ability to translate user intent into a structured, executable specification that aligns with user preferences. The agent is given access to past user conversations and may interact with the user for a fixed number of rounds to ask clarifying questions. We find that existing agents exhibit two extreme behaviors: they either (i) struggle to collaborate proactively with users, entering implementation mode too quickly while overestimating their understanding of user preferences, or (ii) exhaust their question budget by asking about every ambiguous design choice. To address this limitation, we introduce a user-assistant agent: Buddy. It follows a workflow inspired by classical morphological analysis, decomposing user intent into a structured space of design dimensions and candidate choices. It then creates simulated users to evaluate these choices, before engaging the real user to resolve remaining ambiguities and finalize the specification. By shifting the focus from execution to specification, SpecBench and Buddy emphasize agent-user collaboration (not just code generation) as a key frontier in future agent design.

Hao Wang, Ligong Han, Kai Xu +1

Red Hat AI Innovation · MIT-IBM Watson AI Lab · IBM Core AI

Requirements Engineering Software Engineering Agents AI Agent Benchmarks Morphological Analysis Conversational Agents

May 6, 2026

TajikNLP: An Open-Source Toolkit for Comprehensive Text Processing of Tajik (Cyrillic Script)

The Tajik language, written in Cyrillic script, remains severely under-resourced in terms of publicly available natural language processing (NLP) toolkits, hindering both linguistic research and applied development. This paper introduces TajikNLP, an open-source Python library that provides the first comprehensive pipeline for processing authentic Tajik text while preserving the original Cyrillic orthography. The library implements a modular architecture centered around a unified Doc object, enabling sequential application of components for cleaning, normalization, tokenization (including subword BPE), morphemic segmentation, part-of-speech tagging, stemming, lemmatization, and sentence splitting. A novel unified morphology engine is introduced, offering controlled and deep analysis modes that significantly improve handling of Tajik's agglutinative nominal and verbal inflections. The release further incorporates a lexicon-based sentiment analyser and pre-trained Word2Vec/FastText embeddings loaded directly from the Hugging Face Hub. To ensure reproducibility and facilitate future research, four accompanying linguistic datasets -- a POS-tagged corpus (52.5k entries), a sentiment lexicon (3.5k entries), a toponym gazetteer (5.6k entries), and a personal names dataset (3.8k entries) -- have been openly published under permissive licenses. The library's reliability is validated by an extensive test suite of 616 automated tests achieving 93% source code coverage. TajikNLP thus establishes a foundational technological infrastructure for Tajik language processing, lowering the barrier to entry for both academic and industrial applications in low-resource Cyrillic-script environments.

Mullosharif K. Arabov, Karomatullo Habibullozoda, Nurali Shirinov

Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia · Bokhtar State University named after Nosiri Khusrav Bokhtar, Tajikistan

Morphosyntactic Tagging Tokenization Morphological Analysis Low-Resource Language Processing

Apr 24, 2026

Zero-Shot Morphological Discovery in Low-Resource Bantu Languages via Cross-Lingual Transfer and Unsupervised Clustering

We present a method for discovering morphological features in low-resource Bantu languages by combining cross-lingual transfer learning with unsupervised clustering. Applied to Giriama (nyf), a language with only 91 labeled paradigms, our pipeline discovers noun class assignments for 2,455 words and identifies two previously undocumented morphological patterns: an a- prefix variant for Class 2 (vowel coalescence - the merger of two adjacent vowels - of wa-, 95.1% consistency) and a contracted k'- prefix (98.5% consistency). External validation on 444 known Giriama verb paradigms confirms 78.2% lemmatization accuracy, while a v3 corpus expansion to 19,624 words (9,014 unique lemmas) achieves 97.3% segmentation and 86.7% lemmatization rates across all major word classes. Our ensemble of transfer learning from Swahili and unsupervised clustering, combined via weighted voting, exploits complementary strengths: transfer excels at cognate detection (leveraging ~60% vocabulary overlap) while clustering discovers language-specific innovations invisible to transfer. We release all code and discovered lexicons to support morphological documentation for low-resource Bantu languages.

Hillary Mutisya, John Mugane

Thiomi NLP · Harvard University

Low-Resource Language Processing Morphological Analysis

Apr 14, 2026

GLeMM: A large-scale multilingual dataset for morphological research

In derivational morphology, what mechanisms govern the variation in form-meaning relations between words? The answers to this type of questions are typically based on intuition and on observations drawn from limited data, even when a wide range of languages is considered. Many of these studies are difficult to replicate and generalize. To address this issue, we present GLeMM, a new derivational resource designed for experimentation and data-driven description in morphology. GLeMM is characterized by (i) its large size, (ii) its extensive coverage (currently amounting to seven European languages, i.e., German, English, Spanish, French, Italian, Polish, Russian, (iii) its fully automated design, identical across all languages, (iv) the automatic annotation of morphological features on each entry, as well as (v) the encoding of semantic descriptions for a significant subset of these entries. It enables researchers to address difficult questions, such as the role of form and meaning in word-formation, and to develop and experimentally test computational methods that identify the structures of derivational morphology. The article describes how GLeMM is created using Wiktionary articles and presents various case studies illustrating possible applications of the resource.

Hathout Nabil, Basilio Calderone, Fiammetta Namer +1

CLLE, Comue de Toulouse · CLLE, CNRS, Université Toulouse Jean Jaurès, Toulouse, France · CLLE, UBM +4

Morphological Analysis

Apr 3, 2026

Towards a theory of morphology-driven marking in the lexicon: The case of the state

All languages have a noun category, but its realisation varies considerably. Depending on the language, semantic and/or morphosyntactic differences may be more or less pronounced. This paper explores these variations, using Riffian as a reference point before extending the analysis to other languages. We propose a formal model termed morphology-driven marking. Nouns are organised into modular cognitive sets, each with its own morphological template and unmarked form. This approach helps explain differences in marking among noun types within and across languages. By situating these patterns within syntactic functions, we also reassess the notions of markedness and state. It is proposed that the concept of state be extended to all synthetic languages and analysed a novel subcategory of syntax-based inflection like agreement and grammatical case.

Mohamed El Idrissi

INALCO

Morphological Analysis Linguistic Typology

Apr 1, 2026

Valency Classification of Mapudungun Verbal Roots. Established by the language's own morphotactics

In the previous work, a lexical (re)categorisation -- or confirmation of the given category -- of roots identified as verbal was undertaken to determine their original category accurately. Building on this, the present paper offers an account of the valency classification of those Mapudungun roots confirmed to be verbal, using the language's own morphotactics; specifically, by examining the permissible and restricted combinations of various suffixes with roots or verbal stems in the Mapuche verb form. As with all work conducted thus far, the results presented here aim to improve the morphological analyser (Dungupeyum) with all verified findings incorporated into the system. From a theoretical perspective, we also hope to contribute to the recognition and understanding of issues related to the valency of Mapuche verb forms.

Andrés Chandía

Department of Catalan Philology and General Linguistics University of Barcelona

Morphological Analysis

Feb 22, 2026

TurkicNLP: An NLP Toolkit for Turkic Languages

Turkic languages are spoken by about 200 million people and are written in Latin, Cyrillic, and Perso-Arabic scripts, yet the NLP tools that exist for them are scattered across languages, scripts, and incompatible software. We present TurkicNLP, an open-source Python library that offers a single pipeline for 24 Turkic languages, covering tokenization, morphological analysis, POS tagging, dependency parsing, named entity recognition, language identification, sentence embeddings, and machine translation. The library is designed for cross-lingual use: it detects and converts between scripts, maps 23 languages into the shared Common Turkic Alphabet, and includes multilingual models that share parameters across the family and transfer to languages without annotated data. Our evaluation shows the benefits of this design. The multilingual parser outperforms per-language parsers, the multilingual morphological analyzer complements finite-state analyzers where their lexicons are small, and both produce usable analyses for languages unseen in training. Transliteration is almost fully reversible, and converting text to the Common Turkic Alphabet removes most of the additional tokenization cost that large language models incur on Cyrillic- and Arabic-script Turkic languages. Code, models, and documentation are available at: https://github.com/turkic-nlp/turkicnlp.

Sherzod Hakimov

Computational Linguistics, University of Potsdam

Morphological Analysis Low-Resource Language Processing