TurkicNLP: An NLP Toolkit for Turkic Languages

Sherzod Hakimov

cs.CLFeb 22, 2026

TurkicNLP: An NLP Toolkit for Turkic Languages

Authors: Sherzod Hakimov

Organizations: Computational Linguistics, University of Potsdam

Abstract

Turkic languages are spoken by about 200 million people and are written in Latin, Cyrillic, and Perso-Arabic scripts, yet the NLP tools that exist for them are scattered across languages, scripts, and incompatible software. We present TurkicNLP, an open-source Python library that offers a single pipeline for 24 Turkic languages, covering tokenization, morphological analysis, POS tagging, dependency parsing, named entity recognition, language identification, sentence embeddings, and machine translation. The library is designed for cross-lingual use: it detects and converts between scripts, maps 23 languages into the shared Common Turkic Alphabet, and includes multilingual models that share parameters across the family and transfer to languages without annotated data. Our evaluation shows the benefits of this design. The multilingual parser outperforms per-language parsers, the multilingual morphological analyzer complements finite-state analyzers where their lexicons are small, and both produce usable analyses for languages unseen in training. Transliteration is almost fully reversible, and converting text to the Common Turkic Alphabet removes most of the additional tokenization cost that large language models incur on Cyrillic- and Arabic-script Turkic languages. Code, models, and documentation are available at: https://github.com/turkic-nlp/turkicnlp.

Figures & tables

Language (ISO)	Sentence	Language (ISO)	Sentence
Latin script
Turkish (tur)	Ali kitabı gördü.	Azerbaijani (aze)	Ali kitabı gördü.
Turkmen (tuk)	Ali kitaby gördi.	Gagauz (gag)	Ali kitabı gördü.
Crimean Tatar (crh)	Ali kitanı kördi.	Karakalpak (kaa)	Ali kitaptı kórdi.
Uzbek (uzb)	Ali kitobni koʻrdi.
Cyrillic script

Table 1: The sentence “Ali saw the book” in 20 Turkic languages: subject, accusative-marked object, and sentence-final verb, written in the three active script families (Latin, Cyrillic, and Perso-Arabic).

Figure 1: Geographic distribution of the Turkic language family (Source: Wikimedia Commons).

Panel of Figure 2 — Figure 2: Tokenizer efficiency of four LLM tokenizers (Qwen3.5, LLaMA 3, OLMo 3, and GPT-5.2) on the 20 parallel challenge sentences in 20 Turkic languages and English. Higher values indicate less efficient segmentation.

Panel of Figure 3 — Figure 3: High-level component architecture of TurkicNLP. The Pipeline orchestrates a sequential Processor chain; the Script subsystem handles automatic script detection and transliteration; Model Management resolves and downloads model artifacts via catalog.json ; the Document Model provides a hierarchical output representation.

Language	ISO	Direction	Standard
Kazakh	kaz	Cyrl $\leftrightarrow$ Latn	Official 2021
Uzbek	uzb	Cyrl $\leftrightarrow$ Latn	Official 1995
Azerbaijani	aze	Cyrl $\leftrightarrow$ Latn	Official 1991
Tatar	tat	Cyrl $\leftrightarrow$ Latn	Zamanälif
Turkmen	tuk	Cyrl $\leftrightarrow$ Latn	1993 (revised)
Karakalpak	kaa	Cyrl $\leftrightarrow$ Latn	Official 2016

Table 2: Transliteration pairs supported by TurkicNLP. Uyghur additionally supports conversion from CTS back to the native scripts.

Language

Script(s)

Tokenize

MWT

Morph (FST)

Morph (Neu)

POS Tag

Lemmatize

Dep. Parse

NER

Translit.

CTS

Morph. Seg.

Embed./MT

Oghuz

Turkish

Latn

✓

–

✓

\sim

✓

Azerbaijani

Latn/Cyrl

✓

\times

✓

\times

✓

\sim

✓

South Azerb.

Arab

✓

\times

\times

\times

\times

\times

\times

\times

–

✓

\times

✓

Turkmen

Latn/Cyrl

✓

\times

\sim

✓

\times

✓

\sim

✓

Gagauz

Latn

✓

\times

\sim

\sim

\sim

\sim

\times

\times

–

✓

\sim

\times

Table 3: Component availability for the 24 languages of TurkicNLP (primary script first). ✓ = production/stable;

\sim

= beta, prototype, paradigm-only training, or zero-shot;

\times

= not available; – = not applicable (single-script language). Morph (Neu): Glot500 morphological analyzer; POS, Lemmatize, and Dep. Parse combine Stanza and Glot500 backends (without a tagger, UPOS comes from the morphological analyzer). Translit.: native-script conversion; CTS: Common Turkic Alphabet; Morph. Seg.: hybrid morpheme segmenter; Embed./MT: NLLB-200.

Language	ISO	Branch	Quality
Turkish	tur	Oghuz	Production
Kazakh	kaz	Kipchak	Production
Tatar	tat	Kipchak	Production
Azerbaijani	aze	Oghuz	Stable
Kyrgyz	kir	Kipchak	Stable
Uzbek	uzb	Karluk	Stable

Table 4: Apertium FST analyzer quality levels (based on Apertium lexicon size and UD tag-mapping completeness). Production : estimated

>

85% token coverage, full UD FEATS mapping. Stable :

>

70% coverage, partial UD mapping. Beta : functional with known lexicon gaps. Prototype : limited coverage, Apertium-native tags only.

			Apertium FST			Neural (Glot500)
Language	ISO	Scr	Det%	Ana%	Feat%	Det%	Ana%	Feat%
Turkish	tur	Latn	90	100	87	80	100	90
Azerbaijani	aze	Latn	50	53	42	75	100	79
Turkmen	tuk	Latn	80	69	51	75	100	90
Gagauz	gag	Latn	85	90	77	75	100	94
Kazakh	kaz	Cyrl	80	100	78	70	100	82

Table 5: Morphological analysis on the 20-sentence parallel challenge set (Appendix A ). Det% : sentences whose target UD feature is detected. Ana% : content words with a UPOS other than X . Feat% : content words with at least one feature. Bold: better backend per language and metric. Groups: Oghuz, Kipchak, Karluk, Siberian, Oghur. † Zero-shot for the neural model (no training data; proxy language embedding).

			Stanza (per-language)			Glot500 (multilingual)
Language	ISO	$n$	UPOS	UAS	LAS	UPOS	UAS	LAS
Turkish	tur	877	85.1	75.3	63.5	86.9	84.5	70.1
Azerbaijani	aze	869	84.9	72.3	60.8	87.0	76.2	64.5
Turkmen ‡	tuk	825	86.1	76.8	67.8	85.8	77.9	68.4
Kazakh	kaz	851	87.9	67.4	57.4	88.8	71.9	59.9
Kyrgyz	kir	1166	71.1	50.2	29.4	79.0	63.9	41.0

Table 6: POS tagging and dependency parsing. Both backends receive the same automatic tokenization;

n

: test words whose boundaries match the gold tokenization, over which UPOS is computed; UAS and LAS are computed over those of these words whose gold and predicted heads are also matched. Bold: better backend per language and metric. † Zero-shot (shaded): no training data and no Stanza model; Glot500 uses a proxy language embedding (Karakalpak

\to

Uzbek, Kumyk

\to

Tatar, Sakha

\to

Bashkir). ‡ Silver test set (translated and automatically annotated). Averages: nine shared languages.

Language	Datasets — Training				Datasets — Benchmarks		NLP Components						Toolkits
	UD Tree-banks	NER Corpus	Parallel/ MT Corpus	Speech Corpus	NLU Bench.	QA	Morphology	POS + Dep.	NER System	ASR	TTS	MT	Toolkit
Oghuz Branch
Turkish (tur)	( Sulubacak et al., 2016 ) ; ( Türk et al., 2019 ) ; ( Eryiğit et al., 2023 )	MilliyetNER ( Tur et al., 2003 ) ; TWNERTC ( Sahin et al., 2017 )	NLLB ( NLLB Team et al., 2022 ) ; OPUS ( Syzdykov and et al., 2021 ) ; Leipzig ( Goldhahn et al., 2012 )	CommonVoice ( Ardila et al., 2020 ) ; TSC ( ISSAI, 2024 ) ; FLEURS ( Conneau et al., 2022 )	Mukayese ( Safaya et al., 2022 ) ; TR-MMLU ( Bayram et al., 2025 ) ; TurkishMMLU ( Yüksel et al., 2024 ) ; Cetvel ( Er et al., 2026 ) ; TurkBench ( Toraman et al., 2026 ) ; TUMLU ( Isbarov et al., 2025 )	—	Apertium Prod. ( Forcada et al., 2011 ) ; Zemberek ( Akın and Akın, 2007 )	Stanza ( Qi et al., 2020 ) ; TurkishDelightNLP ( Alecakir et al., 2022 ) ; ITU pipeline ( Eryiğit et al., 2023 )	TurkishDelightNLP ( Alecakir et al., 2022 ) ; Stanza ( Qi et al., 2020 )	CommonVoice ( Ardila et al., 2020 ) ; Söyle ( Mussakhojayeva et al., 2023b ) ; TurkicASR ( Mussakhojayeva et al., 2023a )	TurkicTTS ( Yeshpanov et al., 2023 )	NLLB ( NLLB Team et al., 2022 ) ; Apertium ( Forcada et al., 2011 )	Zemberek ( Akın and Akın, 2007 ) ; TurkishDelightNLP ( Alecakir et al., 2022 ) ; ITU NLP ( Eryiğit, 2014 ) ; TurkNLP Suite ( Altinok, 2023 )
Azerbaijani (aze)	( Akhundjanova et al., 2025 )	azWikiNER ( Ibiyev and Novak, 2021 ) ; AzNER ( LocalDoc, 2024 )	NLLB ( NLLB Team et al., 2022 ) ; TIL ( Syzdykov and et al., 2021 ) ; Leipzig ( Goldhahn et al., 2012 )	Multilingual ( Mussakhojayeva et al., 2023a ) ; AzASR ( LocalDoc, 2025 ) ; FLEURS ( Conneau et al., 2022 )	TUMLU ( Isbarov et al., 2025 ) ; Kardeş-NLU ( Senel et al., 2024 )	—	Apertium Stb. ( Forcada et al., 2011 )	UDPipe ( Straka et al., 2016 )	XLM-R NER ( Samadov, 2024 )	TurkicASR ( Mussakhojayeva et al., 2023a )	TurkicTTS ( Yeshpanov et al., 2023 )	NLLB ( NLLB Team et al., 2022 )	—
Turkmen (tuk)	—	—	NLLB ( NLLB Team et al., 2022 ) ; TIL ( Syzdykov and et al., 2021 ) ; Leipzig ( Goldhahn et al., 2012 )	TurkmenSpeech ( Mamedov, 2024 )	—	—	Apertium Beta ( Forcada et al., 2011 )	—	—	Söyle ( Mussakhojayeva et al., 2023b )	TurkicTTS ( Yeshpanov et al., 2023 )	NLLB ( NLLB Team et al., 2022 )	—
Gagauz (gag)	—	—	NLLB ( NLLB Team et al., 2022 )	—	—	—	Apertium Proto. ( Forcada et al., 2011 )	—	—	—	—	NLLB ( NLLB Team et al., 2022 )	—

Table 7: Comprehensive overview of NLP resources for 24 Turkic languages. Each cell lists available tools/datasets as references; — indicates no known public resource. Branches: Oghuz, Kipchak, Karluk, Siberian, Oghur, Arghu, Historical. Apertium quality tiers: Prod. = Production, Stb. = Stable, Beta , Proto. = Prototype.

Language	Encoder-only (MLM)	Generative / Seq2Seq	Multilingual models
	BERT / RoBERTa / ELECTRA	GPT / LLaMA / T5 / BART	(coverage confirmed)
Oghuz Branch
Turkish (tur)	BERTurk (base/cased/uncased/128k; DistilBERT; ELECTRA; ConvBERT) ( Schweter, 2020 )	TURNA (1.1B enc-dec) ( Uludoğan et al., 2024 ) ; Kumru-2B ( Turker et al., 2024 ) ; Trendyol-LLM-8B ( Trendyol Tech, 2024 )	mBERT ( Devlin et al., 2019 ) ; XLM-R ( Conneau et al., 2020 ) ; Glot500 ( Imani et al., 2023 ) ; mGPT ( Shliazhko et al., 2024 ) ; NLLB ( NLLB Team et al., 2022 )
Azerbaijani (aze)	aLLMA (SMALL/BASE/LARGE) ( Isbarov et al., 2024 )	Az-Mistral (7B) ( Isbarov et al., 2024 )	mBERT ( Devlin et al., 2019 ) ; XLM-R ( Conneau et al., 2020 ) ; Glot500 ( Imani et al., 2023 ) ; mGPT ( Shliazhko et al., 2024 ) ; NLLB ( NLLB Team et al., 2022 )
Turkmen (tuk)	—	—	Glot500 ( Imani et al., 2023 ) ; mGPT ( Shliazhko et al., 2024 ) ; NLLB ( NLLB Team et al., 2022 )
Gagauz (gag)	—	—	Glot500 ( Imani et al., 2023 ) ; NLLB ( NLLB Team et al., 2022 )

Table 8: Pretrained neural language models for Turkic languages. Encoder: masked LM (BERT/RoBERTa/ELECTRA-style); Generative: causal/autoregressive LM (GPT/LLaMA-style) and encoder-decoder (T5/BART-style); Multilingual: large multilingual models with confirmed Turkic-language pretraining data. — indicates no publicly available model.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Genitive-Possessive gen_psor
eng	Ali’s book is on the table.
alt	Алинин бичиги столдо.
aze	Ali’nin kitabı masadadır.
bak	Алиҙың китабы өҫтәлдә.
chv	Алин кӗнеки сӗтел ҫинче.
crh	Ali’niñ kitabı masada.

Appendix

Table 9: Challenge sentences: Genitive-Possessive and Accusative Object.

Locative locative
eng	Ali is waiting at school.
alt	Али школдо сакып јат.
aze	Ali məktəbdə gözləyir.
bak	Али мәктәптә көтә.
chv	Али шкулта кӗтет.
crh	Ali mektepte bekley.

Appendix

Table 10: Challenge sentences: Locative and Ablative.

Dative dative
eng	Ali entered the school.
alt	Али школго кирди.
aze	Ali məktəbə girdi.
bak	Али мәктәпкә инде.
chv	Али шкула кӗнӗ.
crh	Ali mektepke kirdi.

Appendix

Table 11: Challenge sentences: Dative and Plural Noun.

1st Person Possessive psor_1sg
eng	My house is big.
alt	Мениҥ айылым јаан.
aze	Mənim evim böyükdür.
bak	Минең өйөм ҙур.
chv	Манӑн кил пысӑк.
crh	Meniñ evim büyük.

Appendix

Table 12: Challenge sentences: 1st Person Possessive and Past Tense.

Present Progressive present_prog
eng	Ali is coming now.
alt	Али эмди келип јат.
aze	Ali indi gəlir.
bak	Али хәҙер килә.
chv	Али халӗ килет.
crh	Ali şimdi kele.

Appendix

Table 13: Challenge sentences: Present Progressive and Future Tense.

Negation negation
eng	Ali did not go to school.
alt	Али школго барган јок.
aze	Ali məktəbə getmədi.
bak	Али мәктәпкә барманы.
chv	Али шкула кайман.
crh	Ali mektepke barmadı.

Appendix

Table 14: Challenge sentences: Negation and Question.

Imperative imperative
eng	Ali, open the door!
alt	Али, эжикти ач!
aze	Ali, qapını aç!
bak	Али, ишекте ас!
chv	Али, алӑка уҫ!
crh	Ali, qapını aç!

Appendix

Table 15: Challenge sentences: Imperative and Conditional.

Evidential evidential
eng	Ali apparently came.
alt	Али келип калган.
aze	Deyəsən, Ali gəlib.
bak	Күрәһең, Али килгән.
chv	Али килнӗ курӑнать.
crh	Ali kelgen kibi.

Appendix

Table 16: Challenge sentences: Evidential and Passive.

Reciprocal reciprocal
eng	They saw each other.
alt	Олор бой-бойын кӧрдилӧр.
aze	Onlar bir-birini gördülər.
bak	Улар бер-береһен күрҙеләр.
chv	Вӗсем пӗр-пӗрне курнӑ.
crh	Olar bir-birini kördiler.

Appendix

Table 17: Challenge sentences: Reciprocal and Pronouns.

Numeral numeral
eng	Three students came.
alt	Ӱч студент келди.
aze	Üç tələbə gəldi.
bak	Өс студент килде.
chv	Виҫӗ студент килнӗ.
crh	Üç student keldi.

Appendix

Table 18: Challenge sentences: Numeral and Converb.

Lang.	Round trip	Words	Word%	CER	Purity
aze	Latn $\to$ Cyrl $\to$ Latn	784	100.0	0.00	100.0
tuk	Latn $\to$ Cyrl $\to$ Latn	62	100.0	0.00	100.0
uzb	Latn $\to$ Cyrl $\to$ Latn	913	100.0	0.00	100.0
crh	Latn $\to$ Cyrl $\to$ Latn	822	97.6	0.37	100.0
kaa	Latn $\to$ Cyrl $\to$ Latn	272	100.0	0.00	99.7
kaz	Cyrl $\to$ Latn $\to$ Cyrl	65	100.0	0.00	100.0

Appendix

Table 19: Transliteration round-trip consistency. Word% : words recovered exactly after the round trip; CER : character error rate (%) of the round trip; Purity : share (%) of letters in the intermediate output that belong to the target script.

	Challenge		TueCL
Lang.	Full	Turkic	Full	Turkic
tur	100	100	100.0	100.0
aze	90	90	89.2	89.2
tuk	80	80	–	–
gag	10	10	–	–
kaz	100	100	–	–

Appendix

Table 20: Language identification accuracy (%) of GlotLID with its full label space (2,102 labels) and restricted to TurkicNLP’s Turkic labels (30 labels), on the 20 challenge sentences per language and the natural TueCL sentences (826 sentences in 9 languages; Sakha 3, Tatar 5, Kumyk 19, Karakalpak 35 sentences).

	Qwen3.5		LLaMA 3		OLMo 3		GPT-5.2
Lang.	Nat.	CTS	Nat.	CTS	Nat.	CTS	Nat.	CTS
kaz	2.86	2.28	3.51	2.32	4.05	2.55	2.37	2.29
kir	2.84	2.16	3.48	2.31	3.80	2.59	2.52	2.12
tat	3.13	2.37	4.13	2.59	4.46	2.71	2.51	2.40
bak	3.35	2.33	4.68	2.57	5.00	2.68	2.54	2.35
nog	2.92	2.02	3.08	2.08	3.48	2.27	2.65	1.98

Appendix

Table 21: Tokenizer fertility (tokens per word) on the 20 parallel challenge sentences in the native script (Nat.) and after conversion to the Common Turkic Alphabet (CTS) by TurkicNLP, for the 13 languages written in Cyrillic or Perso-Arabic script. The Latin-script group covers Turkish, Azerbaijani, Turkmen, Gagauz, Crimean Tatar, Karakalpak, and Uzbek.

			Neural		Apertium		Hybrid
Lang.	C	$n$	Ex.	BF1	Ex.	BF1	Ex.	BF1	Lab
tur	H	34	74	87	94	98	97	99	96
aze	H	30	67	76	23	21	70	80	65
tuk	M	30	67	80	87	95	93	97	88
gag	L	16	88	92	81	89	94	97	92
kaz	H	34	82	91	88	96	97	99	98

Appendix

Table 22: Morpheme segmentation per language (%). C : confidence of the gold annotation (High, Medium, Low). Ex. : exact segmentation; BF1 : morpheme-boundary F1; Lab : morphemes with correct span and label. All : over all 483 words. Bold: best exact score per language.

Category	$n$	Neural	Apertium	Hybrid
Plural	47	94	79	96
Case	92	91	83	97
Possessive	47	77	79	87
Pl.+poss.+case	25	84	80	92
Past + person	40	60	80	88
Present/aorist + person	24	50	63	83

Appendix

Table 23: Exact segmentation (%) per category over all 20 languages.

Category	Turkish	Kazakh	Uzbek	Sakha
Plural	kitap·lar	кітап·тар	kitob·lar	кинигэ·лэр
Case	ev·de	мектеп·ке	maktab·ga	оскуола·ҕа
Possessive	kapı·sı	қала·сы	bola·si	оҕо·то
Pl.+poss.+case	kitap·lar·ımız·dan	кітап·тар·ымыз·дан	kitob·lar·imiz·dan	кинигэ·лэр·бит
Past + person	gel·di·m	бар·ды·м	bor·di·m	бар·ды·м
Present + person	gid·iyor·um	бар·а·мын	bor·a·man	бар·а·бын

Appendix

Table 24: Example items of the morpheme segmentation test set (gold segmentation, morphemes separated by ·). The full set of 483 items in 20 languages is released with the code.

Morphological Analysis

See all 20 papers

TajikNLP: An Open-Source Toolkit for Comprehensive Text Processing of Tajik (Cyrillic Script)
May 6, 2026Mullosharif K. Arabov, Karomatullo Habibullozoda, Nurali ShirinovMorphosyntactic Tagging Tokenization
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia · Bokhtar State University named after Nosiri Khusrav Bokhtar, Tajikistan
Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages
Aug 10, 2026Zijie ZhangCross-Lingual Transfer
The Chinese University of Hong Kong, Shenzhen
Cross-Lingual Transfer for Machine Translation in Turkic Languages
Jul 31, 2026Omer Burak Cinar, Mehmet Mert Dalkilic, Cagri ToramanMultilingual Language Models Low-Resource MT
Middle East Technical University · Computer Engineering Department

May 6, 2026J/K move · Enter open · S save

TajikNLP: An Open-Source Toolkit for Comprehensive Text Processing of Tajik (Cyrillic Script)

Mullosharif K. Arabov, Karomatullo Habibullozoda, Nurali Shirinov

Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia · Bokhtar State University named after Nosiri Khusrav Bokhtar, Tajikistan

Morphosyntactic Tagging Tokenization Morphological Analysis Low-Resource Language Processing

Open paper AskPDF arXiv

The Tajik language, written in Cyrillic script, remains severely under-resourced in terms of publicly available natural language processing (NLP) toolkits, hindering both linguistic research and applied development. This paper introduces TajikNLP, an open-source Python library that provides the first comprehensive pipeline for processing authentic Tajik text while preserving the original Cyrillic orthography. The library implements a modular architecture centered around a unified Doc object, enabling sequential application of components for cleaning, normalization, tokenization (including subword BPE), morphemic segmentation, part-of-speech tagging, stemming, lemmatization, and sentence splitting. A novel unified morphology engine is introduced, offering controlled and deep analysis modes that significantly improve handling of Tajik's agglutinative nominal and verbal inflections. The release further incorporates a lexicon-based sentiment analyser and pre-trained Word2Vec/FastText embeddings loaded directly from the Hugging Face Hub. To ensure reproducibility and facilitate future research, four accompanying linguistic datasets -- a POS-tagged corpus (52.5k entries), a sentiment lexicon (3.5k entries), a toponym gazetteer (5.6k entries), and a personal names dataset (3.8k entries) -- have been openly published under permissive licenses. The library's reliability is validated by an extensive test suite of 616 automated tests achieving 93% source code coverage. TajikNLP thus establishes a foundational technological infrastructure for Tajik language processing, lowering the barrier to entry for both academic and industrial applications in low-resource Cyrillic-script environments.

Abstract

Morphological Analysis

Explore similar work

TajikNLP: An Open-Source Toolkit for Comprehensive Text Processing of Tajik (Cyrillic Script)

Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages

Cross-Lingual Transfer for Machine Translation in Turkic Languages