Tokenization

Latest papers

44 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

TajikNLP: An Open-Source Toolkit for Comprehensive Text Processing of Tajik (Cyrillic Script)
May 6, 2026Mullosharif K. Arabov, Karomatullo Habibullozoda, Nurali ShirinovMorphosyntactic Tagging Tokenization
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia · Bokhtar State University named after Nosiri Khusrav Bokhtar, Tajikistan
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
Apr 18, 2026Disen Liao, Freda ShiLLM Fine-Tuning Language Modeling
University of Waterloo · Vector Institute
From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution
Apr 15, 2026Pavel Chizhov, Egor Bogomolov, Ivan P. YamshchikovTokenization Tokenization in Language Models
CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · JetBrains Research · TU Delft
A Triadic Suffix Tokenization Scheme for Numerical Reasoning
Apr 13, 2026Olga ChetverinaTokenization Subword Tokenization
Lecturer at MIPT, Moscow, Russia.
HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval
Apr 12, 2026Senol GulgonulTokenization Information Retrieval
Electrical and Electronics Engineering, Ostim Technical University, Ankara Turkey
Faster Superword Tokenization
Apr 6, 2026Craig W. Schmidt, Chris Tanner, Yuval PinterTokenization Subword Tokenization
Kensho Technologies, Cambridge, MA 02138, USA · MIT, Cambridge, MA, USA · Institute for Applied AI Research, Ben-Gurion University of the Negev, Beer Sheva 8410501, Israel
Separate Before You Compress: The WWHO Tokenization Architecture
Mar 26, 2026Kusal DarshanaTokenization Subword Tokenization
Remeinium Research
Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
Mar 25, 2026Ben GublerTokenization in Language Models Tokenization
VerChol -- Grammar-First Tokenization for Agglutinative Languages
Mar 6, 2026Prabhu RajaTokenization in Language Models Tokenization
Bengaluru, India
Frequency-Ordered Tokenization for Better Text Compression
Feb 26, 2026Maximilian KalcherLossless Data Compression Tokenization
You Can Learn Tokenization End-to-End with Reinforcement Learning
Feb 15, 2026Sam Dauncey, Roger WattenhoferSubword Tokenization LLM Training
Department of Electrical Engineering, ETH Zurich, Zurich, Switzerland.
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
Aug 6, 2025Negar Foroutan, Clara Meister, Debjit Paul +4Low-Resource Language Processing Subword Tokenization
EPFL · University of Zurich
Less Is More: Reducing Token Counts Without Compromising Performance
Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference Acceleration Token Efficiency
Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9LLM Interpretability Distributional Semantics
Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3

May 6, 2026J/K move · Enter open · S save

TajikNLP: An Open-Source Toolkit for Comprehensive Text Processing of Tajik (Cyrillic Script)

Mullosharif K. Arabov, Karomatullo Habibullozoda, Nurali Shirinov

Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia · Bokhtar State University named after Nosiri Khusrav Bokhtar, Tajikistan

Morphosyntactic Tagging Tokenization Morphological Analysis Low-Resource Language Processing

Open paper AskPDF arXiv

The Tajik language, written in Cyrillic script, remains severely under-resourced in terms of publicly available natural language processing (NLP) toolkits, hindering both linguistic research and applied development. This paper introduces TajikNLP, an open-source Python library that provides the first comprehensive pipeline for processing authentic Tajik text while preserving the original Cyrillic orthography. The library implements a modular architecture centered around a unified Doc object, enabling sequential application of components for cleaning, normalization, tokenization (including subword BPE), morphemic segmentation, part-of-speech tagging, stemming, lemmatization, and sentence splitting. A novel unified morphology engine is introduced, offering controlled and deep analysis modes that significantly improve handling of Tajik's agglutinative nominal and verbal inflections. The release further incorporates a lexicon-based sentiment analyser and pre-trained Word2Vec/FastText embeddings loaded directly from the Hugging Face Hub. To ensure reproducibility and facilitate future research, four accompanying linguistic datasets -- a POS-tagged corpus (52.5k entries), a sentiment lexicon (3.5k entries), a toponym gazetteer (5.6k entries), and a personal names dataset (3.8k entries) -- have been openly published under permissive licenses. The library's reliability is validated by an extensive test suite of 616 automated tests achieving 93% source code coverage. TajikNLP thus establishes a foundational technological infrastructure for Tajik language processing, lowering the barrier to entry for both academic and industrial applications in low-resource Cyrillic-script environments.