A Corpus-Driven SentencePiece Unigram Tokenizer for Sindhi: Design and Benchmark Against Fourteen Existing Tokenizers
Organizations: Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan
Abstract
Sindhi is a low-resource Indo-Aryan language written mainly in an extended Perso-Arabic script. Several Sindhi tokenizers have been released on public model hubs, but they have not been compared with one another on shared data. We train SentencePiece Unigram tokenizers with byte fallback at four vocabulary sizes (32k to 150k) on about 1.10 million Sindhi sentences and a smaller, repeated Urdu, Persian and Arabic slice, from four public sources, and compare them with fourteen existing tokenizers on four evaluation sets: FLORES+, Sindhi Wikipedia, a news-derived sentiment corpus, and a contemporary news corpus. The metrics are fertility, single-token retention, UNK rate, byte-fallback rate, and round-trip character loss. At 64k, our tokenizer averages 1.147 tokens per word, against 1.208 for the best existing tokenizer and 1.512 for XLM-RoBERTa, and keeps 88.6% of words as single tokens. On fertility, single-token retention and UNK rate it is best or tied for best in all four evaluation sets at 64k and above; at 32k it trails one baseline on FLORES+. Margins over the best Sindhi-adapted baselines are small (0.002 fertility on FLORES+ at 64k), and we report no significance tests. A round-trip check finds no character loss for our tokenizer and silent character deletion (up to 0.85% of characters) in two existing ones; seven others lose less than 0.1%. We evaluate intrinsic metrics only, so effects on downstream tasks remain untested. Code and tokenizer files are public (DOI: 10.5281/zenodo.23962474).
Figures & tables
| Tokenizer | Group | Vocabulary | Reference |
|---|---|---|---|
| SindhiLM v1 | Sindhi-specific (initial) | 43 | ( aakashMeghwar01, n.d.a ) |
| SindhiLM v2 | Sindhi-specific (initial) | 151,643 | ( aakashMeghwar01, n.d.b ) |
| SindhiLM v3 | Sindhi-specific (initial) | 32,000 | ( aakashMeghwar01, n.d.c ) |
| FahadQazi BPE | Sindhi-specific (initial) | 13,031 | ( fahadqazi, n.d. ) |
| XLM-RoBERTa | General multilingual | 250,002 | ( Conneau et al., 2020 ) |
| mT5 | General multilingual | 250,100 | ( Xue et al., 2021 ) |
| Source | Content used | License |
|---|---|---|
| Sindhi Wikipedia ( Wikimedia Foundation, n.d. ) | 6,000 articles (articles 2,001 to 8,000 of a seeded shuffle; the first 2,000 are held out for evaluation) | CC BY-SA |
| Sindhi-NLP ( Sindhi-NLP, n.d. ) | Two Twitter accounts, Awami Awaz articles, Daily Kawish articles | MIT (repository) |
| shaikhsalman/sindhi-llm-corpus ( shaikhsalman, n.d. ) | 20,000-row sample (seed 7), no language filter; about 57% of the Sindhi-side sentences | None stated |
| Urdu, Persian and Arabic Wikipedia ( Wikimedia Foundation, n.d. ) | 1,500 articles each, for shared script and loanwords; repeated four times | CC BY-SA |
| Setting | Value |
|---|---|
| model_type | unigram |
| vocab_size | 32,000; 64,000 (default); 100,000; 150,000 |
| character_coverage | 1.0 |
| byte_fallback | true |
| split_by_whitespace | true |
| unk_id , bos_id , eos_id , pad_id | 0, 1, 2, disabled (−1) |
| Set | Source | Sentences | Words | Type |
|---|---|---|---|---|
| FLORES+ | snd_Arab, devtest ( NLLB Team et al., 2022 ) | 1,012 | 25,356 | Translated benchmark |
| Wikipedia | 20231101.sd, first 2,000 articles, 3 to 100 words ( Wikimedia Foundation, n.d. ) | 25,480 | 435,947 | Encyclopedic |
| Sentiment | Sindhi_Sentiment_dataset ( alinawazmahar, n.d. ) | 3,536 | 47,156 | News-derived |
| Sindh Line | Sindh Line Publishers ( Mozilla Data Collective, 2026 ) | 87,158 | 1,014,998 | News |
| Tokenizer | Vocab | Fertility | Single-token % | UNK % | Byte-fallback % | Max loss % |
|---|---|---|---|---|---|---|
| Ours 150k | 150,000 | 1.125 | 89.7 | 0.000 | 0.154 | 0.0000 |
| Ours 100k | 100,000 | 1.133 | 89.3 | 0.000 | 0.149 | 0.0000 |
| Ours 64k | 64,000 | 1.147 | 88.6 | 0.000 | 0.138 | 0.0000 |
| Ours 32k | 32,000 | 1.184 | 86.5 | 0.000 | 0.113 | 0.0000 |
| TinyLlama-Sindhi † | 42,681 | 1.208 | 84.4 | 0.000 | 0.000 | 0.4147 |
| hellosindh-NER | 32,001 | 1.218 | 83.7 | 0.194 | 0.197 | 0.0751 |
| Set | Fertility: Ours | Best baseline | Diff. | Single-token %: Ours | Best baseline | Diff. |
|---|---|---|---|---|---|---|
| FLORES+ | 1.191 | 1.193 (TinyLlama-Sindhi) | 0.002 | 84.7 | 84.4 (TinyLlama-Sindhi) | +0.3 |
| Wikipedia | 1.194 | 1.272 (hellosindh-NER) | 0.078 | 86.9 | 82.6 (hellosindh-NER) | +4.3 |
| Sentiment | 1.110 | 1.149 (TinyLlama-Sindhi) | 0.039 | 90.3 | 87.6 (TinyLlama-Sindhi) | +2.7 |
| Sindh Line | 1.093 | 1.155 (hellosindh-NER) | 0.062 | 92.4 | 87.9 (hellosindh-NER) | +4.5 |
| Mean | 1.147 | 1.208 (TinyLlama-Sindhi) | 0.061 | 88.6 | 84.4 (TinyLlama-Sindhi) | +4.2 |
| Tokenizer | Vocab | Fertility | Single-token % | UNK % |
|---|---|---|---|---|
| Ours 32k | 32,000 | 1.184 | 86.5 | 0.000 |
| hellosindh-NER | 32,001 | 1.218 | 83.7 | 0.194 |
| SindhiLM v3 | 32,000 | 1.730 | 55.8 | 3.605 |
| hellosindh-BERT | 32,000 | 1.960 | 35.5 | 0.122 |
| TinyLlama-Sindhi (larger vocabulary) | 42,681 | 1.208 | 84.4 | 0.000 |
| Tokenizer | Max loss % | Set | Max sentences affected % | Set |
|---|---|---|---|---|
| FahadQazi BPE | 0.8461 | Wikipedia | 16.15 | Wikipedia |
| TinyLlama-Sindhi | 0.4147 | FLORES+ | 24.90 | FLORES+ |
| SindhiLM v1 | 0.0790 | Wikipedia | 0.69 | Wikipedia |
| SindhiLM v3 | 0.0754 | Wikipedia | 1.13 | Wikipedia |
| hellosindh-NER | 0.0751 | Wikipedia | 0.49 | Wikipedia |
| hellosindh-BERT | 0.0089 | Wikipedia | 0.09 | Wikipedia |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| FLORES+ | ||||
|---|---|---|---|---|
| Tokenizer | Fert. | Single % | UNK % | Byte-fb. % |
| Ours 150k | 1.167 | 85.9 | 0.000 | 0.199 |
| Ours 100k | 1.176 | 85.5 | 0.000 | 0.198 |
| Ours 64k | 1.191 | 84.7 | 0.000 | 0.172 |
| Ours 32k | 1.233 | 82.4 | 0.000 | 0.099 |
| TinyLlama-Sindhi | 1.193 | 84.4 | 0.000 | 0.000 |
| Tokenizer | FLORES+ loss | FLORES+ aff. | Wiki loss | Wiki aff. | Sent. loss | Sent. aff. | Sindh Line loss | Sindh Line aff. |
|---|---|---|---|---|---|---|---|---|
| Ours (all sizes) | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 |
| Qwen2.5 / SindhiLM v2 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 |
| BLOOM | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 |
| Gemma 3 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 |
| Llama 3.2 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 | 0.0000 | 0.00 |
| XLM-RoBERTa | 0.0000 | 0.00 | 0.0040 | 0.04 | 0.0000 | 0.00 | 0.0030 | 0.02 |
| Category | Word | Best custom | Best competitor | Result |
|---|---|---|---|---|
| Genitive/possessive | دنيا جو | 32k (2) | 2 (5 tokenizers) | Tie |
| Genitive/possessive | خدا جو | 32k (2) | 2 (6 tokenizers) | Tie |
| Genitive/possessive | پاڻي جي | 32k (2) | 2 (5 tokenizers) | Tie |
| Genitive/possessive | ماڻهن جو | 32k (2) | 2 (5 tokenizers) | Tie |
| Inflectional suffix | ڪتابن | 32k (1) | 1 (5 tokenizers) | Tie |
| Inflectional suffix | شهرن | 32k (1) | 1 (3 tokenizers) | Tie |