Soro: A Lightweight Foundation Model and Chatbot for Tajik

Stanislav Liashkov; Haitz Sáez de Ocáriz Borde; Azizjon Azimi; Khushbakht Shoymardonov; Shuhratjon Khalilbekov; Bonu Boboeva

cs.AIApr 9, 2026

Soro: A Lightweight Foundation Model and Chatbot for Tajik

Authors: Stanislav Liashkov, Haitz Sáez de Ocáriz Borde, Azizjon Azimi, Khushbakht Shoymardonov, Shuhratjon Khalilbekov, Bonu Boboeva

Abstract

We present Soro, a family of Tajik-specialized conversational large language models (LLMs) designed for real-world deployment under tight compute and connectivity constraints in Tajikistan. Starting from open-weight Gemma 3 checkpoints, we perform Tajik-only continual pretraining on a curated 1.9-billion-token corpus spanning filtered web text, PDF documents, and curriculum-aligned educational materials, followed by supervised instruction tuning on 40K Tajik teacher-style examples. To enable rigorous evaluation despite the limited coverage of Tajik in standard benchmarks, we introduce a suite of Tajik benchmarks covering general knowledge, linguistic competence, and school- and university entrance-exam domains, and we open-source them on Hugging Face. Across these Tajik benchmarks, Soro substantially outperforms same-size Gemma 3 baselines while retaining strong English performance on standard datasets. We further show that FP8 and INT4 quantization of Soro preserves most Tajik-language gains while reducing memory requirements for edge deployment, supporting an ongoing education-sector pilot and planned scale-out across schools in Tajikistan.

Figures & tables

Language	Characters	Words	Avg Length	Samples	Source
English	81.0M	11.1M	3.87	20,000	FineWEB EDU sample
Tajik	106.1M	16.2M	5.29	20,000	Our Tajik corpus sample

Table 1: Dataset statistics for tokenization analysis.

Model	Vocab Size	Fertility (English)	Fertility (Tajik)
Llama 3.1-8B	128,000	1.634	2.798
Llama 3.2-3B-Instruct	128,000	1.634	2.798
Gemma 3-12B	262,144	1.776	2.380
Qwen 3-8B	151,643	1.753	2.860
Qwen 2.5-14B-Instruct-1M	151,643	1.753	2.860
Qwen 3-30B-A3B-Instruct-2507	151,643	1.753	2.860

Table 2: Fertility rates comparison across models and languages.

Model	Soro Lite Base (12B)	Soro Base (27B)
Starting model	Gemma 3-12B	Gemma 3-27B
Max sequence length	2048 tokens	2048 tokens
Learning rate	7e-5	5e-5
Learning rate scheduler	Cosine	Cosine
Training steps	11,000	10,000
Training epochs	$\sim$ 1	$\sim$ 0.89

Table 3: Hyperparameters and configuration for continual pretraining.

Panel of Figure 1 — Figure 1: Training and evaluation loss during continual pretraining of Soro Lite Base and Soro Base (based on Gemma 3 models).

Setting	Soro Lite	Soro
Starting model	Soro Lite Base	Soro Base
Sequence length	2048 tokens	2048 tokens
Learning rate	3e-5	1e-5
Learning rate scheduler	Cosine	Cosine
Training steps	10,000	9,000
Training epochs	$\sim$ 2	$\sim$ 1.8

Table 4: Hyperparameters and configuration for instruction tuning.

Panel of Figure 2 — Figure 2: Training and evaluation loss during supervised instruction tuning of Soro Lite and Soro.

Figure 3: Impact of the linear merging weight on mean accuracy across Tajik benchmarks. The optimal ratio for both model sizes is approximately 80% Soro / 20% Gemma-3-IT.

Benchmark	Source	Description	Samples
Tajik MMLU (§ A.7 )	Tajik translation of MMLU	General knowledge across 57 subjects (STEM, humanities, social sciences)	1,525
Tajik-FactQA (§ A.3 )	Centralized University Entrance Exams	History, culture, geography, and national institutions of Tajikistan	436
Tajik History (§ A.4 )	School textbooks (grades 5–11)	National history curriculum from ancient civilizations to post-independence	1,400
Tajik Literature (§ A.6 )	School textbooks (grades 5–11)	Classical and contemporary Tajik/Persian literature	698
TajLib (§ A.5 )	National Testing Center Entrance Exams (2025)	Linguistic competence: phonetics, grammar, lexicology, orthography	919
Tajik Curated (§ A.2 )	Diverse Tajik sources	Multi-domain: History, Biology, Literature, Law, Geography	2,266

Table 5: Overview of Tajik-specific evaluation benchmarks. All benchmarks use a four-option multiple-choice format (A/B/C/D).

Panel of Figure 4 — Figure 4: Average accuracy on Tajik benchmarks across open-source models and full-precision Soro variants. Quantized results are in Section 7.1 .

Size	Gemma-3-IT	Soro pre-merge	Soro	Soro FP8	Soro GPTQ INT4
12B	50.4	56.4	58.2	57.7	56.5
27B	55.8	62.3	63.8	63.9	62.3

Table 6: Average accuracy on Tajik benchmarks (%). Default precision is BF16. Gemma-3-IT and full-precision Soro columns are included as reference baselines.

Panel of Figure 5 — Figure 5: Average accuracy on Tajik benchmarks for quantized Soro variants compared to full-precision Soro and Gemma 3 baselines. FP8 quantization preserves near-identical performance while halving memory; GPTQ INT4 achieves 4 $\times$ compression with modest accuracy trade-offs. Full results across all models are reported in Figure 4 .

Model Variant	Precision	Memory (GB)	BF16 Baseline (GB)
Soro Lite FP8	FP8	13.0	24.0
Soro Lite GPTQ INT4	INT4	6.6	24.0
Soro FP8	FP8	29.0	54.0
Soro GPTQ INT4	INT4	13.5	54.0

Table 7: Model memory footprint for quantized variants.

Panel of Figure 6 — Figure 6: Geographic distribution of Soro deployments. (a) Initial 30-school pilot in Dushanbe; (b) Current national expansion to 100 schools across Dushanbe (61), Khujand (17), Vahdat (16), Kulob (3), and Khorog (3). The long-term NAIS-2040 target is 4,000+ schools by the 2027–28 academic year.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: MMLU Benchmark

Panel of Figure 8 — Figure 8: Tajik Curated Benchmark

Panel of Figure 9 — Figure 9: Tajik Factual Benchmark

Panel of Figure 10 — Figure 10: Tajik History high-school level benchmark

Panel of Figure 11 — Figure 11: Tajik Linguistic Benchmark (TajLib)

Panel of Figure 12 — Figure 12: Tajik Literature Benchmark

Panel of Figure 13 — Figure 13: Tajik MMLU Benchmark

Figure 14: Raw format prompt structure for English evaluation.

Figure 15: Raw format prompt structure for Tajik evaluation.

Figure 16: Chat format prompt structure for English evaluation.

Figure 17: Chat format prompt structure for Tajik evaluation.

Survey Item	Mean	Std Dev	Median	Favorable (%)
Response accuracy	4.15	0.99	4.0	75.5
Tajik language quality	4.47	0.82	5.0	86.8
Lesson prep usefulness	4.15	1.06	5.0	77.4
Ease of use	4.34	1.02	5.0	81.1
Willingness to adopt	4.11	1.15	5.0	73.6
Recommend to colleagues	4.42	0.84	5.0	84.9

Appendix

Table 8: Descriptive statistics for the teacher feedback survey (

n=53

). Scores range from 1 (lowest) to 5 (highest). “Favorable” denotes the percentage of responses rated 4 or 5.

Panel of Figure 18 — Figure 18: Mean teacher ratings across six survey dimensions ( $n=53$ ). Error bars indicate one standard deviation.

Panel of Figure 19 — Figure 19: Distribution of teacher ratings across survey dimensions ( $n=53$ ). Each bar shows the percentage breakdown by score.

Explore similar work

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus
May 5, 2026Mullosharaf K. ArabovLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
SozKZ: Training Efficient Small Language Models for Kazakh from Scratch
Mar 21, 2026Saken TukenovLanguage Model Pretraining Small Language Models
Independent Researcher
Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language
Feb 17, 2026Prathamesh Devadiga, Paras ChopraLow-Resource Language Processing Structured Prompting
Lossfunk

May 5, 2026J/K move · Enter open · S save

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

Mullosharaf K. Arabov

Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia

LLM Fine-Tuning Low-Resource Language Processing Text Generation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

This paper is devoted to the adaptation of generative large language models for the Tajik language, a low-resource language with Cyrillic script. To overcome the shortage of digital text resources, the author created and publicly released the Tajik Web Corpus, the largest open-access corpus of Tajik, comprising 319,298 documents (~1.11 billion characters). On a subsample of 10,000 documents, 17 configurations were benchmarked, covering autoregressive, encoder-decoder, and encoder-only models with three fine-tuning strategies: full fine-tuning, LoRA, and QLoRA (ranks 8 and 16). Quality was assessed via perplexity and cross-entropy loss; peak GPU memory and training time were also recorded. Best results were achieved by Mistral 7B with QLoRA (r=16): mean perplexity 5.03, standard deviation 0.03. Increasing rank from 8 to 16 gave statistically insignificant improvement while raising memory consumption. For small GPT-2 family models, full fine-tuning yielded lower perplexity (3.48 for GPT-2 Medium) than LoRA (7.60-8.42), but induced catastrophic forgetting. The encoder-only XLM-RoBERTa showed the worst results (perplexity 59.3). The novelty lies in creating the largest verified Tajik corpus and the first systematic analysis of PEFT effectiveness for Tajik text generation. Practical value lies in recommendations for architecture and fine-tuning strategy selection, optimizing computational costs without substantial quality loss.

Abstract

Explore similar work

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language