Adapting Qwen3 to Tajik from 0.6B to 8B: Lower Text Loss Did Not Mean Better Understanding
Organizations: Saidzoda AI Lab, Saidzoda Engineering Company, Dushanbe, Tajikistan
Abstract
Small LoRA adapters trained on Tajik news and books lowered Qwen3's Tajik text loss at every size we tried, from 0.6B to 8B, by 0.60 to 1.45 nats per token, but none of them improved Tajik reading comprehension where it can be measured. On Belebele the 1.7B and 8B adapters left accuracy unchanged and the 4B adapter lowered it by 3.7 points; at 0.6B the score rose by 4.2 points but sits near chance and coincides with a change in which answer letter the model prefers, so we do not read it as understanding. At 8B, continued pretraining on about 375 million tokens of raw Tajik text was followed by a Belebele gain of 9.9 points. After a further supervised fine-tuning stage, our main model, e3, is 11.4 points above Qwen3-8B on Belebele; its English MMLU changed by -1.2 points, and a loss larger than 3.3 points is excluded. The same heavy recipe applied to Qwen3-0.6B-Base brought its text loss close to that of the 8B model after books-only pretraining, while its multiple-choice answers stayed near chance and its English MMLU fell by 7.6 points. We report 15 trained models and their 5 starting models from June to October 2025, scored in 2026 on one harness with paired intervals, and we report what went wrong: lost training records, a thinking-mode dataset with almost no reasoning in it, later branches whose training data was never documented, and a 4-bit release that contains no transformer layers.
Figures & tables
| Light, small (June) | Light, 8B (June) | Heavy, 0.6B (Sep to Oct) | Heavy, 8B (Jul to Aug) | |
|---|---|---|---|---|
| Start model | Qwen3 0.6B, 1.7B, 4B | Qwen3-8B | Qwen3-0.6B-Base | Qwen3-8B |
| Text | news and books, 41,877 documents (0.6B: not recorded) | not recorded | 1,122,175 chat examples + raw text | raw text, then 1,227,427 chat examples |
| Tokens per epoch | about 21.4M | not recorded | about 375M (CPT stage) | about 375M (CPT stage) |
| Order | one adapter run per model | not recorded | SFT, SFT, CPT, SFT | CPT (4 epochs), then SFT (3) |
| Loss on | whole text | not recorded | answer only | whole text |
| Adapter | rank 8 to 16, attention only | rank 8, attention only | rank 64, all projections | rank 16, all projections |
| Model | Text loss | Belebele Tajik | English MMLU |
|---|---|---|---|
| (488 passages) | (900) | (1,003) | |
| Light recipe, June 2025 (attention-only LoRA, at most about 21.4M tokens per epoch) | |||
| Qwen3-0.6B | 4.25 | 23.1 | 41.5 |
| + light adapter (QwenTg-v5) | 2.80 ( 1.45) | 27.3 (+4.2 ∗ ) | 32.1 ( 9.4 ∗ ) |
| Qwen3-1.7B | 3.69 | 34.1 | 58.0 |
| + light adapter v1 | 2.40 ( 1.29) | 33.3 ( 0.8) | 54.7 ( 3.3 ∗ ) |
| Model | Belebele Tajik | English MMLU |
|---|---|---|
| Qwen3-0.6B | 95 | 38 |
| + light adapter (QwenTg-v5) | 48 | 45 |
| Qwen3-1.7B | 37 | 35 |
| + light adapter v1 | 46 | 32 |
| + light adapter v2 | 49 | 32 |
| Qwen3-4B | 36 | 28 |
LLM Fine-Tuning
LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.
See all 793 papers