Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language

Mullosharaf K. Arabov; Svetlana S. Khaybullina

cs.CLOct 22, 2026

Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language

Authors: Mullosharaf K. Arabov, Svetlana S. Khaybullina

Organizations: Department of Data Analysis and Programming Technologies, Institute of Computational Mathematics and Information Technologies, Kazan (Volga Region) Federal University, Kazan, Russia

Abstract

Bashkir is a low-resource agglutinative language written in Cyrillic and remains poorly represented in modern LLMs. We present an open resource suite and a systematic benchmark of adaptation methods: the Bashkir Web Corpus (71,567 documents, 46.9M tokens, 16 sources) and three labelled datasets for binary, multiclass and multilabel text classification; the benchmark covers the first two. Experiments use eight architectures (82M-8B: DistilGPT2, GPT-2 base/medium, Phi-3-mini, Mistral-7B, YandexGPT-5-Lite-8B, mBERT, XLM-R) and three adaptation groups (full fine-tuning, QLoRA, in-context learning), plus continual pretraining (CPT). (1) QLoRA on 3.8-8B models outperforms encoder-only fine-tuning on binary classification: YandexGPT-5-Lite-8B with CPT reaches macro-F1 0.748 vs 0.653 (XLM-R) and 0.650 (mBERT); pairwise differences are significant after Holm correction (p<0.001). (2) YandexGPT-5-Lite-8B significantly outperforms Mistral-7B and Phi-3-mini on multiclass classification, BPB and native-speaker ratings (mean 3.28 vs 2.52 and 2.22; n=100; alpha=0.69-0.77; p<0.05); on binary classification its QLoRA advantage is not significant (p>=0.084). (3) CPT effects depend on model and task: perplexity improves for Phi-3-mini (+0.58) and YandexGPT-5-Lite-8B (+0.33) but worsens for Mistral-7B (-1.15, p<0.001); on binary classification CPT helps Mistral-7B most (+4.0 macro-F1). (4) Tokeniser fertility on Bashkir varies 2.67x (2.92-7.80) and tracks adaptation quality, while perplexity across tokenisers is not comparable. Cyrillic tokenisers with Russian pretraining are preferable; LLM judges reproduce native ratings for grammar and coherence (rho=0.33-0.46) but not language correctness (rho=0.16, p=0.11); native evaluation remains necessary. All artefacts are open-licensed.

Figures & tables

Group	lr	Epochs	Batch size	$r$
Full FT (step1)	$2\times 10^{-4}$	3	8 / 4	—
QLoRA (step3)	$10^{-4}$	2	2 / 1	16
CPT (step4)	$10^{-5}$	1	1	16
Downstream QLoRA (step4)	$10^{-4}$	2	1	16
QLoRA CLS (step6)	$10^{-4}$	2	2 / 1	8
Encoder full FT (step6)	$2\times 10^{-5}$	3	16	—

Table 1: Hyperparameters by experiment group

Model	Method	PPL	BPB	Target	Trainable	Peak
				tokens (ba)	parameters, %	memory, MB
GPT-2 medium	Full FT	2.33	0.784	0.271	100	—
GPT-2 base	Full FT	2.49	0.844	0.532	100	—
DistilGPT2	Full FT	2.69	0.917	0.566	100	—
Mistral-7B	QLoRA	2.90	0.549	0.598	1.10	4693
YandexGPT-5-Lite-8B	QLoRA	3.29	0.462	0.601	0.91	6543

Table 2: Language modelling: perplexity and bits-per-byte on the test set of the corpus (128 tokens). For Full FT, all parameters; for QLoRA, the proportion of trainable parameters and peak GPU memory

Comparison ( $a$ vs $b$ )	$\Delta$ PPL [95% CI]	$p_{\text{PPL}}$	$\Delta$ BPB [95% CI]	$p_{\text{BPB}}$	Cohen’s $d$	Cliff’s $\delta$
Mistral-7B vs Phi-3-mini	$+1.226$ $[+1.137,+1.319]$	$<0.001$	$+0.155$ $[+0.150,+0.161]$	$<0.001$	$+0.43$	$-0.25$
Mistral-7B vs YandexGPT-5-Lite-8B	$+0.394$ $[+0.359,+0.431]$	$<0.001$	$-0.087$ $[-0.091,-0.082]$	$<0.001$	$+0.17$	$-0.10$
Phi-3-mini vs YandexGPT-5-Lite-8B	$-0.832$ $[-0.901,-0.760]$	$<0.001$	$-0.242$ $[-0.250,-0.233]$	$<0.001$	$-0.23$	$+0.16$

Table 3: Pairwise comparisons of QLoRA models on perplexity and BPB (test set of the corpus,

n=2000

examples).

\Delta=\text{metric}_{b}-\text{metric}_{a}

; a negative

\Delta

means that model

b

is better.

p_{\text{Holm}}

is the Holm–Bonferroni correction for 6 comparisons

Model	English (WikiText-2)		Russian (Wikipedia)
	PPL ${}_{\text{base}}\to$ after	BPB ${}_{\text{base}}\to$ after	PPL ${}_{\text{base}}\to$ after	BPB ${}_{\text{base}}\to$ after
GPT-2 medium	39.58 $\to$ 1504.67	1.162 $\to$ 2.310	9.11 $\to$ 14.45	1.854 $\to$ 2.241
GPT-2 base	54.49 $\to$ 4121.17	1.263 $\to$ 2.630	11.07 $\to$ 14.14	2.018 $\to$ 2.223
DistilGPT2	78.05 $\to$ 2963.08	1.377 $\to$ 2.526	12.66 $\to$ 14.87	2.131 $\to$ 2.266

Table 4: Catastrophic forgetting: perplexity and BPB on English and Russian before and after full fine-tuning on Bashkir (

n=200

texts per language, average over two seeds)

Model	PPL ${}_{\text{no-CPT}}$	PPL ${}_{\text{CPT+QLoRA}}$	$\Delta$ PPL	$p_{\text{Holm}}$	Cliff’s $\delta$
Mistral-7B	20.48	21.63	$-1.15$	$<0.001$	large
Phi-3-mini	35.81	35.23	$+0.58$	$<0.001$	large
YandexGPT-5-Lite-8B	12.70	12.37	$+0.33$	$<0.001$	large

Table 5: Effect of continual pretraining on the perplexity of the downstream-adapted model. PPL is the aggregate

\exp(\sum\text{NLL}/\sum\text{tokens})

; Cliff’s

\delta

is at the per-example level

Model	Method	macro-F1	weighted-F1	Accuracy	$n_{\text{seed}}$
Full fine-tuning (small models)
GPT-2 medium	Full FT	0.699	0.700	0.702	2
GPT-2 base	Full FT	0.698	0.697	0.700	2
DistilGPT2	Full FT	0.693	0.693	0.696	2
QLoRA without CPT (step6, $r=8$ )
YandexGPT-5-Lite-8B	QLoRA	0.736	0.736	0.737	1

Table 6: Binary classification (news vs analysis): macro-F1, weighted-F1 and accuracy. QLoRA in step6 uses rank

r=8

; QLoRA in step4 (with and without CPT) uses rank

r=16

Comparison ( $a$ vs $b$ )	acc a	acc b	$\Delta$ [95% CI]	$p_{\text{Holm}}$
Yandex vs Mistral	0.737	0.713	$-0.024$ $[-0.050,+0.001]$	$0.183$
Yandex vs Phi-3	0.737	0.702	$-0.035$ $[-0.064,-0.004]$	$0.084$
Yandex vs mBERT	0.737	0.652	$-0.085$ $[-0.117,-0.053]$	$<0.001$
Yandex vs XLM-R	0.737	0.659	$-0.078$ $[-0.109,-0.047]$	$<0.001$
Yandex vs majority	0.737	0.520	$-0.217$ $[-0.260,-0.171]$	$<0.001$

Table 7: Binary classification: key pairwise comparisons on accuracy (paired bootstrap, McNemar, Holm correction for 15 comparisons).

\Delta=\text{acc}_{b}-\text{acc}_{a}

; a positive

\Delta

means that model

b

is better

Model	Method	macro-F1	weighted-F1	Accuracy	$n_{\text{seed}}$
QLoRA (medium-scale generative models)
YandexGPT-5-Lite-8B	QLoRA (step6)	0.368	0.570	0.606	1
YandexGPT-5-Lite-8B	QLoRA no CPT (step4)	0.332	0.545	0.573	1
YandexGPT-5-Lite-8B	CPT + QLoRA (step4)	0.366	0.567	0.595	1
Mistral-7B	QLoRA (step6)	0.292	0.510	0.547	1
Mistral-7B	QLoRA no CPT (step4)	0.298	0.512	0.542	1

Table 8: Multiclass classification (19 categories): macro-F1, weighted-F1 and accuracy. QLoRA in step6 uses rank

r=8

; QLoRA in step4 (with and without CPT) uses rank

r=16

Comparison ( $a$ vs $b$ )	acc a	acc b	$\Delta$ [95% CI]	$p_{\text{Holm}}$
Yandex vs Mistral	0.606	0.547	$-0.059$ $[-0.087,-0.031]$	$<0.001$
Yandex vs Phi-3	0.606	0.506	$-0.100$ $[-0.128,-0.072]$	$<0.001$
Mistral vs Phi-3	0.547	0.506	$-0.041$ $[-0.069,-0.013]$	$0.030$
XLM-R vs majority	0.470	0.470	$+0.000$ $[+0.000,+0.000]$	$1.000$
Phi-3 vs mBERT	0.506	0.498	$-0.008$ $[-0.031,+0.015]$	$1.000$

Table 9: Multiclass classification: key pairwise comparisons on accuracy (paired bootstrap, McNemar, Holm correction for 15 comparisons).

\Delta=\text{acc}_{b}-\text{acc}_{a}

; a positive

\Delta

means that model

b

is better

	Bashkir (ba)			Russian (ru)			English (en)
Model	0	4	8	0	4	8	0	4	8
YandexGPT-5-Lite-8B	0.580	0.579	0.634	0.657	0.640	0.565	0.679	0.583	0.627
Mistral-7B	0.360	0.315	0.421	0.482	0.414	0.452	0.411	0.558	0.397
Phi-3-mini	0.513	0.444	0.411	0.506	0.457	0.516	0.400	0.574	0.650

Table 10: ICL on the binary task: macro-F1 by instruction language and number of examples in the prompt

Model	Language	$K$	$\text{Acc}_{0}$	$\text{Acc}_{K}$	$\Delta$ [95% CI]	$p_{\text{McNemar}}$	$p_{\text{Holm}}$
Mistral-7B	ba	4	0.520	0.460	$-0.060$ $[-0.260,+0.130]$	$0.610$	$1.000$
	ba	8	0.520	0.440	$-0.080$ $[-0.240,+0.080]$	$0.396$	$1.000$
	en	4	0.530	0.590	$+0.060$ $[-0.110,+0.230]$	$0.561$	$1.000$
	en	8	0.530	0.530	$+0.000$ $[-0.060,+0.060]$	$1.000$	$1.000$
	ru	4	0.510	0.470	$-0.040$ $[-0.150,+0.060]$	$0.572$	$1.000$
	ru	8	0.510	0.540	$+0.030$ $[-0.130,+0.190]$	$0.804$	$1.000$

Table 11: Effect of few-shot demonstrations: difference in accuracy versus zero-shot, 95% bootstrap CI and McNemar

p

(binary task,

n=100

test examples per language).

p_{\text{Holm}}

is the Holm–Bonferroni correction for 18 comparisons. No comparison is significant after correction.

Model	Grammar	Coherence	Language	Wins	Losses	Ties
YandexGPT-5-Lite-8B	2.89	2.39	2.56	16	0	20
Mistral-7B	2.83	2.44	2.58	18	0	16
Phi-3-mini	2.75	2.28	2.36	12	5	17
GPT-2 base	2.37	2.19	2.66	1	23	14
DistilGPT2	2.35	2.16	2.56	9	4	21
GPT-2 medium	2.13	1.89	2.16	1	25	12

Table 12: Evaluation of generation by LLM judges: direct scoring (mean over three judges, 1–5 scale) and pairwise win rate

Model	$n$	Grammar	Coherence	Language	Mean
YandexGPT-5-Lite-8B	11	2.94 [2.70, 3.15]	3.09 [2.76, 3.42]	3.82 [3.61, 3.97]	3.28
Mistral-7B	11	2.12 [1.97, 2.30]	2.18 [1.94, 2.48]	3.24 [3.00, 3.48]	2.52
Phi-3-mini	9	1.81 [1.56, 2.07]	1.81 [1.48, 2.11]	3.04 [2.85, 3.22]	2.22
DistilGPT2	23	1.48 [1.28, 1.70]	1.39 [1.22, 1.58]	2.39 [2.16, 2.62]	1.75
GPT-2 base	23	1.39 [1.20, 1.59]	1.30 [1.14, 1.51]	2.33 [2.16, 2.52]	1.68
GPT-2 medium	23	1.16 [1.04, 1.30]	1.22 [1.10, 1.35]	1.70 [1.45, 1.94]	1.36

Table 13: Native Bashkir speaker ratings (mean over three annotators, 1–5 scale, 95% bootstrap CI)

Criterion	$H$	$p$	$\varepsilon^{2}$	Significant pairs
Grammar	50.81	$9.4\times 10^{-10}$	0.487	9/15
Coherence	49.58	$1.7\times 10^{-9}$	0.474	9/15
Language	61.18	$6.9\times 10^{-12}$	0.598	13/15

Table 14: Kruskal–Wallis (omnibus) test and the number of significant pairwise comparisons (Mann–Whitney, Holm correction, family

=15

pairs per criterion)

Model	Fertility (ba)	Fertility (en)	Fertility (tr)
GPT-2 family (DistilGPT2, base, medium)	7.80	1.36	3.58
Phi-2 (2.7B)	7.80	1.36	3.58
Mistral-7B-v0.3	4.81	1.32	3.75
Qwen2.5-7B	4.23	1.37	3.00
YandexGPT-5-Lite-8B	3.80	1.26	3.22
T-lite-2.1	2.92	1.37	3.03

Table 15: Fertility rate (subtokens per word) on the Bashkir, English and Turkish test sets

Model	MorphAcc	OOV rate
Mistral-7B-v0.3	0.093	0.065
YandexGPT-5-Lite-8B	0.086	0.037
GPT-2 family (DistilGPT2, base, medium)	0.057	0.043
Phi-2 (2.7B)	0.057	0.043
Qwen2.5-7B	0.047	0.017
T-lite-2.1	0.047	0.017

Table 16: Morphological Boundary Accuracy and OOV rate on the Bashkir test set. MorphAcc is the proportion of subtokens whose right boundary coincides with a morpheme boundary; OOV rate is the proportion of word forms split into individual bytes or characters

Metric	$\eta^{2}_{\text{model}}$	$\eta^{2}_{\text{method}}$	$\eta^{2}_{\text{seed}}$	$n$
PPL	0.54	0.73	0.13	15
BPB	0.36	0.72	0.05	15
Proportion of target tokens	0.81	0.27	0.08	9

Table 17: Decomposition of variance (

\eta^{2}

) for the key metrics

Low-Resource Language Processing

Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.

See all 565 papers

May 6, 2026

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

This paper presents a comparative study of parameter-efficient fine-tuning (PEFT) methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language, a low-resource agglutinative language of the Turkic family. Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures: DistilGPT2, GPT-2 (base, medium), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To improve the reliability of results, each configuration was trained with three different random seeds. The lowest perplexity on the test set was obtained for GPT-2 medium with full fine-tuning (3.34). Meanwhile, QLoRA applied to Mistral-7B (3.79) and Phi-2 (3.81) achieved comparable quality with over 40 times fewer trainable parameters. However, we also observed cases of significant quality degradation when using PEFT for certain architectures (e.g., DeepSeek-7B with rank 8, perplexity = 129.55), indicating that the outcome depends critically on the choice of the base model and its tokenizer. Additionally, a qualitative analysis of generated texts based on Bashkir prompts revealed that models with the best perplexity do not necessarily produce the most coherent outputs: QLoRA-tuned models generated monolingual Bashkir continuations, whereas the fully fine-tuned model with the lowest perplexity frequently switched to English. The results suggest that QLoRA on 7B-scale models offers an effective compromise between quality and computational cost for Bashkir. To ensure reproducibility, open data, code, and trained adapters will be released upon acceptance.

Mullosharaf K. Arabov, Svetlana S. Khaybullina

Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia

LLM Fine-Tuning Low-Resource Language Processing Parameter-Efficient Fine-Tuning

Ask PDF

May 5, 2026

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

This paper is devoted to the adaptation of generative large language models for the Tajik language, a low-resource language with Cyrillic script. To overcome the shortage of digital text resources, the author created and publicly released the Tajik Web Corpus, the largest open-access corpus of Tajik, comprising 319,298 documents (~1.11 billion characters). On a subsample of 10,000 documents, 17 configurations were benchmarked, covering autoregressive, encoder-decoder, and encoder-only models with three fine-tuning strategies: full fine-tuning, LoRA, and QLoRA (ranks 8 and 16). Quality was assessed via perplexity and cross-entropy loss; peak GPU memory and training time were also recorded. Best results were achieved by Mistral 7B with QLoRA (r=16): mean perplexity 5.03, standard deviation 0.03. Increasing rank from 8 to 16 gave statistically insignificant improvement while raising memory consumption. For small GPT-2 family models, full fine-tuning yielded lower perplexity (3.48 for GPT-2 Medium) than LoRA (7.60-8.42), but induced catastrophic forgetting. The encoder-only XLM-RoBERTa showed the worst results (perplexity 59.3). The novelty lies in creating the largest verified Tajik corpus and the first systematic analysis of PEFT effectiveness for Tajik text generation. Practical value lies in recommendations for architecture and fine-tuning strategy selection, optimizing computational costs without substantial quality loss.

Mullosharaf K. Arabov

Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia

LLM Fine-Tuning Low-Resource Language Processing Text Generation Parameter-Efficient Fine-Tuning

Ask PDF

May 22, 2024

Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. Vikhr'' refers to the name of the Mistral LLM series and means a strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.

Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2

ITMO University · HSE University · Vikhr +2

Language Model Pretraining Multilingual Language Models LLM Fine-Tuning Language Modeling Instruction Tuning Tokenizer Adaptation

Ask PDF

Abstract

Low-Resource Language Processing

Explore similar work

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian