Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language

Mullosharaf K. Arabov; Svetlana S. Khaybullina

cs.CLOct 22, 2026

Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language

Authors: Mullosharaf K. Arabov, Svetlana S. Khaybullina

Organizations: Department of Data Analysis and Programming Technologies, Institute of Computational Mathematics and Information Technologies, Kazan (Volga Region) Federal University, Kazan, Russia

Abstract

Bashkir is a low-resource agglutinative language written in Cyrillic and remains poorly represented in modern LLMs. We present an open resource suite and a systematic benchmark of adaptation methods: the Bashkir Web Corpus (71,567 documents, 46.9M tokens, 16 sources) and three labelled datasets for binary, multiclass and multilabel text classification; the benchmark covers the first two. Experiments use eight architectures (82M-8B: DistilGPT2, GPT-2 base/medium, Phi-3-mini, Mistral-7B, YandexGPT-5-Lite-8B, mBERT, XLM-R) and three adaptation groups (full fine-tuning, QLoRA, in-context learning), plus continual pretraining (CPT). (1) QLoRA on 3.8-8B models outperforms encoder-only fine-tuning on binary classification: YandexGPT-5-Lite-8B with CPT reaches macro-F1 0.748 vs 0.653 (XLM-R) and 0.650 (mBERT); pairwise differences are significant after Holm correction (p<0.001). (2) YandexGPT-5-Lite-8B significantly outperforms Mistral-7B and Phi-3-mini on multiclass classification, BPB and native-speaker ratings (mean 3.28 vs 2.52 and 2.22; n=100; alpha=0.69-0.77; p<0.05); on binary classification its QLoRA advantage is not significant (p>=0.084). (3) CPT effects depend on model and task: perplexity improves for Phi-3-mini (+0.58) and YandexGPT-5-Lite-8B (+0.33) but worsens for Mistral-7B (-1.15, p<0.001); on binary classification CPT helps Mistral-7B most (+4.0 macro-F1). (4) Tokeniser fertility on Bashkir varies 2.67x (2.92-7.80) and tracks adaptation quality, while perplexity across tokenisers is not comparable. Cyrillic tokenisers with Russian pretraining are preferable; LLM judges reproduce native ratings for grammar and coherence (rho=0.33-0.46) but not language correctness (rho=0.16, p=0.11); native evaluation remains necessary. All artefacts are open-licensed.

Figures & tables

Group	lr	Epochs	Batch size	$r$
Full FT (step1)	$2\times 10^{-4}$	3	8 / 4	—
QLoRA (step3)	$10^{-4}$	2	2 / 1	16
CPT (step4)	$10^{-5}$	1	1	16
Downstream QLoRA (step4)	$10^{-4}$	2	1	16
QLoRA CLS (step6)	$10^{-4}$	2	2 / 1	8
Encoder full FT (step6)	$2\times 10^{-5}$	3	16	—

Table 1: Hyperparameters by experiment group

Model	Method	PPL	BPB	Target	Trainable	Peak
				tokens (ba)	parameters, %	memory, MB
GPT-2 medium	Full FT	2.33	0.784	0.271	100	—
GPT-2 base	Full FT	2.49	0.844	0.532	100	—
DistilGPT2	Full FT	2.69	0.917	0.566	100	—
Mistral-7B	QLoRA	2.90	0.549	0.598	1.10	4693
YandexGPT-5-Lite-8B	QLoRA	3.29	0.462	0.601	0.91	6543

Table 2: Language modelling: perplexity and bits-per-byte on the test set of the corpus (128 tokens). For Full FT, all parameters; for QLoRA, the proportion of trainable parameters and peak GPU memory

Comparison ( $a$ vs $b$ )	$\Delta$ PPL [95% CI]	$p_{\text{PPL}}$	$\Delta$ BPB [95% CI]	$p_{\text{BPB}}$	Cohen’s $d$	Cliff’s $\delta$
Mistral-7B vs Phi-3-mini	$+1.226$ $[+1.137,+1.319]$	$<0.001$	$+0.155$ $[+0.150,+0.161]$	$<0.001$	$+0.43$	$-0.25$
Mistral-7B vs YandexGPT-5-Lite-8B	$+0.394$ $[+0.359,+0.431]$	$<0.001$	$-0.087$ $[-0.091,-0.082]$	$<0.001$	$+0.17$	$-0.10$
Phi-3-mini vs YandexGPT-5-Lite-8B	$-0.832$ $[-0.901,-0.760]$	$<0.001$	$-0.242$ $[-0.250,-0.233]$	$<0.001$	$-0.23$	$+0.16$

Table 3: Pairwise comparisons of QLoRA models on perplexity and BPB (test set of the corpus,

n=2000

examples).

\Delta=\text{metric}_{b}-\text{metric}_{a}

; a negative

\Delta

means that model

b

is better.

p_{\text{Holm}}

is the Holm–Bonferroni correction for 6 comparisons

Model	English (WikiText-2)		Russian (Wikipedia)
	PPL ${}_{\text{base}}\to$ after	BPB ${}_{\text{base}}\to$ after	PPL ${}_{\text{base}}\to$ after	BPB ${}_{\text{base}}\to$ after
GPT-2 medium	39.58 $\to$ 1504.67	1.162 $\to$ 2.310	9.11 $\to$ 14.45	1.854 $\to$ 2.241
GPT-2 base	54.49 $\to$ 4121.17	1.263 $\to$ 2.630	11.07 $\to$ 14.14	2.018 $\to$ 2.223
DistilGPT2	78.05 $\to$ 2963.08	1.377 $\to$ 2.526	12.66 $\to$ 14.87	2.131 $\to$ 2.266

Table 4: Catastrophic forgetting: perplexity and BPB on English and Russian before and after full fine-tuning on Bashkir (

n=200

texts per language, average over two seeds)

Model	PPL ${}_{\text{no-CPT}}$	PPL ${}_{\text{CPT+QLoRA}}$	$\Delta$ PPL	$p_{\text{Holm}}$	Cliff’s $\delta$
Mistral-7B	20.48	21.63	$-1.15$	$<0.001$	large
Phi-3-mini	35.81	35.23	$+0.58$	$<0.001$	large
YandexGPT-5-Lite-8B	12.70	12.37	$+0.33$	$<0.001$	large

Table 5: Effect of continual pretraining on the perplexity of the downstream-adapted model. PPL is the aggregate

\exp(\sum\text{NLL}/\sum\text{tokens})

; Cliff’s

\delta

is at the per-example level

Model	Method	macro-F1	weighted-F1	Accuracy	$n_{\text{seed}}$
Full fine-tuning (small models)
GPT-2 medium	Full FT	0.699	0.700	0.702	2
GPT-2 base	Full FT	0.698	0.697	0.700	2
DistilGPT2	Full FT	0.693	0.693	0.696	2
QLoRA without CPT (step6, $r=8$ )
YandexGPT-5-Lite-8B	QLoRA	0.736	0.736	0.737	1

Table 6: Binary classification (news vs analysis): macro-F1, weighted-F1 and accuracy. QLoRA in step6 uses rank

r=8

; QLoRA in step4 (with and without CPT) uses rank

r=16

Comparison ( $a$ vs $b$ )	acc a	acc b	$\Delta$ [95% CI]	$p_{\text{Holm}}$
Yandex vs Mistral	0.737	0.713	$-0.024$ $[-0.050,+0.001]$	$0.183$
Yandex vs Phi-3	0.737	0.702	$-0.035$ $[-0.064,-0.004]$	$0.084$
Yandex vs mBERT	0.737	0.652	$-0.085$ $[-0.117,-0.053]$	$<0.001$
Yandex vs XLM-R	0.737	0.659	$-0.078$ $[-0.109,-0.047]$	$<0.001$
Yandex vs majority	0.737	0.520	$-0.217$ $[-0.260,-0.171]$	$<0.001$

Table 7: Binary classification: key pairwise comparisons on accuracy (paired bootstrap, McNemar, Holm correction for 15 comparisons).

\Delta=\text{acc}_{b}-\text{acc}_{a}

; a positive

\Delta

means that model

b

is better

Model	Method	macro-F1	weighted-F1	Accuracy	$n_{\text{seed}}$
QLoRA (medium-scale generative models)
YandexGPT-5-Lite-8B	QLoRA (step6)	0.368	0.570	0.606	1
YandexGPT-5-Lite-8B	QLoRA no CPT (step4)	0.332	0.545	0.573	1
YandexGPT-5-Lite-8B	CPT + QLoRA (step4)	0.366	0.567	0.595	1
Mistral-7B	QLoRA (step6)	0.292	0.510	0.547	1
Mistral-7B	QLoRA no CPT (step4)	0.298	0.512	0.542	1

Table 8: Multiclass classification (19 categories): macro-F1, weighted-F1 and accuracy. QLoRA in step6 uses rank

r=8

; QLoRA in step4 (with and without CPT) uses rank

r=16

Comparison ( $a$ vs $b$ )	acc a	acc b	$\Delta$ [95% CI]	$p_{\text{Holm}}$
Yandex vs Mistral	0.606	0.547	$-0.059$ $[-0.087,-0.031]$	$<0.001$
Yandex vs Phi-3	0.606	0.506	$-0.100$ $[-0.128,-0.072]$	$<0.001$
Mistral vs Phi-3	0.547	0.506	$-0.041$ $[-0.069,-0.013]$	$0.030$
XLM-R vs majority	0.470	0.470	$+0.000$ $[+0.000,+0.000]$	$1.000$
Phi-3 vs mBERT	0.506	0.498	$-0.008$ $[-0.031,+0.015]$	$1.000$

Table 9: Multiclass classification: key pairwise comparisons on accuracy (paired bootstrap, McNemar, Holm correction for 15 comparisons).

\Delta=\text{acc}_{b}-\text{acc}_{a}

; a positive

\Delta

means that model

b

is better

	Bashkir (ba)			Russian (ru)			English (en)
Model	0	4	8	0	4	8	0	4	8
YandexGPT-5-Lite-8B	0.580	0.579	0.634	0.657	0.640	0.565	0.679	0.583	0.627
Mistral-7B	0.360	0.315	0.421	0.482	0.414	0.452	0.411	0.558	0.397
Phi-3-mini	0.513	0.444	0.411	0.506	0.457	0.516	0.400	0.574	0.650

Table 10: ICL on the binary task: macro-F1 by instruction language and number of examples in the prompt

Model	Language	$K$	$\text{Acc}_{0}$	$\text{Acc}_{K}$	$\Delta$ [95% CI]	$p_{\text{McNemar}}$	$p_{\text{Holm}}$
Mistral-7B	ba	4	0.520	0.460	$-0.060$ $[-0.260,+0.130]$	$0.610$	$1.000$
	ba	8	0.520	0.440	$-0.080$ $[-0.240,+0.080]$	$0.396$	$1.000$
	en	4	0.530	0.590	$+0.060$ $[-0.110,+0.230]$	$0.561$	$1.000$
	en	8	0.530	0.530	$+0.000$ $[-0.060,+0.060]$	$1.000$	$1.000$
	ru	4	0.510	0.470	$-0.040$ $[-0.150,+0.060]$	$0.572$	$1.000$
	ru	8	0.510	0.540	$+0.030$ $[-0.130,+0.190]$	$0.804$	$1.000$

Table 11: Effect of few-shot demonstrations: difference in accuracy versus zero-shot, 95% bootstrap CI and McNemar

p

(binary task,

n=100

test examples per language).

p_{\text{Holm}}

is the Holm–Bonferroni correction for 18 comparisons. No comparison is significant after correction.

Model	Grammar	Coherence	Language	Wins	Losses	Ties
YandexGPT-5-Lite-8B	2.89	2.39	2.56	16	0	20
Mistral-7B	2.83	2.44	2.58	18	0	16
Phi-3-mini	2.75	2.28	2.36	12	5	17
GPT-2 base	2.37	2.19	2.66	1	23	14
DistilGPT2	2.35	2.16	2.56	9	4	21
GPT-2 medium	2.13	1.89	2.16	1	25	12

Table 12: Evaluation of generation by LLM judges: direct scoring (mean over three judges, 1–5 scale) and pairwise win rate

Model	$n$	Grammar	Coherence	Language	Mean
YandexGPT-5-Lite-8B	11	2.94 [2.70, 3.15]	3.09 [2.76, 3.42]	3.82 [3.61, 3.97]	3.28
Mistral-7B	11	2.12 [1.97, 2.30]	2.18 [1.94, 2.48]	3.24 [3.00, 3.48]	2.52
Phi-3-mini	9	1.81 [1.56, 2.07]	1.81 [1.48, 2.11]	3.04 [2.85, 3.22]	2.22
DistilGPT2	23	1.48 [1.28, 1.70]	1.39 [1.22, 1.58]	2.39 [2.16, 2.62]	1.75
GPT-2 base	23	1.39 [1.20, 1.59]	1.30 [1.14, 1.51]	2.33 [2.16, 2.52]	1.68
GPT-2 medium	23	1.16 [1.04, 1.30]	1.22 [1.10, 1.35]	1.70 [1.45, 1.94]	1.36

Table 13: Native Bashkir speaker ratings (mean over three annotators, 1–5 scale, 95% bootstrap CI)

Criterion	$H$	$p$	$\varepsilon^{2}$	Significant pairs
Grammar	50.81	$9.4\times 10^{-10}$	0.487	9/15
Coherence	49.58	$1.7\times 10^{-9}$	0.474	9/15
Language	61.18	$6.9\times 10^{-12}$	0.598	13/15

Table 14: Kruskal–Wallis (omnibus) test and the number of significant pairwise comparisons (Mann–Whitney, Holm correction, family

=15

pairs per criterion)

Model	Fertility (ba)	Fertility (en)	Fertility (tr)
GPT-2 family (DistilGPT2, base, medium)	7.80	1.36	3.58
Phi-2 (2.7B)	7.80	1.36	3.58
Mistral-7B-v0.3	4.81	1.32	3.75
Qwen2.5-7B	4.23	1.37	3.00
YandexGPT-5-Lite-8B	3.80	1.26	3.22
T-lite-2.1	2.92	1.37	3.03

Table 15: Fertility rate (subtokens per word) on the Bashkir, English and Turkish test sets

Model	MorphAcc	OOV rate
Mistral-7B-v0.3	0.093	0.065
YandexGPT-5-Lite-8B	0.086	0.037
GPT-2 family (DistilGPT2, base, medium)	0.057	0.043
Phi-2 (2.7B)	0.057	0.043
Qwen2.5-7B	0.047	0.017
T-lite-2.1	0.047	0.017

Table 16: Morphological Boundary Accuracy and OOV rate on the Bashkir test set. MorphAcc is the proportion of subtokens whose right boundary coincides with a morpheme boundary; OOV rate is the proportion of word forms split into individual bytes or characters

Metric	$\eta^{2}_{\text{model}}$	$\eta^{2}_{\text{method}}$	$\eta^{2}_{\text{seed}}$	$n$
PPL	0.54	0.73	0.13	15
BPB	0.36	0.72	0.05	15
Proportion of target tokens	0.81	0.27	0.08	9

Table 17: Decomposition of variance (

\eta^{2}

) for the key metrics

Low-Resource Language Processing

Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.

See all 565 papers

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
May 6, 2026Mullosharaf K. Arabov, Svetlana S. KhaybullinaLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia
Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus
May 5, 2026Mullosharaf K. ArabovLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian
May 22, 2024Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2Language Model Pretraining Multilingual Language Models
ITMO University · HSE University · Vikhr +2

May 6, 2026J/K move · Enter open · S save

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

Mullosharaf K. Arabov, Svetlana S. Khaybullina

Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia

LLM Fine-Tuning Low-Resource Language Processing Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

This paper presents a comparative study of parameter-efficient fine-tuning (PEFT) methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language, a low-resource agglutinative language of the Turkic family. Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures: DistilGPT2, GPT-2 (base, medium), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To improve the reliability of results, each configuration was trained with three different random seeds. The lowest perplexity on the test set was obtained for GPT-2 medium with full fine-tuning (3.34). Meanwhile, QLoRA applied to Mistral-7B (3.79) and Phi-2 (3.81) achieved comparable quality with over 40 times fewer trainable parameters. However, we also observed cases of significant quality degradation when using PEFT for certain architectures (e.g., DeepSeek-7B with rank 8, perplexity = 129.55), indicating that the outcome depends critically on the choice of the base model and its tokenizer. Additionally, a qualitative analysis of generated texts based on Bashkir prompts revealed that models with the best perplexity do not necessarily produce the most coherent outputs: QLoRA-tuned models generated monolingual Bashkir continuations, whereas the fully fine-tuned model with the lowest perplexity frequently switched to English. The results suggest that QLoRA on 7B-scale models offers an effective compromise between quality and computational cost for Bashkir. To ensure reproducibility, open data, code, and trained adapters will be released upon acceptance.

Abstract

Low-Resource Language Processing

Explore similar work

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian