Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali

Ananda Rimal; Adarsha Rimal

cs.CLMar 25, 2026

Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali

Authors: Ananda Rimal, Adarsha Rimal

Organizations: Nepal Engineering College · Dept. of Computer Science & Engineering, Nepal Engineering College · Tribhuvan University · Central Dept. of CS and IT, Tribhuvan University

Abstract

Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal, yet it remains critically underresourced in the landscape of Large Language Models (LLMs). This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. We evaluate these architectures under zero-shot and fine-tuned settings using a curated bilingual dataset of 10,000 transliterated instruction-following samples. Performance is quantified across five metrics spanning seven measurement dimensions: Perplexity (PPL), BERTScore, chrF++, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU, capturing fluency, phonetic consistency, and semantic integrity. Models were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) with Rank-Stabilized LoRA (rsLoRA) at rank r=32 on dual NVIDIA Tesla T4 GPUs, training only approximately 1% of each model's parameters in under 27 total GPU-hours. At zero-shot, all three models fail to generate Romanized Nepali, each exhibiting a distinct architecture-specific failure mode. Following fine-tuning, all three resolve these failures and converge to BERTScore approximately 0.75 and chrF++ greater than 23. Overall dimension-wise assessment across ten criteria identifies Qwen3-8B as the overall recommended architecture, being the only model to produce semantically relevant zero-shot output and leading all structural alignment metrics post-SFT. The adaptation headroom hypothesis is confirmed: Llama-3.1-8B, despite its weakest zero-shot baseline, achieves the largest absolute fine-tuning gains in PPL (Delta = -49.77) and BERTScore (Delta = +0.3287), making it the preferred choice for iterative low-resource development pipelines. This work establishes the first rigorous baseline for Romanized Nepali adaptation in comparable-sized open-weight LLMs.

Figures & tables

Panel of Figure 1 — Figure 1: Experimental pipeline: data preparation, parameter-efficient fine-tuning, and evaluation across three model architectures.

Category	Parameter	Value
Optimization	Optimizer	AdamW (8-bit) [ 32 ]
	Learning Rate	$1\times 10^{-4}$
	Warmup Steps	200
Batch Logic	Per-Device Batch Size	2
Batch Logic	Effective Batch Size	16 ( $2\times 4\times 2$ GPUs)
Hardware	GPUs	Dual NVIDIA Tesla T4 (16 GB each)

Table 1: Training, hardware, adapter, and checkpointing configuration applied identically to all three models.

Phase	Name	Method	Output
1	Zero-Shot	Unmodified weights; raw Romanized input; no prompting	Performance floor across five metrics, seven measurement dimensions
2	SFT	QLoRA [ 1 ] + rsLoRA [ 2 ] ; 4-bit NF4; 3 epochs; Golden Point recovery	Best-checkpoint adapters per model
3A	Quantitative	5-metric 7 dimension scoring; zero-shot vs. fine-tuned; $\Delta$ per metric	Per-model gain table
3B	Qualitative	10 Golden Questions; pre- and post-SFT	Linguistic failure analysis

Table 2: Summary of the three experimental phases.

Model	PPL $\downarrow$	BERT $\uparrow$	BLEU $\uparrow$	chrF++ $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$
Llama-3.1-8B	52.79	0.4224	0.0143	8.15	0.0732	0.0311	0.0681
Qwen3-8B	27.89	0.5631	0.0099	13.10	0.0570	0.0195	0.0513
Mistral-7B	27.53	0.2327	0.0105	4.95	0.0309	0.0134	0.0281

Table 3: Zero-shot baseline performance on the 1k test set.

\downarrow

= lower is better;

\uparrow

= higher is better.

Model	PPL $\downarrow$	BERT $\uparrow$	BLEU $\uparrow$	chrF++ $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$
Llama-3.1-8B	3.024	0.7511	0.0498	26.97	0.2756	0.1002	0.2359
Qwen3-8B	2.946	0.7505	0.0550	27.47	0.2915	0.1162	0.2511
Mistral-7B	2.812	0.7339	0.0404	23.95	0.2460	0.0842	0.2144

Table 4: Fine-tuned performance after QLoRA + rsLoRA SFT with Golden Point recovery. Bold = best per metric.

Bold = largest gain per column. $\downarrow$ = lower is better; $\uparrow$ = higher is better.
Model	$\Delta$ PPL $\downarrow$	$\Delta$ BERT $\uparrow$	$\Delta$ BLEU $\uparrow$	$\Delta$ chrF++ $\uparrow$	$\Delta$ R-1 $\uparrow$	$\Delta$ R-2 $\uparrow$	$\Delta$ R-L $\uparrow$
Llama-3.1-8B	$-$ 49.77	$+$ 0.3287	$+$ 0.0355	$+$ 18.82	$+$ 0.2024	$+$ 0.0691	$+$ 0.1678
Qwen3-8B	$-$ 24.95	$+$ 0.1874	$+$ 0.0451	$+$ 14.37	$+$ 0.2345	$+$ 0.0967	$+$ 0.1998
Mistral-7B	$-$ 24.72	$+$ 0.5012	$+$ 0.0299	$+$ 19.00	$+$ 0.2151	$+$ 0.0708	$+$ 0.1863

Table 5: Absolute fine-tuning gain (

\Delta

= fine-tuned

-

zero-shot) across five metrics spanning seven measurement dimensions.

Figure 2: Training and validation loss for Llama-3.1-8B over 3,375 steps. The green dotted line marks the best checkpoint at step 2,200 (minimum validation loss = 1.1585).

Figure 3: Training and validation loss for Mistral-7B-v0.1 over 3,375 steps. The green dotted line marks the best checkpoint at step 2,200 (minimum validation loss = 1.0930).

Figure 4: Training and validation loss for Qwen3-8B over 3,375 steps. The green dotted line marks the best checkpoint at step 2,200 (minimum validation loss = 1.1313).

Model Scale
Property	Llama-3.1-8B	Qwen3-8B	Mistral-7B-v0.1
Total Parameters	8,114,147,328	8,278,029,312	7,325,618,176
Trainable Parameters	83,886,080	87,293,952	83,886,080
Trainable %	1.03%	1.05%	1.15%
Training Cost
Wall-Clock Time	8h 50m 51s	8h 26m 01s	9h 09m 31s

Table 6: Trainable parameter counts and wall-clock training time from Unsloth [ 30 ] runtime logs. All models: 3 epochs / 3,375 steps, rank

r=32

[ 18 ] , dual NVIDIA Tesla T4.

Model	Failure Mode	Technical Description
Llama-3.1-8B	Semantic Void	Early EOS Triggering [ 6 ] : Tiktoken over-fragments Romanized Nepali; model immediately predicts End-of-Sequence, yielding null output on 6 of 10 instructions.
Mistral-7B-v0.1	Newline Loop	Distributional Collapse [ 7 ] : SentencePiece produces plausible fragments but the model fails to initiate a response, defaulting to continuous \n generation.
Qwen3-8B	Script Drift	Latent Script Bias [ 8 ] : Model correctly identifies semantic intent but defaults to dominant pre-training scripts rather than Romanized format.

Table 7: Taxonomy of structural failures in zero-shot base models.

Dimension	Llama-3.1-8B	Qwen3-8B	Mistral-7B-v0.1	Best
Post-SFT Quality (five metrics, seven dimensions)
Post-SFT PPL $\downarrow$	3rd (3.024)	2nd (2.946)	1st (2.812)	Mistral
Post-SFT BERTScore $\uparrow$	1st (0.7511)	2nd (0.7505)	3rd (0.7339)	Llama
Post-SFT chrF++ $\uparrow$	2nd (26.97)	1st (27.47)	3rd (23.95)	Qwen3-8B
Post-SFT ROUGE-1 $\uparrow$	2nd (0.2756)	1st (0.2915)	3rd (0.2460)	Qwen3-8B
Post-SFT ROUGE-2 $\uparrow$	2nd (0.1002)	1st (0.1162)	3rd (0.0842)	Qwen3-8B

Table 8: Dimension-wise post-SFT ranking across ten evaluation criteria. Bold indicates best per row.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Instruction	Input	Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.	—	[no response]
Nepal ko barema thodai bhannus ta.	—	[no response]
Cache memory bhaneko k ho?	—	[no response]
Yo numerical pattern pura garnuhos.	1, 2, 3, 4, …	5
Diyeko vakya lai shuddha banayera lekhnuhos.	Ma bholi ghar janchu thiyo.	Diyeko vakya lai shuddha banayera lekhnuho. Ma bholi ghar janchu thiyo.
Operating System ko bhumika k huncha?	—	[no response]

Appendix

Table 9: Llama-3.1-8B base model responses (zero-shot). [no response] on 6 of 10 instructions.

Instruction	Input	Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.	—	kampyutara eka yantra ho jasale janakari prashodhana garcha ra samasyaharu samadhana gardacha
Nepal ko barema thodai bhannus ta.	—	nepala uttara purva esiyama avasthita eka sano rashtra ho
Cache memory bhaneko k ho?	—	kyasa memori eka choto-avadhi bhandarana pranali ho jasale druta rupama prayoga hune data va nirdeshanaharu pahuncha garna sakshama cha
Yo numerical pattern pura garnuhos.	1, 2, 3, 4, …	5
Diyeko vakya lai shuddha banayera lekhnuhos.	Ma bholi ghar janchu thiyo.	ma bholi ghara janchu thien
Operating System ko bhumika k huncha?	—	operetimga sistama kampyutara pranaliko adhikarika ho ra prayogakartaharusanga kampyutaralai niyantrana garna anumati dincha\| yasale pranali srotaharu vyavasthapana gardacha, prashodhana ra pradarshana pradana gardacha, ra prayogakarta anuprayogaharulai pratyaksha pahuncha pradana gardacha\|

Appendix

Table 10: Llama-3.1-8B fine-tuned responses (post-SFT). All 10 instructions answered in Romanized Nepali.

Instruction	Input	Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.	—	[Devanagari script] : manche haru le aphno kampyutar ko samasya hal garna prayaas gardachhan … (off-script; definition request misread as troubleshooting)
Nepal ko barema thodai bhannus ta.	—	Sure! Here’s how you can complete this sentence in Nepali: “Nepal ma raatko sabai chhato janaa.” This translates to “In Nepal, it gets very cold at night.”
Cache memory bhaneko k ho?	—	Cache memory in computer systems refers to high-speed data storage mechanisms designed to reduce the average time it takes for active data to be requested by serving frequently accessed information from faster (but smaller) sources than main memory. This helps improve system performance and efficiency because accessing cache typically requires less power compared to reading directly from primary or secondary storage devices like hard drives or SSDs.
Yo numerical pattern pura garnuhos.	1, 2, 3, 4, …	The given sequence appears to be increasing by one each time: Start at 1; Add 1 to get the next number (e.g., 1 + 1 = 2); Continue this process indefinitely. So the next few numbers in the series would be: 5, 6, 7, …
Diyeko vakya lai shuddha banayera lekhnuhos.	Ma bholi ghar janchu thiyo.	Here’s my attempt at completing this task: “Ma ko ghar ma janayau.” I hope I did it correctly!
Operating System ko bhumika k huncha?	—	An operating system (OS) plays several crucial roles in managing computer hardware and software resources. Key functions: (1) Resource Management — manages CPU time, memory, disk, network; (2) Process Control — scheduling algorithms for task execution; (3) Memory Protection & Sharing — secure access control; (4) Device Drivers Interface — intermediary between apps and hardware; (5) Security Measures — authentication protocols and encryption.

Appendix

Table 11: Qwen3-8B base model responses (zero-shot). All responses in English or Devanagari — no Romanized Nepali generated.

Instruction	Input	Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.	—	kampyutara eka yantra ho jasale janakari prashodhana garcha ra prayogakartalai janakari utpanna gardacha\| yasama prosesara, memori, inaputa autaputa upakaranaharu samavesha hunchan\|
Nepal ko barema thodai bhannus ta.	—	nepala dakshina esiyama avasthita eka puratana desha ho\| yo himali kshetraharu ra vishvako sabaibhanda aglo pahada sagaramatha (8,848 mitara) samga jivita cha\| yasale dherai prakaraka paridrrishyaharu pradana gardacha, jasama ghanse maidanaharu, tatiya ghumauro bagaimchaharu, shanta tala ra charko nadiharu chan\| nepalale vibhinna prakaraka samskrriti ra bhasha pani prastuta gardacha, juna 125 mukhya bhashaharu milera baneko cha\|
Cache memory bhaneko k ho?	—	kyashe memori eka prakarako chito pahuncha garna dijaina garieko samana ra data ko lagi avasthita huna sakcha\| yasale dherai prayoga garine programina nirdeshanaharu ra detako lagi thula matrama kyacheda va “bhandarana” gardacha juna ramro-pradarshana vikalpa hunu parne sambhavana cha\|
Yo numerical pattern pura garnuhos.	1, 2, 3, 4, …	5
Diyeko vakya lai shuddha banayera lekhnuhos.	Ma bholi ghar janchu thiyo.	maile bholi ghara gaen
Operating System ko bhumika k huncha?	—	operiuna sistamako mukhya karya prayogakartaharulai hardaveyara srotaharu pahuncha garna ra pranalilai vyavasthapana garna anumati dina ho\| yasale memori vyavasthapanama maddata gardacha, pranali suru garnecha ra banda garnecha, phailaharu padhne ra lekhne, prakriyaharu chalaune, netavarka jadana sthapita garne, ra thapa\|

Appendix

Table 12: Qwen3-8B fine-tuned responses (post-SFT). All 10 instructions answered correctly in Romanized Nepali.

Instruction	Input	Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.	—	The instruction in this example is written in Nepali and asks for a computer to be built. The input provided is a list of components that need to be included in the build. The response should provide a detailed explanation of how to assemble the computer using the provided components. This could include step-by-step instructions on how to install each component, such as the motherboard, CPU, RAM, graphics card, power supply, hard drive, and case. It should also explain how to connect all the components together and ensure they are properly installed and functioning correctly.
Nepal ko barema thodai bhannus ta.	—	[empty line]
Cache memory bhaneko k ho?	—	[empty line]
Yo numerical pattern pura garnuhos.	1, 2, 3, 4, …	1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
Diyeko vakya lai shuddha banayera lekhnuhos.	Ma bholi ghar janchu thiyo.	The input “Ma bholi ghar janchu thiyo” means “I went to my mother’s house.” The response “Ma bholi ghar janchu thiyo” is a complete sentence that correctly expresses the same meaning as the input. It uses the correct verb form (thiyo) and subject-verb agreement (ma).
Operating System ko bhumika k huncha?	—	[empty line]

Appendix

Table 13: Mistral-7B-v0.1 base model responses (zero-shot). [empty line] on 5 of 10 instructions; remaining 5 in English, off-task.

Instruction	Input	Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.	—	kampyutara bhaneko kampyutina pranaliko eka prakara ho jasale manisaharulai janakari ra suchanako vishleshana garna sakshama gardacha
Nepal ko barema thodai bhannus ta.	—	nepala eka samrriddha ra vividha desha ho jasale vishvavyapi rupama manava ra janavaraharuko ghara banauncha\| yo yasa prakhyata kshetrako ghara ho, ra yasako sabaibhanda najikaka shaharaharu madhye eka ho\| yo vishvavyapi rupama manava ra janavaraharule baneko cha, ra yo vishvavyapi rupama manava-jastai cha\|
Cache memory bhaneko k ho?	—	ache memori eka prakarako samana ho jasale deta va janakari bhandarana garna prayoga gardacha\|
Yo numerical pattern pura garnuhos.	1, 2, 3, 4, …	5
Diyeko vakya lai shuddha banayera lekhnuhos.	Ma bholi ghar janchu thiyo.	ma bholi ghara janchu thien\|
Operating System ko bhumika k huncha?	—	aparetina sistama prayogakarta anuprayoga ra anuprayogaharuma pahuncha pradana garna dijaina garieko saphtaveyara vikasa pranali ho\| yo prayogakartako avashyakataharu pura garna dijaina garieko ho, jastai prayogakartale kuna anuprayogaharu chaliraheka chan, kasari prayogakarta intaraphesa samrachita cha, ra kasari prayogakarta deta vyavasthita cha

Appendix

Table 14: Mistral-7B-v0.1 fine-tuned responses (post-SFT). All 10 instructions answered in Romanized Nepali. Two residual semantic errors are annotated (

\hookrightarrow

Explore similar work

Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR
Mar 8, 2026Rishikesh Kumar Sharma, Safal Narshing Shrestha, Jenny Poudel +4Automatic Speech Recognition Low-Resource Language Processing
Information and Language Processing Research Lab, Kathmandu University
LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language
May 9, 2026Luca BalloreLanguage Model Pretraining Adapter Tuning
Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language
Jun 15, 2026Rabindra Nath NandiDecoder-Only Language Models Small Language Models
Independent Researcher

Mar 8, 2026J/K move · Enter open · S save

Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR

Rishikesh Kumar Sharma, Safal Narshing Shrestha, Jenny Poudel, Rupak Tiwari+3

Information and Language Processing Research Lab, Kathmandu University

Automatic Speech Recognition Low-Resource Language Processing Low-Resource Speech Recognition Cross-Lingual Transfer

Open paper AskPDF arXiv

Nepal Bhasha (Newari), an endangered language of the Kathmandu Valley, remains digitally marginalized due to the severe scarcity of annotated speech resources. In this work, we introduce Nwāchā Munā, a newly curated 5.39-hour manually transcribed Devanagari speech corpus for Nepal Bhasha, and establish the first benchmark using script-preserving acoustic modeling. We investigate whether proximal cross-lingual transfer from a geographically and linguistically adjacent language (Nepali) can rival large-scale multilingual pretraining in an ultra-low-resource Automatic Speech Recognition (ASR) setting. Fine-tuning a Nepali Conformer model reduces the Character Error Rate (CER) from a 52.54% zero-shot baseline to 17.59% with data augmentation, effectively matching the performance of the multilingual Whisper-Small model despite utilizing significantly fewer parameters. Our findings demonstrate that proximal transfer from Nepali language serves as a computationally efficient alternative to massive multilingual models. We openly release the dataset and benchmarks to digitally enable the Newari community and foster further research in Nepal Bhasha.

Abstract

Explore similar work

Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR

LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language

Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language