Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali
Authors: Ananda Rimal, Adarsha Rimal
Organizations: Nepal Engineering College · Dept. of Computer Science & Engineering, Nepal Engineering College · Tribhuvan University · Central Dept. of CS and IT, Tribhuvan University
Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal, yet it remains critically underresourced in the landscape of Large Language Models (LLMs). This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. We evaluate these architectures under zero-shot and fine-tuned settings using a curated bilingual dataset of 10,000 transliterated instruction-following samples. Performance is quantified across five metrics spanning seven measurement dimensions: Perplexity (PPL), BERTScore, chrF++, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU, capturing fluency, phonetic consistency, and semantic integrity. Models were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) with Rank-Stabilized LoRA (rsLoRA) at rank r=32 on dual NVIDIA Tesla T4 GPUs, training only approximately 1% of each model's parameters in under 27 total GPU-hours. At zero-shot, all three models fail to generate Romanized Nepali, each exhibiting a distinct architecture-specific failure mode. Following fine-tuning, all three resolve these failures and converge to BERTScore approximately 0.75 and chrF++ greater than 23. Overall dimension-wise assessment across ten criteria identifies Qwen3-8B as the overall recommended architecture, being the only model to produce semantically relevant zero-shot output and leading all structural alignment metrics post-SFT. The adaptation headroom hypothesis is confirmed: Llama-3.1-8B, despite its weakest zero-shot baseline, achieves the largest absolute fine-tuning gains in PPL (Delta = -49.77) and BERTScore (Delta = +0.3287), making it the preferred choice for iterative low-resource development pipelines. This work establishes the first rigorous baseline for Romanized Nepali adaptation in comparable-sized open-weight LLMs.
Figures & tables
Figure 1: Experimental pipeline: data preparation, parameter-efficient fine-tuning, and evaluation across three model architectures.
Category
Parameter
Value
Optimization
Optimizer
AdamW (8-bit) [ 32 ]
Learning Rate
1×10−4
Warmup Steps
200
Batch Logic
Per-Device Batch Size
2
Effective Batch Size
16 ( 2×4×2 GPUs)
Hardware
GPUs
Dual NVIDIA Tesla T4 (16 GB each)
Table 1: Training, hardware, adapter, and checkpointing configuration applied identically to all three models.
Phase
Name
Method
Output
1
Zero-Shot
Unmodified weights; raw Romanized input; no prompting
Performance floor across five metrics, seven measurement dimensions
2
SFT
QLoRA [ 1 ] + rsLoRA [ 2 ] ; 4-bit NF4; 3 epochs; Golden Point recovery
Best-checkpoint adapters per model
3A
Quantitative
5-metric 7 dimension scoring; zero-shot vs. fine-tuned; Δ per metric
Per-model gain table
3B
Qualitative
10 Golden Questions; pre- and post-SFT
Linguistic failure analysis
Table 2: Summary of the three experimental phases.
Model
PPL ↓
BERT ↑
BLEU ↑
chrF++ ↑
R-1 ↑
R-2 ↑
R-L ↑
Llama-3.1-8B
52.79
0.4224
0.0143
8.15
0.0732
0.0311
0.0681
Qwen3-8B
27.89
0.5631
0.0099
13.10
0.0570
0.0195
0.0513
Mistral-7B
27.53
0.2327
0.0105
4.95
0.0309
0.0134
0.0281
Table 3: Zero-shot baseline performance on the 1k test set. ↓ = lower is better; ↑ = higher is better.
Model
PPL ↓
BERT ↑
BLEU ↑
chrF++ ↑
R-1 ↑
R-2 ↑
R-L ↑
Llama-3.1-8B
3.024
0.7511
0.0498
26.97
0.2756
0.1002
0.2359
Qwen3-8B
2.946
0.7505
0.0550
27.47
0.2915
0.1162
0.2511
Mistral-7B
2.812
0.7339
0.0404
23.95
0.2460
0.0842
0.2144
Table 4: Fine-tuned performance after QLoRA + rsLoRA SFT with Golden Point recovery. Bold = best per metric.
Model
Δ PPL ↓
Δ BERT ↑
Δ BLEU ↑
Δ chrF++ ↑
Δ R-1 ↑
Δ R-2 ↑
Δ R-L ↑
Llama-3.1-8B
− 49.77
+ 0.3287
+ 0.0355
+ 18.82
+ 0.2024
+ 0.0691
+ 0.1678
Qwen3-8B
− 24.95
+ 0.1874
+ 0.0451
+ 14.37
+ 0.2345
+ 0.0967
+ 0.1998
Mistral-7B
− 24.72
+ 0.5012
+ 0.0299
+ 19.00
+ 0.2151
+ 0.0708
+ 0.1863
Bold = largest gain per column. ↓ = lower is better; ↑ = higher is better.
Table 5: Absolute fine-tuning gain ( Δ = fine-tuned − zero-shot) across five metrics spanning seven measurement dimensions.
Figure 2: Training and validation loss for Llama-3.1-8B over 3,375 steps. The green dotted line marks the best checkpoint at step 2,200 (minimum validation loss = 1.1585).
Figure 3: Training and validation loss for Mistral-7B-v0.1 over 3,375 steps. The green dotted line marks the best checkpoint at step 2,200 (minimum validation loss = 1.0930).
Figure 4: Training and validation loss for Qwen3-8B over 3,375 steps. The green dotted line marks the best checkpoint at step 2,200 (minimum validation loss = 1.1313).
Property
Llama-3.1-8B
Qwen3-8B
Mistral-7B-v0.1
Model Scale
Total Parameters
8,114,147,328
8,278,029,312
7,325,618,176
Trainable Parameters
83,886,080
87,293,952
83,886,080
Trainable %
1.03%
1.05%
1.15%
Training Cost
Wall-Clock Time
8h 50m 51s
8h 26m 01s
9h 09m 31s
Table 6: Trainable parameter counts and wall-clock training time from Unsloth [ 30 ] runtime logs. All models: 3 epochs / 3,375 steps, rank r=32 [ 18 ] , dual NVIDIA Tesla T4.
Model
Failure Mode
Technical Description
Llama-3.1-8B
Semantic Void
Early EOS Triggering [ 6 ] : Tiktoken over-fragments Romanized Nepali; model immediately predicts End-of-Sequence, yielding null output on 6 of 10 instructions.
Mistral-7B-v0.1
Newline Loop
Distributional Collapse [ 7 ] : SentencePiece produces plausible fragments but the model fails to initiate a response, defaulting to continuous \n generation.
Qwen3-8B
Script Drift
Latent Script Bias [ 8 ] : Model correctly identifies semantic intent but defaults to dominant pre-training scripts rather than Romanized format.
Table 7: Taxonomy of structural failures in zero-shot base models.
Dimension
Llama-3.1-8B
Qwen3-8B
Mistral-7B-v0.1
Best
Post-SFT Quality (five metrics, seven dimensions)
Post-SFT PPL ↓
3rd (3.024)
2nd (2.946)
1st (2.812)
Mistral
Post-SFT BERTScore ↑
1st (0.7511)
2nd (0.7505)
3rd (0.7339)
Llama
Post-SFT chrF++ ↑
2nd (26.97)
1st (27.47)
3rd (23.95)
Qwen3-8B
Post-SFT ROUGE-1 ↑
2nd (0.2756)
1st (0.2915)
3rd (0.2460)
Qwen3-8B
Post-SFT ROUGE-2 ↑
2nd (0.1002)
1st (0.1162)
3rd (0.0842)
Qwen3-8B
Table 8: Dimension-wise post-SFT ranking across ten evaluation criteria. Bold indicates best per row.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Instruction
Input
Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.
—
[no response]
Nepal ko barema thodai bhannus ta.
—
[no response]
Cache memory bhaneko k ho?
—
[no response]
Yo numerical pattern pura garnuhos.
1, 2, 3, 4, …
5
Diyeko vakya lai shuddha banayera lekhnuhos.
Ma bholi ghar janchu thiyo.
Diyeko vakya lai shuddha banayera lekhnuho. Ma bholi ghar janchu thiyo.
Operating System ko bhumika k huncha?
—
[no response]
Appendix
Table 9: Llama-3.1-8B base model responses (zero-shot). [no response] on 6 of 10 instructions.
Instruction
Input
Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.
—
kampyutara eka yantra ho jasale janakari prashodhana garcha ra samasyaharu samadhana gardacha
Nepal ko barema thodai bhannus ta.
—
nepala uttara purva esiyama avasthita eka sano rashtra ho
Cache memory bhaneko k ho?
—
kyasa memori eka choto-avadhi bhandarana pranali ho jasale druta rupama prayoga hune data va nirdeshanaharu pahuncha garna sakshama cha
Yo numerical pattern pura garnuhos.
1, 2, 3, 4, …
5
Diyeko vakya lai shuddha banayera lekhnuhos.
Ma bholi ghar janchu thiyo.
ma bholi ghara janchu thien
Operating System ko bhumika k huncha?
—
operetimga sistama kampyutara pranaliko adhikarika ho ra prayogakartaharusanga kampyutaralai niyantrana garna anumati dincha| yasale pranali srotaharu vyavasthapana gardacha, prashodhana ra pradarshana pradana gardacha, ra prayogakarta anuprayogaharulai pratyaksha pahuncha pradana gardacha|
Appendix
Table 10: Llama-3.1-8B fine-tuned responses (post-SFT). All 10 instructions answered in Romanized Nepali.
Instruction
Input
Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.
—
[Devanagari script] : manche haru le aphno kampyutar ko samasya hal garna prayaas gardachhan … (off-script; definition request misread as troubleshooting)
Nepal ko barema thodai bhannus ta.
—
Sure! Here’s how you can complete this sentence in Nepali: “Nepal ma raatko sabai chhato janaa.” This translates to “In Nepal, it gets very cold at night.”
Cache memory bhaneko k ho?
—
Cache memory in computer systems refers to high-speed data storage mechanisms designed to reduce the average time it takes for active data to be requested by serving frequently accessed information from faster (but smaller) sources than main memory. This helps improve system performance and efficiency because accessing cache typically requires less power compared to reading directly from primary or secondary storage devices like hard drives or SSDs.
Yo numerical pattern pura garnuhos.
1, 2, 3, 4, …
The given sequence appears to be increasing by one each time: Start at 1; Add 1 to get the next number (e.g., 1 + 1 = 2); Continue this process indefinitely. So the next few numbers in the series would be: 5, 6, 7, …
Diyeko vakya lai shuddha banayera lekhnuhos.
Ma bholi ghar janchu thiyo.
Here’s my attempt at completing this task: “Ma ko ghar ma janayau.” I hope I did it correctly!
Operating System ko bhumika k huncha?
—
An operating system (OS) plays several crucial roles in managing computer hardware and software resources. Key functions: (1) Resource Management — manages CPU time, memory, disk, network; (2) Process Control — scheduling algorithms for task execution; (3) Memory Protection & Sharing — secure access control; (4) Device Drivers Interface — intermediary between apps and hardware; (5) Security Measures — authentication protocols and encryption.
Appendix
Table 11: Qwen3-8B base model responses (zero-shot). All responses in English or Devanagari — no Romanized Nepali generated.
Instruction
Input
Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.
nepala dakshina esiyama avasthita eka puratana desha ho| yo himali kshetraharu ra vishvako sabaibhanda aglo pahada sagaramatha (8,848 mitara) samga jivita cha| yasale dherai prakaraka paridrrishyaharu pradana gardacha, jasama ghanse maidanaharu, tatiya ghumauro bagaimchaharu, shanta tala ra charko nadiharu chan| nepalale vibhinna prakaraka samskrriti ra bhasha pani prastuta gardacha, juna 125 mukhya bhashaharu milera baneko cha|
Cache memory bhaneko k ho?
—
kyashe memori eka prakarako chito pahuncha garna dijaina garieko samana ra data ko lagi avasthita huna sakcha| yasale dherai prayoga garine programina nirdeshanaharu ra detako lagi thula matrama kyacheda va “bhandarana” gardacha juna ramro-pradarshana vikalpa hunu parne sambhavana cha|
Yo numerical pattern pura garnuhos.
1, 2, 3, 4, …
5
Diyeko vakya lai shuddha banayera lekhnuhos.
Ma bholi ghar janchu thiyo.
maile bholi ghara gaen
Operating System ko bhumika k huncha?
—
operiuna sistamako mukhya karya prayogakartaharulai hardaveyara srotaharu pahuncha garna ra pranalilai vyavasthapana garna anumati dina ho| yasale memori vyavasthapanama maddata gardacha, pranali suru garnecha ra banda garnecha, phailaharu padhne ra lekhne, prakriyaharu chalaune, netavarka jadana sthapita garne, ra thapa|
Appendix
Table 12: Qwen3-8B fine-tuned responses (post-SFT). All 10 instructions answered correctly in Romanized Nepali.
Instruction
Input
Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.
—
The instruction in this example is written in Nepali and asks for a computer to be built. The input provided is a list of components that need to be included in the build. The response should provide a detailed explanation of how to assemble the computer using the provided components. This could include step-by-step instructions on how to install each component, such as the motherboard, CPU, RAM, graphics card, power supply, hard drive, and case. It should also explain how to connect all the components together and ensure they are properly installed and functioning correctly.
The input “Ma bholi ghar janchu thiyo” means “I went to my mother’s house.” The response “Ma bholi ghar janchu thiyo” is a complete sentence that correctly expresses the same meaning as the input. It uses the correct verb form (thiyo) and subject-verb agreement (ma).
Operating System ko bhumika k huncha?
—
[empty line]
Appendix
Table 13: Mistral-7B-v0.1 base model responses (zero-shot). [empty line] on 5 of 10 instructions; remaining 5 in English, off-task.
Instruction
Input
Output
Computer bhaneko k ho? Yaslai nepali ma paribhashit garnuhos.
—
kampyutara bhaneko kampyutina pranaliko eka prakara ho jasale manisaharulai janakari ra suchanako vishleshana garna sakshama gardacha
Nepal ko barema thodai bhannus ta.
—
nepala eka samrriddha ra vividha desha ho jasale vishvavyapi rupama manava ra janavaraharuko ghara banauncha| yo yasa prakhyata kshetrako ghara ho, ra yasako sabaibhanda najikaka shaharaharu madhye eka ho| yo vishvavyapi rupama manava ra janavaraharule baneko cha, ra yo vishvavyapi rupama manava-jastai cha|
Cache memory bhaneko k ho?
—
ache memori eka prakarako samana ho jasale deta va janakari bhandarana garna prayoga gardacha|
Yo numerical pattern pura garnuhos.
1, 2, 3, 4, …
5
Diyeko vakya lai shuddha banayera lekhnuhos.
Ma bholi ghar janchu thiyo.
ma bholi ghara janchu thien|
Operating System ko bhumika k huncha?
—
aparetina sistama prayogakarta anuprayoga ra anuprayogaharuma pahuncha pradana garna dijaina garieko saphtaveyara vikasa pranali ho| yo prayogakartako avashyakataharu pura garna dijaina garieko ho, jastai prayogakartale kuna anuprayogaharu chaliraheka chan, kasari prayogakarta intaraphesa samrachita cha, ra kasari prayogakarta deta vyavasthita cha
Appendix
Table 14: Mistral-7B-v0.1 fine-tuned responses (post-SFT). All 10 instructions answered in Romanized Nepali. Two residual semantic errors are annotated ( ↪ ).
Nepal Bhasha (Newari), an endangered language of the Kathmandu Valley, remains digitally marginalized due to the severe scarcity of annotated speech resources. In this work, we introduce Nwāchā Munā, a newly curated 5.39-hour manually transcribed Devanagari speech corpus for Nepal Bhasha, and establish the first benchmark using script-preserving acoustic modeling. We investigate whether proximal cross-lingual transfer from a geographically and linguistically adjacent language (Nepali) can rival large-scale multilingual pretraining in an ultra-low-resource Automatic Speech Recognition (ASR) setting. Fine-tuning a Nepali Conformer model reduces the Character Error Rate (CER) from a 52.54% zero-shot baseline to 17.59% with data augmentation, effectively matching the performance of the multilingual Whisper-Small model despite utilizing significantly fewer parameters. Our findings demonstrate that proximal transfer from Nepali language serves as a computationally efficient alternative to massive multilingual models. We openly release the dataset and benchmarks to digitally enable the Newari community and foster further research in Nepal Bhasha.
Sardinian, a Romance language with roughly one million speakers, has minimal presence in modern NLP. Commercial services do not support it, and current language models do not produce it reliably. We present LLiMba, a 3B parameter Sardinian-ready model adapted from Qwen2.5-3B-Instruct through continued pretraining (CPT) and supervised fine-tuning (SFT) on a single 24 GB consumer GPU. The corpus contains 11.5 million tokens of Sardinian spanning LSC, Logudorese, and Campidanese, augmented with 2.4 million tokens of related Romance text as replay against register blurring. After CPT the model reaches a perplexity of 6.76 on held out Sardinian and outperforms the base across all six FLORES-200 directions. We compare five SFT configurations under matched conditions: full fine-tuning, LoRA r64, rsLoRA r128, rsLoRA r256, and DoRA r256. rsLoRA r256 wins on every direction into Sardinian, reaching 28.5 BLEU from English against 17.3 after CPT and 21.0 with full fine-tuning. The rank ablation places r128 between LoRA r64 and rsLoRA r256 on BLEU but reveals failure modes invisible to the metric, including leakage across scripts no other variant produces. LoRA r64 retains less factual content from SFT than configurations at higher rank and produces more confident fabrications, though all methods fabricate on content absent from training. DoRA r256 yields the smallest gap between training and evaluation but the worst factual accuracy. The findings indicate that adapter capacity matters more than the choice among LoRA variants for adapting a Romance pretrained base to a low resource Romance target, that stronger regularization is not uniformly beneficial, and that translation metrics smoothly order configurations whose qualitative behavior differs categorically. Perplexity comparisons across scripts must account for byte fallback tokenization, which deflates the metric for scripts other than Latin.
While the NLP landscape is dominated by multi-billion parameter architectures, their deployment in low-resource, non-Latin scripts remains computationally prohibitive for edge configurations, mobile systems, and decentralized local hardware. This paper presents bangla-smollm-135m, a highly compact 135-million parameter decoder-only foundational model engineered explicitly for high-efficiency language modeling in the Bangla script. By leveraging a deterministic intersect-and-append token merging strategy between TituLLMs and SmolLM2-135M, the model overcomes subword script fragmentation without destabilizing early pretrained parameter states. In zero-shot multi-task benchmark evaluations (PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, and Bangla_MMLU), bangla-smollm-135m matches or outperforms models twice its size (Gemma-3-270m) and achieves parity with models in the 1B parameter tier. The model is available at rnnandi/bangla-smollm-135m