Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics
Authors: Pujan Pant, Sagar Newpane, Hari Sitaula, Manish Sakhakarmy
Organizations: Herald College Kathmandu, University of Wolverhampton, Naxal, Kathmandu 44600, Nepal · Department of Mathematical Sciences, Montana Technological University, Butte, MT 59701, USA · ASI Industrial, Billings, MT 59101, USA
Romanized Nepali, Nepali written in Latin script, is common in digital communication but has no standard spelling and few resources, and it is unclear how reliably small open language models can be adapted to it. We adapt three 2-3B base models from different families, Llama-3.2-3B, Qwen2.5-3B, and Gemma-2-2B, with QLoRA on 9,000 instruction examples produced by deterministic transliteration of Nepali instruction data. Each model is trained with one shared recipe and three seeds and evaluated on 1,000 held-out examples with reference-overlap metrics, byte-normalized reference likelihood (bits per byte, BPB), and an output audit, against zero-shot baselines under two prompts. Fine-tuning raises chrF++ from 10-14 to 29.0-30.6 and lowers BPB from 2.23-2.41 to 0.78-0.84; most zero-shot outputs (54-92%) are English-like. Likelihood and overlap disagree on how far apart the models are. Llama-3.2-3B has the lowest BPB in every seed, 0.042 below Qwen2.5-3B (95% CI 0.039-0.046), yet the two cannot be separated on chrF++ (30.51 and 30.61) or BLEU, even with paired item-level tests. Gemma-2-2B, whose tokenizer splits Romanized Nepali words least (2.72 tokens per word), scores worst on every measure. Which model gains most over its zero-shot baseline depends on the zero-shot prompt. In a supplementary comparison with an identical tokenizer, Llama-3.1-8B lowers BPB by 0.053 (0.050-0.056) relative to Llama-3.2-3B, while chrF++ and BLEU rise by only 0.68 and 0.58 points. Lower fertility and higher reference likelihood therefore did not identify the model whose outputs were closest to the references.
Figures & tables
Devanagari
Gloss
R1: ITRANS
R2: lowercased
R3: R2, final vowel dropped
Nepal
nepAla
nepala
nepal
house
ghara
ghara
ghar
is
Cha
cha
cha
also
pani
pani
pani
water
pAnI
pani
pani
three
tIna
tina
tin
Table 1: Example words under the three romanization conventions. R2 (lowercased ITRANS) is the benchmark scheme; R1 and R3 are used only in the fertility analysis (Section 5.1 ). Under R2, and , and , and and become identical.
Model
Total params
Non-embedding params
Embedding share
Vocabulary
Tokenizer
Pretraining tokens
Licence
Llama-3.2-3B
3.21B
2.82B
12.3%
128,256
byte-level BPE
up to 9T
Llama 3.2 Community License
Qwen2.5-3B
3.09B
2.77B
10.1%
151,643
byte-level BPE
18T
Qwen Research License
Gemma-2-2B
2.61B
2.02B
22.6%
256k
SentencePiece
2T
Gemma Terms of Use
Llama-3.1-8B
8.03B
6.98B
13.1%
128,256
byte-level BPE
~15T
Llama 3.1 Community License
Table 2: Models. Parameter counts are computed from the published model configurations; the embedding share of Llama-3.1-8B includes its separate output matrix. Vocabulary, pretraining tokens and licences are from the model reports and cards [ 12 , 14 , 29 , 30 , 38 ] . The LoRA adapter sizes implied by these configurations match the trainable-parameter counts in Section 4.2 .
Tokenizer
Tokens per word [95% CI]
Median
Tokens per char
Fine-tuned chrF++
Fine-tuned BPB
Gemma-2-2B
2.717 [2.689, 2.747]
2.677
0.341
29.02
0.842
Llama-3.2-3B (= Llama-3.1-8B)
2.938 [2.908, 2.968]
2.928
0.374
30.51
0.779
Qwen2.5-3B (= Qwen3-8B)
2.997 [2.965, 3.029]
3.000
0.380
30.61
0.821
Mistral-7B-v0.1
3.295 [3.262, 3.329]
3.276
0.419
–
–
Table 3: Tokenizer fertility on the 1,000 benchmark references (41,388 words) beside the three-seed fine-tuned results. Fertility is the mean over references of tokens per whitespace-delimited word, with 95% bootstrap intervals (1,000 resamples). Higher chrF++ and lower BPB are better. Mistral-7B-v0.1 was not trained here.
Tokenizer
R1
R2 [95% CI]
R3
R2 single-token words
R2 words of ≥ 3 tokens
R2 tokens per char
Gemma-2
2.493
2.135 [2.107, 2.165]
2.069
30.9%
31.6%
0.339
Llama 3
2.712
2.358 [2.332, 2.385]
2.264
22.4%
39.7%
0.377
Qwen2.5
2.733
2.387 [2.359, 2.414]
2.294
22.2%
40.9%
0.381
Mistral v0.1
3.053
2.648 [2.612, 2.683]
2.536
19.0%
49.7%
0.420
Table 4: Fertility on a public sample of 1,233 Nepali sentences under three romanization conventions: R1, case-preserving ITRANS; R2, lowercased ITRANS (the benchmark scheme); R3, R2 without the final inherent vowel in words of two or more aksharas. Word shares are over 8,840 running words under R2. Bootstrap intervals have a half-width of 0.03–0.04 under every convention.
Model
Prompt
chrF++
BLEU
BERTScore F1
Empty outputs
BPB
Qwen2.5-3B
Alpaca
13.65
0.64
0.545
0
2.373
Qwen2.5-3B
bare
11.45
0.43
0.492 (0.541)
91
2.404
Llama-3.2-3B
Alpaca
12.51
1.01
0.587
4
2.225
Llama-3.2-3B
bare
12.65
0.38
0.536
9
2.276
Gemma-2-2B
Alpaca
10.13
0.41
0.539
3
2.415
Gemma-2-2B
bare
10.96
0.12
0.526
2
2.466
Table 5: Zero-shot results on the 1,000 test items (one deterministic run per condition). Empty generations are kept and score 0 in BERTScore; the mean over non-empty outputs is in parentheses. BPB is comparable across models within a prompt, not across prompts.
Condition
English-like
Input copy
Repetition loop
Empty
Reference responses
0 / 922
10
1 / 181
0
Fine-tuned (9 runs)
0–2 / 909–931
4–20
0–2 / 175–210
0
Qwen2.5-3B, Alpaca
923
2
2
0
Qwen2.5-3B, bare
608
3
29
91
Llama-3.2-3B, Alpaca
543
135
24
4
Llama-3.2-3B, bare
641
22
110 / 794
9
Table 6: Output audit (heuristic flags, not quality judgments). Counts are out of 1,000 outputs unless an evaluable denominator is shown; the English and loop checks apply only to outputs long enough to evaluate.
Model
chrF++
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
BERTScore F1
BPB
Qwen2.5-3B
30.61 ± 0.97
5.86 ± 0.26
0.252 ± 0.004
0.082 ± 0.001
0.203 ± 0.002
0.757 ± 0.001
0.821 ± 0.003
Llama-3.2-3B
30.51 ± 0.76
5.81 ± 0.10
0.251 ± 0.005
0.080 ± 0.003
0.203 ± 0.004
0.755 ± 0.001
0.779 ± 0.001
Gemma-2-2B
29.02 ± 0.62
4.78 ± 0.24
0.220 ± 0.004
0.064 ± 0.005
0.176 ± 0.004
0.750 ± 0.001
0.842 ± 0.002
Table 7: Fine-tuned results on the 1,000 test items: mean ± sample SD over seeds 42, 43 and 44. Higher is better except for BPB. Qwen2.5-3B seed 42 is the first non-diverged run (Section 4.3 ).
Model
chrF++ gain, Alpaca
chrF++ gain, bare
BLEU gain, Alpaca
BLEU gain, bare
BPB ratio, Alpaca
BPB ratio, bare
Llama-3.1-8B
+21.30
+17.77
+4.99
+6.00
0.358
0.349
Gemma-2-2B
+18.89
+18.05
+4.37
+4.66
0.349
0.341
Llama-3.2-3B
+18.00
+17.86
+4.80
+5.43
0.350
0.342
Qwen2.5-3B
+16.95
+19.16
+5.22
+5.42
0.346
0.342
Table 8: Gains from zero-shot to fine-tuned: the three-seed fine-tuned mean minus the single zero-shot run under each prompt, and the ratio of fine-tuned to zero-shot BPB under each prompt. Computed from Tables 5 , 7 and 10 before rounding.
Figure 1: Gain in chrF++ from zero-shot to fine-tuned, measured from the Alpaca-prompt and the bare-prompt zero-shot baselines. Lines connect the same model. Each model’s fine-tuned score is the same in both columns; only the baseline changes.
Comparison
Measure
Difference
Paired item-level 95% CI
Seed-level 95% CI
Llama-3.2-3B − Qwen2.5-3B
BPB
− 0.0423
[ − 0.0462, − 0.0387]
[ − 0.0493, − 0.0353]
chrF++
− 0.09
[ − 0.50, 0.31]
[ − 2.12, 1.93]
BLEU
− 0.04
[ − 0.56, 0.39]
[ − 0.62, 0.53]
Gemma-2-2B − Qwen2.5-3B
BPB
+0.0205
[0.0168, 0.0244]
[0.0142, 0.0268]
chrF++
− 1.59
[ − 2.01, − 1.14]
[ − 3.58, 0.40]
BLEU
− 1.08
[ − 1.51, − 0.68]
[ − 1.65, − 0.50]
Table 9: Differences between models (first model minus second) with two kinds of 95% interval. Paired item-level: the 1,000 test items resampled 2,000 times, scores recomputed for every seed and averaged over each model’s seeds; it conditions on the trained seeds. Seed-level: Welch intervals over the three per-seed scores of each model; it conditions on the test set. Negative BPB differences mean higher likelihood for the first model. Intervals in bold exclude zero.
Figure 2: Fine-tuned reference likelihood (BPB; lower means higher likelihood) against (a) chrF++ and (b) BLEU. Points are three-seed means and bars show ± 1 seed SD. Llama-3.2-3B and Qwen2.5-3B differ clearly in BPB but in neither generation metric.
Condition
Model
chrF++
BLEU
BERTScore F1
BPB
Empty
Fine-tuned
Llama-3.1-8B
31.19 ± 0.70
6.40 ± 0.18
0.760 ± 0.0004
0.726 ± 0.002
0
Fine-tuned
Llama-3.2-3B
30.51 ± 0.76
5.81 ± 0.10
0.755 ± 0.001
0.779 ± 0.001
0
Zero-shot, Alpaca
Llama-3.1-8B
9.89
1.41
0.500 (0.583)
2.027
141
Zero-shot, Alpaca
Llama-3.2-3B
12.51
1.01
0.587
2.225
4
Zero-shot, bare
Llama-3.1-8B
13.42
0.40
0.533 (0.541)
2.082
14
Zero-shot, bare
Llama-3.2-3B
12.65
0.38
0.536
2.276
9
Table 10: Llama-3.1-8B against Llama-3.2-3B. Fine-tuned rows: mean ± sample SD over seeds 42, 43 and 44 (8B per seed: chrF++ 30.79, 30.78 and 32.00; BLEU 6.20, 6.46 and 6.54; BPB 0.7263, 0.7274 and 0.7236). Zero-shot rows: single runs, with the BERTScore mean over non-empty outputs in parentheses. BPB is comparable across models within a condition, not across zero-shot prompts.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Detail
Dataset splits
train 9ed86da7…, validation 3f7f6854…, test d656353e…; hash-identical copies of the dataset directory were uploaded to Kaggle more than once and read by different sessions and accounts (the zero-shot launch records and the 8B seed-44 run use a second copy); all 19 retained runs have the same 1,000 test references.
Loaded through Unsloth, which substitutes its pre-quantized 4-bit checkpoints (Section 4.2 ); their revisions were not recorded.
Training, primary models
train.py v2 (579d36c7…, canonical) with per-run patched copies; Qwen2.5-3B seeds 43 and 44 were trained in parallel, one per GPU, with Unsloth 2026.10.11, from a copy that differs only in its CONFIG values and one comment line (d9d03919…); the excluded second seed-42 replicate used train.py v1, which has the same training settings, with Unsloth 2026.10.11 and Transformers 5.16.1; other Unsloth versions, bitsandbytes and TRL versions not recorded.
Training, Llama-3.1-8B
canonical train.py with only the model, data and output paths changed; patched copies b6949e2f… (seed 42), d6212c1f… (seed 43) and 662c1093… (seed 44); Unsloth 2026.11.3; seed 44 trained in a separate Kaggle account from a hash-identical copy of the data.
Fine-tuned evaluation, primary models
eval.py v2.2 (metric files record the script hash as unknown); Llama-3.2-3B evaluations used Unsloth 2026.10.11 (seed 42) and 2026.11.1 (seeds 43 and 44) with Transformers 5.17.0, PEFT 0.21.2 and torch 2.11.0+cu130; Qwen2.5-3B seed 42 was evaluated with Unsloth 2026.10.11 and Transformers 5.16.1, and seeds 43 and 44 with a copy of eval.py v2.2 whose hash was recorded (a231a1ae…); other Qwen2.5-3B and the Gemma-2-2B evaluation versions not recorded.
Appendix
Table 11: Scripts, environments and records behind the reported numbers.
Model
Seed
Status
Unsloth (training)
Notes
Qwen2.5-3B
42
used (first run)
not recorded
Qwen2.5-3B
42
excluded (diverged)
2026.10.11
second replicate, train.py v1
Qwen2.5-3B
43
used
2026.10.11
selected step 2200
Qwen2.5-3B
44
used
2026.10.11
selected step 2200
Llama-3.2-3B
42
used
not recorded
Llama-3.2-3B
43
used
not recorded
one non-finite gradient norm (step 2150)
Appendix
Table 12: Training runs and their use. Checkpoints were selected on validation loss (Section 4.3 ); the retained records give the selected step only where noted.
Devanagari
ITRANS (case-preserving)
Lowercased
/
ch / Ch
ch
/
t / T
t
/
th / Th
th
/
d / D
d
/
dh / Dh
dh
/
n / N
n
Appendix
Table 13: Letter groups merged by lowercasing ITRANS. Vowel groups include the corresponding vowel signs.
Model
Seed
chrF++
BLEU
ROUGE-L
BPB
chrF++ type A
chrF++ type B
Length
Qwen2.5-3B
42
29.53
5.61
0.202
0.8208
30.70
28.39
0.75
Qwen2.5-3B
43
31.42
5.84
0.201
0.8243
32.68
30.21
0.84
Qwen2.5-3B
44
30.87
6.13
0.205
0.8182
32.16
29.61
0.79
Llama-3.2-3B
42
30.19
5.82
0.201
0.7795
31.67
28.75
0.79
Llama-3.2-3B
43
31.38
5.91
0.200
0.7779
32.64
30.15
0.83
Llama-3.2-3B
44
29.97
5.72
0.207
0.7790
31.45
28.53
0.76
Appendix
Table 14: Per-seed results of the fine-tuned runs on the 1,000 test items. Length is total generated characters divided by total reference characters.
Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal, yet it remains critically underresourced in the landscape of Large Language Models (LLMs). This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. We evaluate these architectures under zero-shot and fine-tuned settings using a curated bilingual dataset of 10,000 transliterated instruction-following samples. Performance is quantified across five metrics spanning seven measurement dimensions: Perplexity (PPL), BERTScore, chrF++, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU, capturing fluency, phonetic consistency, and semantic integrity. Models were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) with Rank-Stabilized LoRA (rsLoRA) at rank r=32 on dual NVIDIA Tesla T4 GPUs, training only approximately 1% of each model's parameters in under 27 total GPU-hours. At zero-shot, all three models fail to generate Romanized Nepali, each exhibiting a distinct architecture-specific failure mode. Following fine-tuning, all three resolve these failures and converge to BERTScore approximately 0.75 and chrF++ greater than 23. Overall dimension-wise assessment across ten criteria identifies Qwen3-8B as the overall recommended architecture, being the only model to produce semantically relevant zero-shot output and leading all structural alignment metrics post-SFT. The adaptation headroom hypothesis is confirmed: Llama-3.1-8B, despite its weakest zero-shot baseline, achieves the largest absolute fine-tuning gains in PPL (Delta = -49.77) and BERTScore (Delta = +0.3287), making it the preferred choice for iterative low-resource development pipelines. This work establishes the first rigorous baseline for Romanized Nepali adaptation in comparable-sized open-weight LLMs.
Ananda Rimal, Adarsha Rimal
Nepal Engineering College · Dept. of Computer Science & Engineering, Nepal Engineering College · Tribhuvan University +1
We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-file C/CUDA training stack and a Devanagari-aware byte-level BPE tokenizer built for this project. On ARC-Easy and ARC-Challenge, Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens, likely aided by a match between our educational-web-text pretraining data and ARC's grade-school-science format rather than a general capability advantage. We report full evaluation results under standard protocols, including a correction to an earlier partial-sample estimate, and findings specific to evaluating small models in a low-resource language: the standard multiple-choice-letter prompt format used by common evaluation harnesses places this model at chance on Nepali reading comprehension, and simultaneously at chance on English in the same format, which would lead a naive benchmark run to conclude the model has no Nepali ability when in fact it does. Concretely, both languages score at chance in the letter-choice format (0.240 Nepali, 0.236 English, against a chance baseline of 0.250), while scoring the answer text directly reveals genuine, English-favoring comprehension (0.306 Nepali, 0.387 English). We describe a manifest-conditioned tool-use contract introduced during instruction tuning, where tool calls are permitted only when a tool manifest is declared in context and suppressed otherwise, and report where that contract holds and where it does not. We release both the base and instruction-tuned model weights under Apache-2.0. The training code and a small privately-sourced portion of the Nepali pretraining corpus are not released; everything needed to reproduce the reported numbers from the released weights is included here.
Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun
Hundreds of millions of Indians communicate with digital systems in romanized Latin characters -- typing "aap kaise hain" rather than native script -- yet romanized Indic generation is absent from NLP benchmarks and evaluation frameworks. We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation. We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs. On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004. Two distinct model-specific failure modes emerge: Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses). We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses -- revealing failures metrics alone cannot detect. Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.