Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics
Organizations: Herald College Kathmandu, University of Wolverhampton, Naxal, Kathmandu 44600, Nepal · Department of Mathematical Sciences, Montana Technological University, Butte, MT 59701, USA · ASI Industrial, Billings, MT 59101, USA
Abstract
Romanized Nepali, Nepali written in Latin script, is common in digital communication but has no standard spelling and few resources, and it is unclear how reliably small open language models can be adapted to it. We adapt three 2-3B base models from different families, Llama-3.2-3B, Qwen2.5-3B, and Gemma-2-2B, with QLoRA on 9,000 instruction examples produced by deterministic transliteration of Nepali instruction data. Each model is trained with one shared recipe and three seeds and evaluated on 1,000 held-out examples with reference-overlap metrics, byte-normalized reference likelihood (bits per byte, BPB), and an output audit, against zero-shot baselines under two prompts. Fine-tuning raises chrF++ from 10-14 to 29.0-30.6 and lowers BPB from 2.23-2.41 to 0.78-0.84; most zero-shot outputs (54-92%) are English-like. Likelihood and overlap disagree on how far apart the models are. Llama-3.2-3B has the lowest BPB in every seed, 0.042 below Qwen2.5-3B (95% CI 0.039-0.046), yet the two cannot be separated on chrF++ (30.51 and 30.61) or BLEU, even with paired item-level tests. Gemma-2-2B, whose tokenizer splits Romanized Nepali words least (2.72 tokens per word), scores worst on every measure. Which model gains most over its zero-shot baseline depends on the zero-shot prompt. In a supplementary comparison with an identical tokenizer, Llama-3.1-8B lowers BPB by 0.053 (0.050-0.056) relative to Llama-3.2-3B, while chrF++ and BLEU rise by only 0.68 and 0.58 points. Lower fertility and higher reference likelihood therefore did not identify the model whose outputs were closest to the references.
Figures & tables
| Devanagari | Gloss | R1: ITRANS | R2: lowercased | R3: R2, final vowel dropped |
|---|---|---|---|---|
| Nepal | nepAla | nepala | nepal | |
| house | ghara | ghara | ghar | |
| is | Cha | cha | cha | |
| also | pani | pani | pani | |
| water | pAnI | pani | pani | |
| three | tIna | tina | tin |
| Model | Total params | Non-embedding params | Embedding share | Vocabulary | Tokenizer | Pretraining tokens | Licence |
|---|---|---|---|---|---|---|---|
| Llama-3.2-3B | 3.21B | 2.82B | 12.3% | 128,256 | byte-level BPE | up to 9T | Llama 3.2 Community License |
| Qwen2.5-3B | 3.09B | 2.77B | 10.1% | 151,643 | byte-level BPE | 18T | Qwen Research License |
| Gemma-2-2B | 2.61B | 2.02B | 22.6% | 256k | SentencePiece | 2T | Gemma Terms of Use |
| Llama-3.1-8B | 8.03B | 6.98B | 13.1% | 128,256 | byte-level BPE | ~15T | Llama 3.1 Community License |
| Tokenizer | Tokens per word [95% CI] | Median | Tokens per char | Fine-tuned chrF++ | Fine-tuned BPB |
|---|---|---|---|---|---|
| Gemma-2-2B | 2.717 [2.689, 2.747] | 2.677 | 0.341 | 29.02 | 0.842 |
| Llama-3.2-3B (= Llama-3.1-8B) | 2.938 [2.908, 2.968] | 2.928 | 0.374 | 30.51 | 0.779 |
| Qwen2.5-3B (= Qwen3-8B) | 2.997 [2.965, 3.029] | 3.000 | 0.380 | 30.61 | 0.821 |
| Mistral-7B-v0.1 | 3.295 [3.262, 3.329] | 3.276 | 0.419 | – | – |
| Tokenizer | R1 | R2 [95% CI] | R3 | R2 single-token words | R2 words of 3 tokens | R2 tokens per char |
|---|---|---|---|---|---|---|
| Gemma-2 | 2.493 | 2.135 [2.107, 2.165] | 2.069 | 30.9% | 31.6% | 0.339 |
| Llama 3 | 2.712 | 2.358 [2.332, 2.385] | 2.264 | 22.4% | 39.7% | 0.377 |
| Qwen2.5 | 2.733 | 2.387 [2.359, 2.414] | 2.294 | 22.2% | 40.9% | 0.381 |
| Mistral v0.1 | 3.053 | 2.648 [2.612, 2.683] | 2.536 | 19.0% | 49.7% | 0.420 |
| Model | Prompt | chrF++ | BLEU | BERTScore F1 | Empty outputs | BPB |
|---|---|---|---|---|---|---|
| Qwen2.5-3B | Alpaca | 13.65 | 0.64 | 0.545 | 0 | 2.373 |
| Qwen2.5-3B | bare | 11.45 | 0.43 | 0.492 (0.541) | 91 | 2.404 |
| Llama-3.2-3B | Alpaca | 12.51 | 1.01 | 0.587 | 4 | 2.225 |
| Llama-3.2-3B | bare | 12.65 | 0.38 | 0.536 | 9 | 2.276 |
| Gemma-2-2B | Alpaca | 10.13 | 0.41 | 0.539 | 3 | 2.415 |
| Gemma-2-2B | bare | 10.96 | 0.12 | 0.526 | 2 | 2.466 |
| Condition | English-like | Input copy | Repetition loop | Empty |
|---|---|---|---|---|
| Reference responses | 0 / 922 | 10 | 1 / 181 | 0 |
| Fine-tuned (9 runs) | 0–2 / 909–931 | 4–20 | 0–2 / 175–210 | 0 |
| Qwen2.5-3B, Alpaca | 923 | 2 | 2 | 0 |
| Qwen2.5-3B, bare | 608 | 3 | 29 | 91 |
| Llama-3.2-3B, Alpaca | 543 | 135 | 24 | 4 |
| Llama-3.2-3B, bare | 641 | 22 | 110 / 794 | 9 |
| Model | chrF++ | BLEU | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore F1 | BPB |
|---|---|---|---|---|---|---|---|
| Qwen2.5-3B | 30.61 0.97 | 5.86 0.26 | 0.252 0.004 | 0.082 0.001 | 0.203 0.002 | 0.757 0.001 | 0.821 0.003 |
| Llama-3.2-3B | 30.51 0.76 | 5.81 0.10 | 0.251 0.005 | 0.080 0.003 | 0.203 0.004 | 0.755 0.001 | 0.779 0.001 |
| Gemma-2-2B | 29.02 0.62 | 4.78 0.24 | 0.220 0.004 | 0.064 0.005 | 0.176 0.004 | 0.750 0.001 | 0.842 0.002 |
| Model | chrF++ gain, Alpaca | chrF++ gain, bare | BLEU gain, Alpaca | BLEU gain, bare | BPB ratio, Alpaca | BPB ratio, bare |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | +21.30 | +17.77 | +4.99 | +6.00 | 0.358 | 0.349 |
| Gemma-2-2B | +18.89 | +18.05 | +4.37 | +4.66 | 0.349 | 0.341 |
| Llama-3.2-3B | +18.00 | +17.86 | +4.80 | +5.43 | 0.350 | 0.342 |
| Qwen2.5-3B | +16.95 | +19.16 | +5.22 | +5.42 | 0.346 | 0.342 |
| Comparison | Measure | Difference | Paired item-level 95% CI | Seed-level 95% CI |
|---|---|---|---|---|
| Llama-3.2-3B Qwen2.5-3B | BPB | 0.0423 | [ 0.0462, 0.0387] | [ 0.0493, 0.0353] |
| chrF++ | 0.09 | [ 0.50, 0.31] | [ 2.12, 1.93] | |
| BLEU | 0.04 | [ 0.56, 0.39] | [ 0.62, 0.53] | |
| Gemma-2-2B Qwen2.5-3B | BPB | +0.0205 | [0.0168, 0.0244] | [0.0142, 0.0268] |
| chrF++ | 1.59 | [ 2.01, 1.14] | [ 3.58, 0.40] | |
| BLEU | 1.08 | [ 1.51, 0.68] | [ 1.65, 0.50] |
| Condition | Model | chrF++ | BLEU | BERTScore F1 | BPB | Empty |
|---|---|---|---|---|---|---|
| Fine-tuned | Llama-3.1-8B | 31.19 0.70 | 6.40 0.18 | 0.760 0.0004 | 0.726 0.002 | 0 |
| Fine-tuned | Llama-3.2-3B | 30.51 0.76 | 5.81 0.10 | 0.755 0.001 | 0.779 0.001 | 0 |
| Zero-shot, Alpaca | Llama-3.1-8B | 9.89 | 1.41 | 0.500 (0.583) | 2.027 | 141 |
| Zero-shot, Alpaca | Llama-3.2-3B | 12.51 | 1.01 | 0.587 | 2.225 | 4 |
| Zero-shot, bare | Llama-3.1-8B | 13.42 | 0.40 | 0.533 (0.541) | 2.082 | 14 |
| Zero-shot, bare | Llama-3.2-3B | 12.65 | 0.38 | 0.536 | 2.276 | 9 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Detail |
|---|---|
| Dataset splits | train 9ed86da7…, validation 3f7f6854…, test d656353e…; hash-identical copies of the dataset directory were uploaded to Kaggle more than once and read by different sessions and accounts (the zero-shot launch records and the 8B seed-44 run use a second copy); all 19 retained runs have the same 1,000 test references. |
| Tokenizer revisions | Fertility and length analyses (Transformers 5.16.1, tokenizers 0.23.1): Llama-3.2-3B 13afe512…, Qwen2.5-3B 3aab1f19…, gemma-2-2b c5ebcd40…, Llama-3.1-8B d04e592b…, Mistral-7B-v0.1 27d67f1b…, Qwen3-8B b968826d…, Qwen3-8B-Base 49e3418f…; fertility script a0c856c5…, length-audit script 3938c8a4…. |
| Model weights | Loaded through Unsloth, which substitutes its pre-quantized 4-bit checkpoints (Section 4.2 ); their revisions were not recorded. |
| Training, primary models | train.py v2 (579d36c7…, canonical) with per-run patched copies; Qwen2.5-3B seeds 43 and 44 were trained in parallel, one per GPU, with Unsloth 2026.10.11, from a copy that differs only in its CONFIG values and one comment line (d9d03919…); the excluded second seed-42 replicate used train.py v1, which has the same training settings, with Unsloth 2026.10.11 and Transformers 5.16.1; other Unsloth versions, bitsandbytes and TRL versions not recorded. |
| Training, Llama-3.1-8B | canonical train.py with only the model, data and output paths changed; patched copies b6949e2f… (seed 42), d6212c1f… (seed 43) and 662c1093… (seed 44); Unsloth 2026.11.3; seed 44 trained in a separate Kaggle account from a hash-identical copy of the data. |
| Fine-tuned evaluation, primary models | eval.py v2.2 (metric files record the script hash as unknown); Llama-3.2-3B evaluations used Unsloth 2026.10.11 (seed 42) and 2026.11.1 (seeds 43 and 44) with Transformers 5.17.0, PEFT 0.21.2 and torch 2.11.0+cu130; Qwen2.5-3B seed 42 was evaluated with Unsloth 2026.10.11 and Transformers 5.16.1, and seeds 43 and 44 with a copy of eval.py v2.2 whose hash was recorded (a231a1ae…); other Qwen2.5-3B and the Gemma-2-2B evaluation versions not recorded. |
| Model | Seed | Status | Unsloth (training) | Notes |
|---|---|---|---|---|
| Qwen2.5-3B | 42 | used (first run) | not recorded | |
| Qwen2.5-3B | 42 | excluded (diverged) | 2026.10.11 | second replicate, train.py v1 |
| Qwen2.5-3B | 43 | used | 2026.10.11 | selected step 2200 |
| Qwen2.5-3B | 44 | used | 2026.10.11 | selected step 2200 |
| Llama-3.2-3B | 42 | used | not recorded | |
| Llama-3.2-3B | 43 | used | not recorded | one non-finite gradient norm (step 2150) |
| Devanagari | ITRANS (case-preserving) | Lowercased |
|---|---|---|
| / | ch / Ch | ch |
| / | t / T | t |
| / | th / Th | th |
| / | d / D | d |
| / | dh / Dh | dh |
| / | n / N | n |
| Model | Seed | chrF++ | BLEU | ROUGE-L | BPB | chrF++ type A | chrF++ type B | Length |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B | 42 | 29.53 | 5.61 | 0.202 | 0.8208 | 30.70 | 28.39 | 0.75 |
| Qwen2.5-3B | 43 | 31.42 | 5.84 | 0.201 | 0.8243 | 32.68 | 30.21 | 0.84 |
| Qwen2.5-3B | 44 | 30.87 | 6.13 | 0.205 | 0.8182 | 32.16 | 29.61 | 0.79 |
| Llama-3.2-3B | 42 | 30.19 | 5.82 | 0.201 | 0.7795 | 31.67 | 28.75 | 0.79 |
| Llama-3.2-3B | 43 | 31.38 | 5.91 | 0.200 | 0.7779 | 32.64 | 30.15 | 0.83 |
| Llama-3.2-3B | 44 | 29.97 | 5.72 | 0.207 | 0.7790 | 31.45 | 28.53 | 0.76 |
Language Modeling
Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.
See all 542 papers