Nepali Passport Question Answering: A Low-Resource Dataset for Public Service Applications
Authors: Funghang Limbu Begha, Praveen Acharya, Bal Krishna Bal
Organizations: ILPRL, Department of Computer Science & Engineering, Kathmandu University, Dhulikhel, Nepal · School of Computing, Dublin City University, Dublin, Ireland
Nepali, a low-resource language, faces significant challenges in building an effective information retrieval system due to the unavailability of annotated data and computational linguistic resources. In this study, we attempt to address this gap by preparing a pair-structured Nepali Question-Answer dataset. We focus on Frequently Asked Questions (FAQs) for passport-related services, building a data set for training and evaluation of IR models. In our study, we have fine-tuned transformer-based embedding models for semantic similarity in question-answer retrieval. The fine-tuned models were compared with the baseline BM25. In addition, we implement a hybrid retrieval approach, integrating fine-tuned models with BM25, and evaluate the performance of the hybrid retrieval. Our results show that the fine-tuned SBERT-based models outperform BM25, whereas multilingual E5 embedding-based models achieve the highest retrieval performance among all evaluated models.
Figures & tables
Fig. 1: Sample of FAQs from Our Dataset
Original QA Pairs
Features
Train
Validation
Test
Total/Avg
Query
384
82
82
548
Answer
384
82
82
548
Tokens per Query
16.94
17.17
16.17
16.86
Tokens per Answer
47.39
49.49
54.65
48.79
Augmented QA Pairs
TABLE I: Statistics of the Nepali Passport Question-Answering Dataset
Query
Relevancy
Corpus
Avg D/Q
Avg Word Length
82
Binary
37,013
10
10.9
TABLE II: Nepali Passport Question-Answering Retrieval Test Set Statistics
Fig. 2: Token count distribution across the Nepali question-answer pair dataset for training, validation, and test splits using the intfloat/multilingual-e5-large tokenizer. The histogram shows the number of tokens per pair, calculated by summing the tokens of the query and its corresponding positive entry, highlighting the overall sequence length patterns in the dataset.
Fig. 3: Workflow of the proposed information retrieval framework, which evaluates lexical (BM25), fine-tuned embedding-based model, and hybrid (BM25 + intfloat/multilingual-e5-base) retrieval models, followed by statistical significance testing against the BM25 baseline.
Parameter
Value
Evaluation Strategy
Steps
Evaluation Steps
100
Warmup ratio
0.01
Fp16
True
TABLE III: Common Training Configuration Across All Models
Model
Batch Size
Learning Rate
Yunika/sentence-transformer-nepali [ 9 ]
8
1×10−5
Syubraj/sentence_similarity_nepali [ 12 ]
8
2×10−5
universalml/Nepali_Embedding_Model [ 10 ]
8
1×10−5
jangedoo/all-MiniLM-L6-v2-nepali [ 11 ]
8
1×10−5
intfloat/e5-small (multilingual) [ 13 ]
8
4×10−5
intfloat/e5-base (multilingual) [ 13 ]
8
3×10−5
TABLE IV: Fine-tuning Configurations for All Embedding Models Used in Our Study
Model
Recall@5
Recall@10
NDCG@5
NDCG@10
MRR@5
MRR@10
BM25
0.3537
0.4793
0.7393
0.5693
0.8689
0.8757
Syubraj/sentence_similarity_nepali
0.3683
0.5951 β
0.7599
0.6546 β
0.8598
0.8598
Yunika/sentence-transformer-nepali
0.4671 β
0.7988 β
0.9464 β
0.8493 β
0.9858 α
0.9858 α
jangedoo/all-MiniLM-L6-v2-nepali
0.3598
0.6037 β
0.7361
0.6510 α
0.8327
0.8388
intfloat/e5-small
0.4695 β
0.8232 β
0.9472 β
0.8663 β
0.9760 β
0.9760 β
intfloat/e5-base
0.4841 β
0.8573 β
0.9756 β
0.8973 β
1.0000 β
1.0000 β
TABLE V: Evaluation Results of Retrieval Models on Nepali Passport Question-Answering Dataset. Superscripts indicate statistical significance against BM25 baseline: α:p<0.05 , β:p<0.01 . Values without superscripts are not statistically significant.
Information and Language Processing Research Lab Kathmandu University , Dhulikhel, Kavre, Nepal · School of Law Kathmandu Universit; Dhulikhel, Kavre, Nepal · Dublin City University Dublin, Ireland