BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

Md. Najib Hasan; Mst. Jannatun Ferdous Rain; Fyad Mohammed; Nazmul Siddique

cs.CLFeb 16, 2026

BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

Authors: Md. Najib Hasan, Mst. Jannatun Ferdous Rain, Fyad Mohammed, Nazmul Siddique

Organizations: Wichita State University · Begum Rokeya University, Rangpur · Khulna University of Engineering & Technology · Ulster University

Abstract

IR in low-resource languages remains limited by the scarcity of high-quality, task-specific annotated datasets. Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity. This work presents a Bangla IR dataset constructed using a BETA-labeling framework involving multiple LLM annotators from diverse model families. The framework incorporates contextual alignment, consistency checks, and majority agreement, followed by human evaluation to verify label quality. Beyond dataset creation, we examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation. Using LLM-based translation across multiple language pairs, we experimented on meaning preservation and task validity between source and translated datasets. Our experiment reveal substantial variation across languages, reflecting language-dependent biases and inconsistent semantic preservation that directly affect the reliability of cross-lingual dataset reuse. Overall, this study highlights both the potential and limitations of LLM-assisted dataset creation for low-resource IR. It provides empirical evidence of the risks associated with cross-lingual dataset reuse and offers practical guidance for constructing more reliable benchmarks and evaluation pipelines in low-resource language settings.

Figures & tables

Figure 1: Overview of BETA-Labeling and LRL-to-LRL cross-lingual reusability evaluation.

Figure 2: BETA-labeling framework to annotate implicit feature-based data in low-resource languages.

Language Pair	Bangla_Lite									Bangla_Culture
	GPT-4o Mini			Gemini 2.5 Flash Lite			DeepSeek R1 7B			GPT-4o Mini			Gemini 2.5 Flash Lite			DeepSeek R1 7B
	CS	BL	ME	CS	BL	ME	CS	BL	ME	CS	BL	ME	CS	BL	ME	CS	BL	ME
BN $\rightarrow$ EN	78.9	12.3	52.4	78.0	25.0	51.2	23.4	60.0	15.5	93.1	47.4	71.8	92.6	76.3	67.5	33.3	45.9	21.4
BN $\rightarrow$ ZH	72.7	3.16	1.50	69.5	10.8	1.39	30.0	7.04	1.60	93.4	64.2	6.16	91.4	39.1	5.68	35.1	0.89	1.25
BN $\rightarrow$ AR	36.4	33.0	33.1	36.4	11.5	29.3	46.9	8.89	5.27	75.6	30.7	55.0	74.8	24.7	48.6	45.8	5.15	3.94
BN $\rightarrow$ RU	71.7	8.01	44.4	70.6	55.0	42.4	31.1	5.97	5.68	92.5	38.8	63.1	91.8	83.7	58.0	35.2	11.0	5.82

Table 3: Cross-lingual one-hop translation evaluation of three SOTA LLMs (GPT-4o mini, Gemini 2.5 Flash Lite, and DeepSeek R1 7B) on the Bangla_Lite and Bangla_History datasets. In this table, CS, BL, and ME denote Cosine Similarity, BLEU, and METEOR, respectively. BN, EN, ZH, AR, and RU represent Bangla, English, Chinese, Arabic, and Russian, respectively.

Language Pair	Bangla_Lite									Bangla_Culture
	GPT-4o Mini			Gemini 2.5 Flash Lite			DeepSeek R1 7B			GPT-4o Mini			Gemini 2.5 Flash Lite			DeepSeek R1 7B
	BS	CS	JS	BS	CS	JS	BS	CS	JS	BS	CS	JS	BS	CS	JS	BS	CS	JS
BN $\rightarrow$ EN	67.6	59.5	10.9	66.9	57.3	9.7	56.1	38.3	6.12	72.3	71.0	13.7	68.5	66.4	11.4	57.6	44.6	7.92
BN $\rightarrow$ ZH	68.4	63.3	0.01	67.5	60.1	0.0	54.9	34.1	0.16	73.2	77.2	0.02	69.5	72.5	0.02	56.5	34.9	0.09
BN $\rightarrow$ AR	67.5	43.2	6.40	67.5	41.8	4.8	53.3	41.1	0.56	70.8	66.1	8.64	68.8	61.6	6.33	56.9	39.8	0.68
BN $\rightarrow$ RU	67.1	60.3	7.45	67.2	59.6	6.1	52.6	34.1	0.72	71.0	76.3	8.71	68.5	71.2	6.21	57.3	36.4	0.42

Table 4: Cross-lingual one-hop implicit context evaluation of three SOTA LLMs (GPT-4o mini, Gemini 2.5 Flash Lite, and DeepSeek R1 7B) on the Bangla_Lite and Bangla_History datasets. In this table, BS, CS, and JS denote BERTScore, Cosine Similarity, and Jaccard Similarity, respectively. BN, EN, ZH, AR, and RU represent Bangla, English, Chinese, Arabic, and Russian, respectively.

Explore similar work

LLM Probe: Evaluating LLMs for Low-Resource Languages
Mar 31, 2026Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang NejdlMultilingual Language Model Evaluation Low-Resource Language Processing
L3S Research Center, Leibniz University Hannover, Germany · Aksum University, Ethiopia
Are LLMs Ready to Replace Bangla Annotators?
Feb 18, 2026Md. Najib Hasan, Touseef Hasan, Souvika SarkarLLM-Assisted Annotation LLM Reliability
A2I Lab, School of Computing, Wichita State University
Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic
Mar 17, 2026Finnur Ágúst Ingimundarson, Steinunn Rut Friðriksdóttir, Bjarki Ármannsson +2LLM Evaluation Benchmark Validity
University of Zürich · University of Iceland · The Árni Magnússon Institute for Icelandic Studies

Mar 31, 2026J/K move · Enter open · S save

LLM Probe: Evaluating LLMs for Low-Resource Languages

Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang Nejdl

L3S Research Center, Leibniz University Hannover, Germany · Aksum University, Ethiopia

Multilingual Language Model Evaluation Low-Resource Language Processing Language Model Probing

Open paper AskPDF arXiv

Despite rapid advances in large language models (LLMs), their linguistic abilities in low-resource and morphologically rich languages are still not well understood due to limited annotated resources and the absence of standardized evaluation frameworks. This paper presents LLM Probe, a lexicon-based assessment framework designed to systematically evaluate the linguistic skills of LLMs in low-resource language environments. The framework analyzes models across four areas of language understanding: lexical alignment, part-of-speech recognition, morphosyntactic probing, and translation accuracy. To illustrate the framework, we create a manually annotated benchmark dataset using a low-resource Semitic language as a case study. The dataset comprises bilingual lexicons with linguistic annotations, including part-of-speech tags, grammatical gender, and morphosyntactic features, which demonstrate high inter-annotator agreement to ensure reliable annotations. We test a variety of models, including causal language models and sequence-to-sequence architectures. The results reveal notable differences in performance across various linguistic tasks: sequence-to-sequence models generally excel in morphosyntactic analysis and translation quality, whereas causal models demonstrate strong performance in lexical alignment but exhibit weaker translation accuracy. Our results emphasize the need for linguistically grounded evaluation to better understand LLM limitations in low-resource settings. We release LLM Probe and the accompanying benchmark dataset as open-source tools to promote reproducible benchmarking and to support the development of more inclusive multilingual language technologies.

Abstract

Explore similar work

LLM Probe: Evaluating LLMs for Low-Resource Languages

Are LLMs Ready to Replace Bangla Annotators?

Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic