IR in low-resource languages remains limited by the scarcity of high-quality, task-specific annotated datasets. Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity. This work presents a Bangla IR dataset constructed using a BETA-labeling framework involving multiple LLM annotators from diverse model families. The framework incorporates contextual alignment, consistency checks, and majority agreement, followed by human evaluation to verify label quality. Beyond dataset creation, we examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation. Using LLM-based translation across multiple language pairs, we experimented on meaning preservation and task validity between source and translated datasets. Our experiment reveal substantial variation across languages, reflecting language-dependent biases and inconsistent semantic preservation that directly affect the reliability of cross-lingual dataset reuse. Overall, this study highlights both the potential and limitations of LLM-assisted dataset creation for low-resource IR. It provides empirical evidence of the risks associated with cross-lingual dataset reuse and offers practical guidance for constructing more reliable benchmarks and evaluation pipelines in low-resource language settings.
Figures & tables
Figure 1: Overview of BETA-Labeling and LRL-to-LRL cross-lingual reusability evaluation.
Figure 2: BETA-labeling framework to annotate implicit feature-based data in low-resource languages.
Language Pair
Bangla_Lite
Bangla_Culture
GPT-4o Mini
Gemini 2.5 Flash Lite
DeepSeek R1 7B
GPT-4o Mini
Gemini 2.5 Flash Lite
DeepSeek R1 7B
CS
BL
ME
CS
BL
ME
CS
BL
ME
CS
BL
ME
CS
BL
ME
CS
BL
ME
BN → EN
78.9
12.3
52.4
78.0
25.0
51.2
23.4
60.0
15.5
93.1
47.4
71.8
92.6
76.3
67.5
33.3
45.9
21.4
BN → ZH
72.7
3.16
1.50
69.5
10.8
1.39
30.0
7.04
1.60
93.4
64.2
6.16
91.4
39.1
5.68
35.1
0.89
1.25
BN → AR
36.4
33.0
33.1
36.4
11.5
29.3
46.9
8.89
5.27
75.6
30.7
55.0
74.8
24.7
48.6
45.8
5.15
3.94
BN → RU
71.7
8.01
44.4
70.6
55.0
42.4
31.1
5.97
5.68
92.5
38.8
63.1
91.8
83.7
58.0
35.2
11.0
5.82
Table 3: Cross-lingual one-hop translation evaluation of three SOTA LLMs (GPT-4o mini, Gemini 2.5 Flash Lite, and DeepSeek R1 7B) on the Bangla_Lite and Bangla_History datasets. In this table, CS, BL, and ME denote Cosine Similarity, BLEU, and METEOR, respectively. BN, EN, ZH, AR, and RU represent Bangla, English, Chinese, Arabic, and Russian, respectively.
Language Pair
Bangla_Lite
Bangla_Culture
GPT-4o Mini
Gemini 2.5 Flash Lite
DeepSeek R1 7B
GPT-4o Mini
Gemini 2.5 Flash Lite
DeepSeek R1 7B
BS
CS
JS
BS
CS
JS
BS
CS
JS
BS
CS
JS
BS
CS
JS
BS
CS
JS
BN → EN
67.6
59.5
10.9
66.9
57.3
9.7
56.1
38.3
6.12
72.3
71.0
13.7
68.5
66.4
11.4
57.6
44.6
7.92
BN → ZH
68.4
63.3
0.01
67.5
60.1
0.0
54.9
34.1
0.16
73.2
77.2
0.02
69.5
72.5
0.02
56.5
34.9
0.09
BN → AR
67.5
43.2
6.40
67.5
41.8
4.8
53.3
41.1
0.56
70.8
66.1
8.64
68.8
61.6
6.33
56.9
39.8
0.68
BN → RU
67.1
60.3
7.45
67.2
59.6
6.1
52.6
34.1
0.72
71.0
76.3
8.71
68.5
71.2
6.21
57.3
36.4
0.42
Table 4: Cross-lingual one-hop implicit context evaluation of three SOTA LLMs (GPT-4o mini, Gemini 2.5 Flash Lite, and DeepSeek R1 7B) on the Bangla_Lite and Bangla_History datasets. In this table, BS, CS, and JS denote BERTScore, Cosine Similarity, and Jaccard Similarity, respectively. BN, EN, ZH, AR, and RU represent Bangla, English, Chinese, Arabic, and Russian, respectively.
Despite rapid advances in large language models (LLMs), their linguistic abilities in low-resource and morphologically rich languages are still not well understood due to limited annotated resources and the absence of standardized evaluation frameworks. This paper presents LLM Probe, a lexicon-based assessment framework designed to systematically evaluate the linguistic skills of LLMs in low-resource language environments. The framework analyzes models across four areas of language understanding: lexical alignment, part-of-speech recognition, morphosyntactic probing, and translation accuracy. To illustrate the framework, we create a manually annotated benchmark dataset using a low-resource Semitic language as a case study. The dataset comprises bilingual lexicons with linguistic annotations, including part-of-speech tags, grammatical gender, and morphosyntactic features, which demonstrate high inter-annotator agreement to ensure reliable annotations. We test a variety of models, including causal language models and sequence-to-sequence architectures. The results reveal notable differences in performance across various linguistic tasks: sequence-to-sequence models generally excel in morphosyntactic analysis and translation quality, whereas causal models demonstrate strong performance in lexical alignment but exhibit weaker translation accuracy. Our results emphasize the need for linguistically grounded evaluation to better understand LLM limitations in low-resource settings. We release LLM Probe and the accompanying benchmark dataset as open-source tools to promote reproducible benchmarking and to support the development of more inclusive multilingual language technologies.
Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang Nejdl
L3S Research Center, Leibniz University Hannover, Germany · Aksum University, Ethiopia
Large Language Models (LLMs) are increasingly used as automated annotators to scale dataset creation, yet their reliability as unbiased annotators--especially for low-resource and identity-sensitive settings--remains poorly understood. In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences. We conduct a systematic benchmark of 17 LLMs using a unified evaluation framework. Our analysis uncovers annotator bias and substantial instability in model judgments. Surprisingly, increased model scale does not guarantee improved annotation quality--smaller, more task-aligned models frequently exhibit more consistent behavior than their larger counterparts. These results highlight important limitations of current LLMs for sensitive annotation tasks in low-resource languages and underscore the need for careful evaluation before deployment.
Md. Najib Hasan, Touseef Hasan, Souvika Sarkar
A2I Lab, School of Computing, Wichita State University
This paper evaluates current Large Language Model (LLM) benchmarking for Icelandic, identifies problems, and calls for improved evaluation methods in low/medium-resource languages in particular. We show that benchmarks that include synthetic or machine-translated data that have not been verified in any way, commonly contain severely flawed test examples that are likely to skew the results and undermine the tests' validity. We warn against the use of such methods without verification in low/medium-resource settings as the translation quality can, at best, only be as good as MT quality for a given language at any given time. Indeed, the results of our quantitative error analysis on existing benchmarks for Icelandic show clear differences between human-authored/-translated benchmarks vs. synthetic or machine-translated benchmarks.
Finnur Ágúst Ingimundarson, Steinunn Rut Friðriksdóttir, Bjarki Ármannsson +2
University of Zürich · University of Iceland · The Árni Magnússon Institute for Icelandic Studies