Mining Large Language Models for Low-Resource Language Data: Comparing Elicitation Strategies for Hausa and Fongbe

Mahounan Pericles Adjovi; Roald Eiselen; Prasenjit Mitra

cs.CLApr 14, 2026

Mining Large Language Models for Low-Resource Language Data: Comparing Elicitation Strategies for Hausa and Fongbe

Authors: Mahounan Pericles Adjovi, Roald Eiselen, Prasenjit Mitra

Organizations: Carnegie Mellon University Africa, Kigali, Rwanda · Centre for Text Technology, North-West University, Potchefstroom, South Africa

Abstract

Large language models (LLMs) are trained on data contributed by low-resource language communities, yet the linguistic knowledge encoded in these models remains accessible only through commercial APIs. This paper investigates whether strategic prompting can extract usable text data from LLMs for two West African languages: Hausa (Afroasiatic, approximately 80 million speakers) and Fongbe (Niger-Congo, approximately 2 million speakers). We systematically compare six elicitation task types across two commercial LLMs (GPT-4o Mini and Gemini 2.5 Flash). GPT-4o Mini extracts 6-41 times more usable target-language words per API call than Gemini. Optimal strategies differ by language: Hausa benefits from functional text and dialogue, while Fongbe requires constrained generation prompts. We release all generated corpora and code.

Figures & tables

Task Type	Rationale	N
Creative Writing (poems, stories, folktales, songs, proverbs)	Tests deep cultural and linguistic knowledge; generates domain-diverse narrative text	25
Functional Text (letters, instructions, news, recipes, announcements)	Tests practical domain coverage; generates text useful for downstream NLP	25
Structured Knowledge (definitions, grammar examples, vocabulary lists, translations)	Tests metalinguistic knowledge; produces high-density lexical output	25
Dialogue (conversations, interviews, negotiations, family discussions)	Tests colloquial register and spoken-form generation	25
Topic Switching (domestic→sports, narrative shifts, knowledge switches)	Tests language maintenance robustness under topic changes	25
Constrained Gen. (vocab-constrained, no-code-switching, technical monolingual)	Tests ability to stay in target language under explicit constraints	25

Table 1: Elicitation task taxonomy: 6 types

\times

25 prompts = 150 per language.

	Gemini		GPT-4o Mini
Task	Fon	Hau	Fon	Hau
Creative	28/18	76/27	100/90	100/104
Functional	36/17	68/20	100/153	100/205
Structured	40/18	56/20	100/92	100/114
Dialogue	80/23	52/20	100/145	100/190
Topic Switch	4/15	88/73	100/157	100/190

Table 2: Output validity (% valid/avg. words) by task and model.

	Gemini		GPT-4o Mini
Task	Fon	Hau	Fon	Hau
Creative	60	92	48	100
Functional	68	96	40	100
Structured	40	72	52	100
Dialogue	12	76	60	100
Topic Switch	96	100	32	100

Table 3: Document-level target language detection (%) by GlotLID, per task and model.

Type-Token Ratio
	Gemini		GPT-4o Mini
Task	Fon	Hau	Fon	Hau
Creative	.93	.92	.58	.67
Functional	.88	.92	.48	.60
Structured	.96	.95	.60	.71
Dialogue	.88	.92	.46	.58

Table 4: Lexical diversity measured by Type-Token Ratio (TTR) and average vocabulary size per condition.

Model	Task	Fon	Hau
Gemini	Creative	0.0	20.7
	Functional	1.7	13.5
	Structured	0.9	5.8
	Dialogue	0.0	7.0
	Topic Switch	0.0	70.6
	Constrained	5.7	32.7

Table 5: Usable target-language words per API call. GPT-4o Mini is 6

\times

more efficient for Hausa, 41

\times

for Fongbe.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Extraction efficiency: usable target-language words per API call, by model, language, and task type. GPT-4o Mini dominates across all conditions; the gap is most extreme for Fongbe.

Figure 2: Document-level target language detection heatmap (GlotLID). Green = high fidelity; red = low. Hausa is uniformly high for GPT-4o Mini; Fongbe fidelity depends strongly on task type.

Figure 3: Fongbe diacritic analysis by task type. Left: diacritic-to-alphabetic ratio; Right: proportion of outputs containing any diacritics. Constrained generation reliably elicits diacritics from both models.

Figure 4: Character trigram cosine similarity between generated Hausa text and MasakhaNER 2.0 training text, used as a proxy for potential memorization. All values are well below 0.15 (dashed line), suggesting outputs represent novel generation rather than training data reproduction.

Figure 5: Sentence-level code-switching rates by model, language, and task type. Constrained generation consistently achieves the lowest code-switching. Fongbe shows much higher rates than Hausa across all tasks.

Model	Lang	Task	Valid%	Words	TTR	Hapax	Vocab	CS	LangConf	Quality
Gemini	fon	constrained	0.20	17.1	0.891	0.802	14.7	0.087	0.998	0.927
		creative	0.28	17.6	0.932	0.876	16.4	0.387	0.782	0.791
		dialogue	0.80	22.8	0.880	0.787	20.2	0.913	0.744	0.555
		functional	0.36	16.7	0.883	0.795	14.7	0.420	0.929	0.816
		structured	0.40	18.5	0.955	0.915	17.7	0.660	0.616	0.645
		topic switch	0.04	15.0	0.892	0.800	13.4	0.093	0.995	0.941

Appendix

Table 6: Full evaluation summary across all 24 conditions. CS = code-switching rate; LangConf = GlotLID language confidence score; Quality = composite score.

Explore similar work

Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability
Jun 20, 2026Mahounan Pericles Adjovi, Roald Eiselen, Prasenjit MitraLLM Evaluation Low-Resource MT
Carnegie Mellon University Africa, Kigali, Rwanda · North-West University, South Africa
From Script to Semantics: Prompting Strategies for African NLI
Jun 2, 2026Anuj Tiwari, Terry Oko-odion, Hannah NwokochaMultilingual Language Model Evaluation LLM Prompting
Noida Institute of Engineering and Technology1, India1 · ML Collective1,2,3, Nigeria2,3
From Speech to Text Corpora: Evaluating ASR-Based Data Acquisition for Low-Resource Fongbe and Hausa
Jun 20, 2026Mahounan Pericles Adjovi, Victor Olufemi, Roald Eiselen +1Automatic Speech Recognition Low-Resource Language Processing
Carnegie Mellon University Africa, Kigali, Rwanda · North-West University, South Africa

Jun 20, 2026J/K move · Enter open · S save

Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

Mahounan Pericles Adjovi, Roald Eiselen, Prasenjit Mitra

Carnegie Mellon University Africa, Kigali, Rwanda · North-West University, South Africa

LLM Evaluation Low-Resource MT MT Evaluation

Open paper AskPDF arXiv

We investigate the translation quality of current large language models (LLMs) for English-to-Hausa and English-to-Fongbe - two typologically distinct West African languages from the Afroasiatic and Niger-Congo families respectively - and evaluate whether standard automatic metrics reliably reflect human judgment for these low-resource languages. We evaluate four models (GPT-4o Mini, Claude Sonnet 4, Gemini 2.5 Flash, and Qwen2.5-7B) at progressive scales (500 to 10,000 sentences) using automatic metrics (BLEU, chrF++, TER, COMET, BERTScore) validated against native-speaker judgment. Our results reveal three key findings. First, translation quality varies substantially by language: Hausa achieves acceptable quality (human scores 4.0-4.5/5) while Fongbe achieves poor quality (1.0-2.2/5), with a consistent 3x BLEU gap across all systems. Second, model rankings differ by language - Gemini leads for Fongbe while GPT-4o leads for Hausa by human evaluation - indicating that performance on one low-resource African language does not predict performance on another. Third, metric-human correlation varies dramatically: perfect rank correlation for Fongbe (rho=1.0) but weak correlation for Hausa (rho=0.5), where human evaluators preferred GPT-4o despite all automatic metrics ranking Claude first. We further show that neural metrics like BERTScore exhibit embedding collapse (within-language similarity >0.99) for both languages, limiting their ability to differentiate translation quality. Based on these findings, we recommend multi-metric evaluation for low-resource African languages, with particular caution when interpreting neural metrics. We establish that minimum sample sizes of n=2,500 sentences are required for stable system rankings, as smaller samples produced artifact findings that reversed at scale.

Abstract

Explore similar work

Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

From Script to Semantics: Prompting Strategies for African NLI

From Speech to Text Corpora: Evaluating ASR-Based Data Acquisition for Low-Resource Fongbe and Hausa