YoNER: A New Yorùbá Multi-domain Named Entity Recognition Dataset
Authors: Peace Busola Falola, Jesujoba O. Alabi, Solomon O. Akinola, Folashade T. Ogunajo, Emmanuel Oluwadunsin Alabi, David Ifeoluwa Adelani
Organizations: University of Ibadan, Nigeria · Saarland University, Germany · Atiba University, Nigeria · Mila - Quebec AI Institute, McGill University, and Canada CIFAR AI Chair
Named Entity Recognition (NER) is a foundational NLP task, yet research in Yorùbá has been constrained by limited and domain-specific resources. Existing resources, such as MasakhaNER (a manually annotated news-domain corpus) and WikiAnn (automatically created from Wikipedia), are valuable but restricted in domain coverage. To address this gap, we present YoNER, a new multidomain Yorùbá NER dataset that extends entity coverage beyond news and Wikipedia. The dataset comprises about 5,000 sentences and 100,000 tokens collected from five domains including Bible, Blogs, Movies, Radio broadcast and Wikipedia, and annotated with three entity types: Person (PER), Organization (ORG) and Location (LOC), following CoNLL-style guidelines. Annotation was conducted manually by three native Yorùbá speakers, with an inter-annotator agreement of over 0.70, ensuring high quality and consistency. We benchmark several transformer encoder models using cross-domain experiments with MasakhaNER 2.0, and we also assess the effect of few-shot in-domain data using YoNER and cross-lingual setups with English datasets. Our results show that African-centric models outperform general multilingual models for Yorùbá, but cross-domain performance drops substantially, particularly for blogs and movie domains. Furthermore, we observed that closely related formal domains, such as news and Wikipedia, transfer more effectively. In addition, we introduce a new Yorùbá-specific language model (OyoBERT) that outperforms multilingual models in in-domain evaluation. We publicly release the YoNER dataset and pretrained OyoBERT models to support future research on Yorùbá natural language processing.
Figures & tables
Domain
Yorùbá Sentence
Bible
Àwọn ọmọkùnrin Simeoni , Jemueli , Jamini , Ohadi , Jakini , Sohari , Saulu , tí ìyá rẹ jẹ́ ọmọbìnrin ará Kenaani .
Blog
Ipò obinrin kò pin si idi àdìrò àti inú ilé yòkù gẹ́gẹ́ bi Olórí Òṣèlú Nigeria , Muhammadu Buhari ti sọ.
Ìràwọ̀ agbábọọ̀lù tìnú ẹgbẹ́ ìkọ Liverpool Salah nló tí ńlérí wí pé ẹgbẹ́ agbábọọ̀lù náà yóò gba ìfẹ ẹ̀yẹ ní sáà tó ńbọ̀.
Wikipedia
Òun ló gba àmì ẹ̀yẹ PEN Pinter Prize ní ọdún 2018 Iléẹ̀kọ́ sẹ́kọ́ndìrì Yunifásítì ti Naijiria , Nsukka ni Adichie ti parí ẹ̀kọ́ sẹ́kọ́ndìrì rẹ̀ níbi tí ó ti ọ̀pọ̀lọpọ̀ àmì ẹ̀yẹ ajemọ́ akadá.
Table 1: Examples of named entities across sentences from different Yorùbá domains in YoNER. ( PER , ORG , LOC ).
Table 3: Inter-annotator agreement (Fleiss’ κ ) across Yoruba NER domains.
Topic
News Topic
Twitter
Movie
Model
Size
Class.
Class.
Sentiment
Sentiment
NER
POS
Avg.
Baselines
AfriBERTa Ogueji et al. (2021)
126M
70.6
90.3
72.9
90.9
87.7
94.5
84.5
AfroXLMR-large ( Alabi et al., 2022 )
550M
74.8
94.0
74.1
84.3
89.3
95.0
85.3
AfroXLMR-large-76L ( Adelani et al., 2024 )
550M
79.9
94.7
75.1
88.0
88.4
95.2
86.8
YoBERT-base (ours)
110M
69.2
92.1
71.1
90.0
85.0
94.0
83.6
Table 4: Evaluation of OyoBERT models and other multilingual baselines on four sequence-level and two token-level classification tasks for Yorùbá.
Figure 1: Frequency of Entity Types By Domain in YoNER dataset.
Model
Bible
Blog
Movies
Radio
Wiki
News
Avg.
MasakhaNER 2.0
OyoBERT-large
69.80 1.10
54.30 1.23
42.17 1.53
72.50 3.03
74.14 1.31
87.78 0.49
66.78
AfriBERTa
58.85 1.72
52.77 1.18
44.47 1.76
75.26 0.72
68.96 0.95
87.42 0.58
64.62
AfriBERTa-V2
71.01 1.22
50.38 2.10
47.12 1.84
76.47 2.11
75.01 0.45
87.22 0.29
67.87
mBERT
61.94 1.54
43.38 3.71
25.72 4.34
76.10 1.22
70.10 0.63
84.30 0.35
60.26
XLM-R-base
63.20 1.21
40.22 1.54
42.40 3.47
68.54 1.52
66.28 1.18
84.19 0.51
60.81
Table 5: F1 score (%) for cross-domain transfer to the YoNER dataset from the News domain. Scores are averaged over five runs. All models are fine-tuned on the training split of MasakhaNER 2.0.
Figure 2: Cross-domain evaluation of NER model trained on each YoNER domain and on domain-specific English NER datasets. English-trained models are evaluated on YoNER domains to assess cross-lingual transfer. Average over 5 runs.
Model
Bible
Blog
Movies
Radio
Wiki
Avg.
In-domain
OyoBERT-large
81.47
7.78
34.06
65.76
43.09
46.43
AfroXLMR-large
84.85
5.67
18.78
50.82
50.26
42.08
Multi-domain
OyoBERT-large
85.00
71.53
56.10
89.60
76.37
75.72
AfroXLMR-large
89.63
76.09
53.72
90.84
79.64
77.98
Table 6: F1 score (%) for In-domain evaluation of each of the YoNER domains. Average is over 5 runs.
Model
CoNLL
WB
BC
Wikiann
Avg.
In-domain (all data)
RoBERTa-large
93.14
64.37
88.16
82.48
82.04
XLM-R-large
93.70
63.12
91.37
82.74
82.73
In-domain (1000 examples)
RoBERTa-large
74.33
1.79
40.98
54.14
42.81
XLM-R-large
53.59
0.00
18.23
40.32
28.04
Table 7: F1 score (%) for In-domain evaluation of each of the English domains. Average is over 5 runs.
Domain
PER
LOC
ORG
OyoBERT-large
Bible
0.89
0.80
0.00
Blog
0.70
0.80
0.42
Movies
0.61
0.41
0.00
Radio
0.88
0.95
0.80
Wiki
0.87
0.84
0.53
Table 8: F1 score (%) for multi-domain training on YoNER. Average is over 5 runs.
Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resource languages, but in the absence of target language benchmarks, it is unclear which auxiliary language selection strategy leads to the best transfer. We introduce RunyaNER, the first publicly available NER benchmark for the East African language Runyankore, and use it to investigate the choice of which languages to use for transfer. Created with a semi-automated pipeline and fully manually verified, RunyaNER contains over 237k annotated words across 30k sentences. We benchmark pretrained models on RunyaNER, establishing that our dataset is of sufficient quality and size to produce effective Runyankore NER models. We then use RunyaNER to investigate auxiliary language selection in cross-lingual zero-shot and multilingual fine-tuning settings. Our experiments show that while transfer performance is highly sensitive to auxiliary language selection, embedding-based measures computed from labelled training spans correlate more strongly with downstream transfer performance than traditional linguistic features based on metadata or typology. By releasing RunyaNER and providing a systematic analysis of auxiliary language selection strategies, this work contributes both a new benchmark resource and practical insights for multilingual transfer in low-resource settings.
Prosper Arineitwe Asiimwe, Francois Meyer, Jan Buys
Department of Computer Science, University of Cape Town
Sarcasm detection poses a fundamental challenge in computational semantics, requiring models to resolve disparities between literal and intended meaning. The challenge is amplified in low-resource languages where annotated datasets are scarce or nonexistent. We present Yor-Sarc, the first gold-standard dataset for sarcasm detection in Yorùbá, a tonal Niger-Congo language spoken by over 50 million people. The dataset comprises 436 instances annotated by three native speakers from diverse dialectal backgrounds using an annotation protocol specifically designed for Yorùbá sarcasm by taking culture into account. This protocol incorporates context-sensitive interpretation and community-informed guidelines and is accompanied by a comprehensive analysis of inter-annotator agreement to support replication in other African languages. Substantial to almost perfect agreement was achieved (Fleiss' κ=0.7660; pairwise Cohen's κ=0.6732--0.8743), with 83.3% unanimous consensus. One annotator pair achieved almost perfect agreement (κ=0.8743; 93.8% raw agreement), exceeding a number of reported benchmarks for English sarcasm research works. The remaining 16.7% majority-agreement cases are preserved as soft labels for uncertainty-aware modelling. Yor-Sarc\footnote{https://github.com/toheebadura/yor-sarc} is expected to facilitate research on semantic interpretation and culturally informed NLP for low-resource African languages.
Toheeb Aduramomi Jimoh, Tabea De Wille, Nikola S. Nikolov
Department of Computer Science and Information Systems, University of Limerick, Castletroy, V94 T9PX, Limerick, Ireland
ÌròyìnSpeech is a 42-hour, 80-speaker Yorùbá read-speech corpus whose audio has been distributed by ELRA since 2024. This paper describes the release of its text component: 24,905 unique, hand-verified, tone-marked Yorùbá sentences (275,897 tokens; 15,687 types), curated in 2022 as recording prompts. Roughly 11,000 sentences were adapted from openly licensed news material; the remainder were written in-house to broaden coverage beyond the religious translation that dominates existing Yorùbá corpora. Every sentence was checked by hand for tone-mark accuracy, edited for read-aloud clarity and a neutral register, and localised so that non-Yorùbá personal and place names appear in Yorùbá form. Preparing the text for release surfaced systematic Unicode normalisation failures affecting more than 60% of lines (with precomposed and decomposed forms of the same letter co-occurring within single sentences) which we document and correct. The corpus supports diacritic restoration, grapheme-to-phoneme conversion, TTS front-end development and orthographic research, and serves as a validated prompt set for new recording.
Kola Tubosun, Aanuoluwapo Aremu, Tolulope Ogunremi +2
Stanford University · Niger-Volta LTI · Mila / McGill University