YoNER: A New Yorùbá Multi-domain Named Entity Recognition Dataset
Authors: Peace Busola Falola, Jesujoba O. Alabi, Solomon O. Akinola, Folashade T. Ogunajo, Emmanuel Oluwadunsin Alabi, David Ifeoluwa Adelani
Organizations: University of Ibadan, Nigeria · Saarland University, Germany · Atiba University, Nigeria · Mila - Quebec AI Institute, McGill University, and Canada CIFAR AI Chair
Named Entity Recognition (NER) is a foundational NLP task, yet research in Yorùbá has been constrained by limited and domain-specific resources. Existing resources, such as MasakhaNER (a manually annotated news-domain corpus) and WikiAnn (automatically created from Wikipedia), are valuable but restricted in domain coverage. To address this gap, we present YoNER, a new multidomain Yorùbá NER dataset that extends entity coverage beyond news and Wikipedia. The dataset comprises about 5,000 sentences and 100,000 tokens collected from five domains including Bible, Blogs, Movies, Radio broadcast and Wikipedia, and annotated with three entity types: Person (PER), Organization (ORG) and Location (LOC), following CoNLL-style guidelines. Annotation was conducted manually by three native Yorùbá speakers, with an inter-annotator agreement of over 0.70, ensuring high quality and consistency. We benchmark several transformer encoder models using cross-domain experiments with MasakhaNER 2.0, and we also assess the effect of few-shot in-domain data using YoNER and cross-lingual setups with English datasets. Our results show that African-centric models outperform general multilingual models for Yorùbá, but cross-domain performance drops substantially, particularly for blogs and movie domains. Furthermore, we observed that closely related formal domains, such as news and Wikipedia, transfer more effectively. In addition, we introduce a new Yorùbá-specific language model (OyoBERT) that outperforms multilingual models in in-domain evaluation. We publicly release the YoNER dataset and pretrained OyoBERT models to support future research on Yorùbá natural language processing.
Figures & tables
Domain
Yorùbá Sentence
Bible
Àwọn ọmọkùnrin Simeoni , Jemueli , Jamini , Ohadi , Jakini , Sohari , Saulu , tí ìyá rẹ jẹ́ ọmọbìnrin ará Kenaani .
Blog
Ipò obinrin kò pin si idi àdìrò àti inú ilé yòkù gẹ́gẹ́ bi Olórí Òṣèlú Nigeria , Muhammadu Buhari ti sọ.
Ìràwọ̀ agbábọọ̀lù tìnú ẹgbẹ́ ìkọ Liverpool Salah nló tí ńlérí wí pé ẹgbẹ́ agbábọọ̀lù náà yóò gba ìfẹ ẹ̀yẹ ní sáà tó ńbọ̀.
Wikipedia
Òun ló gba àmì ẹ̀yẹ PEN Pinter Prize ní ọdún 2018 Iléẹ̀kọ́ sẹ́kọ́ndìrì Yunifásítì ti Naijiria , Nsukka ni Adichie ti parí ẹ̀kọ́ sẹ́kọ́ndìrì rẹ̀ níbi tí ó ti ọ̀pọ̀lọpọ̀ àmì ẹ̀yẹ ajemọ́ akadá.
Table 1: Examples of named entities across sentences from different Yorùbá domains in YoNER. ( PER , ORG , LOC ).
Table 3: Inter-annotator agreement (Fleiss’ κ ) across Yoruba NER domains.
Topic
News Topic
Twitter
Movie
Model
Size
Class.
Class.
Sentiment
Sentiment
NER
POS
Avg.
Baselines
AfriBERTa Ogueji et al. (2021)
126M
70.6
90.3
72.9
90.9
87.7
94.5
84.5
AfroXLMR-large ( Alabi et al., 2022 )
550M
74.8
94.0
74.1
84.3
89.3
95.0
85.3
AfroXLMR-large-76L ( Adelani et al., 2024 )
550M
79.9
94.7
75.1
88.0
88.4
95.2
86.8
YoBERT-base (ours)
110M
69.2
92.1
71.1
90.0
85.0
94.0
83.6
Table 4: Evaluation of OyoBERT models and other multilingual baselines on four sequence-level and two token-level classification tasks for Yorùbá.
Figure 1: Frequency of Entity Types By Domain in YoNER dataset.
Model
Bible
Blog
Movies
Radio
Wiki
News
Avg.
MasakhaNER 2.0
OyoBERT-large
69.80 1.10
54.30 1.23
42.17 1.53
72.50 3.03
74.14 1.31
87.78 0.49
66.78
AfriBERTa
58.85 1.72
52.77 1.18
44.47 1.76
75.26 0.72
68.96 0.95
87.42 0.58
64.62
AfriBERTa-V2
71.01 1.22
50.38 2.10
47.12 1.84
76.47 2.11
75.01 0.45
87.22 0.29
67.87
mBERT
61.94 1.54
43.38 3.71
25.72 4.34
76.10 1.22
70.10 0.63
84.30 0.35
60.26
XLM-R-base
63.20 1.21
40.22 1.54
42.40 3.47
68.54 1.52
66.28 1.18
84.19 0.51
60.81
Table 5: F1 score (%) for cross-domain transfer to the YoNER dataset from the News domain. Scores are averaged over five runs. All models are fine-tuned on the training split of MasakhaNER 2.0.
Figure 2: Cross-domain evaluation of NER model trained on each YoNER domain and on domain-specific English NER datasets. English-trained models are evaluated on YoNER domains to assess cross-lingual transfer. Average over 5 runs.
Model
Bible
Blog
Movies
Radio
Wiki
Avg.
In-domain
OyoBERT-large
81.47
7.78
34.06
65.76
43.09
46.43
AfroXLMR-large
84.85
5.67
18.78
50.82
50.26
42.08
Multi-domain
OyoBERT-large
85.00
71.53
56.10
89.60
76.37
75.72
AfroXLMR-large
89.63
76.09
53.72
90.84
79.64
77.98
Table 6: F1 score (%) for In-domain evaluation of each of the YoNER domains. Average is over 5 runs.
Model
CoNLL
WB
BC
Wikiann
Avg.
In-domain (all data)
RoBERTa-large
93.14
64.37
88.16
82.48
82.04
XLM-R-large
93.70
63.12
91.37
82.74
82.73
In-domain (1000 examples)
RoBERTa-large
74.33
1.79
40.98
54.14
42.81
XLM-R-large
53.59
0.00
18.23
40.32
28.04
Table 7: F1 score (%) for In-domain evaluation of each of the English domains. Average is over 5 runs.
Domain
PER
LOC
ORG
OyoBERT-large
Bible
0.89
0.80
0.00
Blog
0.70
0.80
0.42
Movies
0.61
0.41
0.00
Radio
0.88
0.95
0.80
Wiki
0.87
0.84
0.53
Table 8: F1 score (%) for multi-domain training on YoNER. Average is over 5 runs.