BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition

Wazir Ali; Adeeb Noor; Sanaullah Mahar; Alia; Muhammad Mazhar Younas

cs.CLApr 3, 2026

BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition

Authors: Wazir Ali, Adeeb Noor, Sanaullah Mahar, Alia, Muhammad Mazhar Younas

Organizations: Department of Data Science, Quaid-e-Awam University of Engineering, Sciences & Technology, Pakistan · Department of Information Technology, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia · Department of Artificial Intelligence, Aror University, Sukkur, Pakistan

Abstract

In this article, we present a gold-standard benchmark dataset for Biomedical Urdu Named Entity Recognition (BioUNER), developed by crawling health-related articles from online Urdu news portals, medical prescriptions, and hospital health blogs and websites. After preprocessing, three native annotators with familiarity in the medical domain participated in the annotation process using the Doccano text annotation tool and annotated 153K tokens. Following annotation, the proposed BioiUNER dataset was evaluated both intrinsically and extrinsically. An inter-annotator agreement score of 0.78 was achieved, thereby validating the dataset as gold-standard quality. To demonstrate the utility and benchmarking capability of the dataset, we evaluated several machine learning and deep learning models, including Support Vector Machines (SVM), Long Short-Term Memory networks (LSTM), Multilingual BERT (mBERT), and XLM-RoBERTa. The gold-standard BioUNER dataset serves as a reliable benchmark and a valuable addition to Urdu language processing resources.

Figures & tables

Figure 1: An example of annotation of BioUNER using Doccano text annotation tool

Table 1: An example of annotated BioUNER dataset from token 87 to 104

Entity Type	Count
Disease	8959
Chemical	2557
Drug	2185
Protein	604
CellLine	520
Gene	248

Table 2: The complete entity type Counts in BioUNER dataset

Tag	Count
S-*	5206
B-*	3888
I-*	2094
E-*	3883

Table 3: The overall statistics of BIES-Tag Counts in BioUNER dataset

Model	Precision	Recall	F1-score
SVM	0.65	0.66	0.65
CRF	0.93	0.93	0.93
LSTM	0.95	0.95	0.95

Table 4: Performance of baseline models evaluated using standard metrics—precision, recall, and F1-score. Boldfaced values indicate the best results.

Model	Precision	Recall	F1-score
mBERT	0.72	0.71	0.72
XLM-RoBERTa	0.72	0.73	0.73

Table 5: Performance of fine-tuned transformer models on the BioUNER dataset. Boldfaced values indicate the best results.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 110 papers

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark
Apr 14, 2026Terra Blevins, Stephen Mayhew, Marek Šuppa +11Named Entity Recognition Multilingual Language Model Evaluation
Northeastern University, USA · Duolingo, USA · Comenius University Bratislava, Slovakia +10
DunbaaBERT: From Sacrifice to Semantics
May 26, 2026Iffat Maab, Waleed Jamil, Raphael SchmittLanguage Model Pretraining Language Modeling
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1
UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding
Jun 5, 2026Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal +3Multilingual Language Model Evaluation Multilingual QA
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

Apr 14, 2026J/K move · Enter open · S save

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

Terra Blevins, Stephen Mayhew, Marek Šuppa, Hila Gonen+10

Northeastern University, USA · Duolingo, USA · Comenius University Bratislava, Slovakia+10

Named Entity Recognition Multilingual Language Model Evaluation

Open paper AskPDF arXiv

While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most languages to interrogate these assumptions remain scarce. The Universal NER project, now entering its fourth year, is dedicated to building gold-standard multilingual Named Entity Recognition (NER) benchmark datasets. Inspired by existing massively multilingual efforts for other core NLP tasks (e.g., Universal Dependencies), the project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans. The first installment (UNER v1) was released in 2024, and the project has continued and expanded since then, with various organizers, annotators, and collaborators in an active community.

Abstract

Named Entity Recognition

Explore similar work

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

DunbaaBERT: From Sacrifice to Semantics

UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding