Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

Terra Blevins; Stephen Mayhew; Marek Šuppa; Hila Gonen; Shachar Mirkin; Vasile Pais; Kaja Dobrovoljc; Voula Giouli

cs.CLApr 14, 2026

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

Authors: Terra Blevins, Stephen Mayhew, Marek Šuppa, Hila Gonen, Shachar Mirkin, Vasile Pais, Kaja Dobrovoljc, Voula Giouli,

Organizations: Northeastern University, USA · Duolingo, USA · Comenius University Bratislava, Slovakia · University of British Columbia, Canada · Alpinference, France · Research Institute for Artificial Intelligence, Romanian Academy, Romania · University of Ljubljana, Slovenia · Aristotle University of Thessaloniki / ILSP, Athena Research Center, Greece · Universitas Pelita Harapan, Indonesia · Seoul National University, South Korea · Independent Researcher, South Korea · Democritus University of Thrace, Greece · Ben-Gurion University of the Negev, Israel

Abstract

While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most languages to interrogate these assumptions remain scarce. The Universal NER project, now entering its fourth year, is dedicated to building gold-standard multilingual Named Entity Recognition (NER) benchmark datasets. Inspired by existing massively multilingual efforts for other core NLP tasks (e.g., Universal Dependencies), the project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans. The first installment (UNER v1) was released in 2024, and the project has continued and expanded since then, with various organizers, annotators, and collaborators in an active community.

Figures & tables

Figure 1: Comparison of the dataset statistics for UNER v1 and UNER v2.

Language		Dataset
el	Greek	gdt	Greek Dependency Treebank
he	Hebrew	htb	Hebrew Dependency Treebank
nno	Norwegian Nynorsk	ndt	Norwegian Dependency Treebank
nob	Norwegian Bokmål	ndt	Norwegian Dependency Treebank
sl	Slovenian	ssj	Slovenian SSJ Treebank
sv	Swedish	lines	Swedish LinES Treebank

Table 1: Languages and associated original dataset names used in UNER v2.

		Sentences				Entities				Tokens
Lang.	Dataset	Train	Dev	Test	All	Train	Dev	Test	All	Train	Dev	Test	All
el	gdt	1,662	403	456	2,521	1,551	501	436	2,488	42,326	10,443	10,672	63,441
he	htb	5,241	484	491	6,216	6,013	434	439	6,886	137,717	11,412	12,282	161,411
nno	ndt	14,174	1,890	1,511	17,575	10,348	1,111	897	12,356	245,330	31,250	24,773	301,353
nob	ndt	15,696	2,409	1,939	20,044	10,062	1,438	1,259	12,759	243,886	36,369	29,966	310,221
sl	ssj	10,903	1,250	1,282	13,435	7,293	907	724	8,924	215,155	26,500	25,442	267,097

Table 2: Dataset statistics for the new benchmarks included in UNER v2.

				Entity Dist. (%)
Data Source	Lang.	Dataset	Domains	LOC	ORG	PER
Prokopidis and Papageorgiou (2017)	el	gdt	news, spoken, wiki	38.0%	41.5%	20.5%
Tsarfaty (2013) ; McDonald et al. (2013)	he	htb	news	24.0%	39.7%	36.3%
Øvrelid and Hohle (2016)	nno	ndt	blog, news, nonfiction	28.4%	30.1%	41.5%
Øvrelid and Hohle (2016)	nob	ndt	blog, news, nonfiction	27.3%	31.6%	41.1%
Dobrovoljc et al. (2017)	sl	ssj	fiction, news, nonfiction	34.1%	22.1%	43.8%

Table 3: Domains and distribution of entity types for datasets in UNER v2. Domains are categorized for the underlying UD datasets at https://universaldependencies.org/ or author descriptions if not included in UD.

		Train				Dev				Test
Lang.	Dataset	LOC	ORG	PER	% Docs	LOC	ORG	PER	% Docs	LOC	ORG	PER	% Docs
el	gdt	0.750	0.556	0.962	25%	0.843	0.607	0.883	25%	0.750	0.493	0.923	26.9%
he	htb	0.777	0.785	0.839	29.7%	0.741	0.664	0.968	100%	0.746	0.831	0.914	100%
sv	lines	**	**	**	**	0.927	0.526	0.974	100%	0.571	0.667	0.962	25%
cs	pud	–	–	–	–	–	–	–	–	0.656	0.840	0.775	5.3%
id	pud	–	–	–	–	–	–	–	–	0.629	0.590	0.930	8.8%

Table 4: Inter-annotator agreement (IAA) for the new benchmarks included in UNER v2. – indicates that the data split does not exist, while ** indicates that IAA was not collected. IAA was also not calculated for datasets transferred into UNER v2: nno and nob ndt , sl ssj , and ro legalnero .

Figure 2: A cross-lingual comparison of UNER annotations on top of parallel text (PUD). We consider the overlap of datasets from UNER v1 and v2. Left : The overall tag distribution in each PUD dataset. Center : Sentence-level agreement between languages for entity count. Right : Sentence-level agreement on entity count within each class ( loc , org , per ) between languages.

Figure 3: Experimental results for in-language and cross-lingual UNER performance with XLM-R ${}_{\text{large}}$ .

Figure 4: F1 score comparison across 19 multilingual NER datasets (test sets) for three LLMs against human Inter-Annotator Agreement (IAA) baseline. Bold values indicate the best-performing model for each dataset; underlined values indicate scores exceeding human agreement.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 110 papers

What Matters When Building Universal Multilingual Named Entity Recognition Models?
Jan 9, 2026Jonas Golde, Patrick Haller, Alan AkbikNamed Entity Recognition Multilingual Language Models
Humboldt Universität zu Berlin
BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition
Apr 3, 2026Wazir Ali, Adeeb Noor, Sanaullah Mahar +2Named Entity Recognition Clinical NLP
Department of Data Science, Quaid-e-Awam University of Engineering, Sciences & Technology, Pakistan · Department of Information Technology, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia · Department of Artificial Intelligence, Aror University, Sukkur, Pakistan
RunyaNER: Auxiliary Language Selection for Runyankore NER
Sep 29, 2026Prosper Arineitwe Asiimwe, Francois Meyer, Jan BuysNamed Entity Recognition Multilingual Language Model Evaluation
Department of Computer Science, University of Cape Town

Jan 9, 2026J/K move · Enter open · S save

What Matters When Building Universal Multilingual Named Entity Recognition Models?

Jonas Golde, Patrick Haller, Alan Akbik

Humboldt Universität zu Berlin

Named Entity Recognition Multilingual Language Models Multilingual Language Model Evaluation Cross-Lingual Transfer

Open paper AskPDF arXiv

Recent progress in universal multilingual named entity recognition (NER) has been driven by multilingual transformer models, task-specific architectures, custom loss functions, and large-scale training datasets. However, despite substantial prior work, we find that many design decisions for such models are made without systematic justification, with individual components evaluated only in combination rather than in isolation. We argue that this impedes progress in the field by making it difficult to identify which choices improve multilingual generalization. In this work, we conduct an extensive empirical evaluation on transformer backbones, architectures, training objectives, data composition, and threshold selection for zero-shot, multilingual NER models. Building on these findings, we present Otter, a small encoder model that achieves consistent improvements over strong multilingual NER baselines, outperforming similarly sized models by 5.7 percentage points in F1. Further, it remains within 1.4 points of Qwen3-32B and 5.4 points below Gemma3-27B, while being roughly 90x smaller and one to two orders of magnitude faster at inference.

Abstract

Named Entity Recognition

Explore similar work

What Matters When Building Universal Multilingual Named Entity Recognition Models?

BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition

RunyaNER: Auxiliary Language Selection for Runyankore NER