Learning Nested Named Entity Recognition from Flat Annotations

Igor Rozhkov; Natalia Loukachevitch

cs.CLFeb 28, 2026

Learning Nested Named Entity Recognition from Flat Annotations

Authors: Igor Rozhkov, Natalia Loukachevitch

Organizations: Lomonosov Moscow State University Leninskie Gory, 1/4, Moscow, Russia

Abstract

Nested named entity recognition identifies entities contained within other entities, but requires expensive multi-level annotation. While flat NER corpora exist abundantly, nested resources remain scarce. We investigate whether models can learn nested structure from flat annotations alone, evaluating four approaches: string inclusions (substring matching), entity corruption (pseudo-nested data), flat neutralization (reducing false negative signal), and a hybrid fine-tuned + LLM pipeline. On NEREL, a Russian benchmark with 29 entity types where 21% of entities are nested, our best combined method achieves 26.37% inner F1, closing 40% of the gap to full nested supervision. Code is available at https://github.com/fulstock/Learning-from-Flat-Annotations.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of methods for learning nested NER from flat annotations. (a) Fine-tuned approaches augment flat data with pseudo-nested signal (inclusions, entity corruption) and modify training (flat neutralization) before training a Binder model. (b) LLM-based approaches use either a pure few-shot LLM or a hybrid pipeline where a fine-tuned Binder detects outer entities and an LLM identifies inner entities within each span.

Training	Prompt	Ovrl.	Inner	Outer
Flat	Keyword	76.59	3.84	83.28
Flat	Lex-all-out	76.72	3.11	83.40
Flat+Incl.	Keyword	76.70	21.36	83.05
Flat+Incl.	Lex-all-out	76.93	22.16	83.17
Full nested	Keyword	85.81	65.48	83.95
Full nested	Lex-all-out	85.55	65.33	83.59

Table 1: Baseline comparison (Micro F1, %) on NEREL test set. Flat and Flat+Inclusions represent weak supervision; Full nested is the upper bound with complete annotations. Full prompt comparison in Appendix E .

Symbol	Position	Overall	Inner	Outer
Digits	end	77.85	23.96	82.61
Letters	end	77.81	25.92	82.39
Diglets	start	78.08	22.43	82.27
Semicolon	start	74.59	19.16	77.13
Comma	start	74.23	19.46	76.38

Table 2: Best entity corruption results (Micro F1, %) for each symbol type using Strategy 1 (train on corrupted, predict on clean). Full results in Appendix F .

Baseline
Approach	Micro F1 (%)
Approach	Overall	Inner	Outer
Flat	76.59 $\pm$ 0.23	3.84 $\pm$ 0.77	83.28 $\pm$ 0.23
Inclusions	76.70 $\pm$ 0.17	21.36 $\pm$ 1.53	83.05 $\pm$ 0.26
Neutralization
Flat+Neutral.	76.82 $\pm$ 0.19	5.43 $\pm$ 0.81	83.01 $\pm$ 0.22

Table 3: Results for flat neutralization approaches on NEREL test set. Neutralization marks potential nested mentions as neutral (neither positive nor negative) during training. “Lem.Incl.” refers to lemmatized inclusions.

Traditional Approaches
Method	Overall	Inner	Outer
Flat	76.59 $\pm$ 0.23	3.84 $\pm$ 0.77	83.28 $\pm$ 0.23
Flat+Inclusions	76.70 $\pm$ 0.17	21.36 $\pm$ 1.53	83.05 $\pm$ 0.26
LLM Approaches (MFE)
DeepSeek-R1 (0-shot)	38.35	2.69	38.86
DeepSeek-R1 (1-shot)	39.15	4.57	40.14

Table 4: LLM and hybrid results on NEREL test set (Micro F1, %). Traditional approaches show mean

\pm

std across 3 runs; LLM approaches show mean across 3 seeds.

Weak Supervision (Fine-tuned)
Method	Overall	Inner	Outer
Flat (baseline)	76.59	3.84	83.28
+ Inclusions	76.70	21.36	83.05
+ Corruption (letters, end)	77.81	25.92	82.39
+ Lem.Incl.+Corr.+Neutral.	77.89	26.37	82.54
LLM-based

Table 5: Summary comparison of best methods (Micro F1, %). Our best weak supervision method (Lem.Incl.+Corr.+Neutral.) closes 40% of the inner F1 gap between flat training and full supervision.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Type	True	Incl.	Prec.	Lem.	Prec.
	inner		(%)	incl.	(%)
PERSON	433	2,845	2.81	41,347	0.07
PROFESSION	1,820	884	28.05	14,042	0.97
ORGANIZATION	2,001	662	59.82	13,456	1.43
COUNTRY	1,773	619	82.55	13,874	1.28
NUMBER	194	456	3.29	2,477	0.04

Appendix

Table 6: Inclusion statistics per entity type on training data, sorted by exact inclusion count. True inner : gold nested entities. Incl. : exact substring inclusions. Lem. incl. : lemmatized substring inclusions. Precision measures the fraction of inclusions matching gold inner entities. Entity types like COUNTRY and ORGANIZATION have high exact-match precision, while PERSON produces many inclusions with low precision due to name components matching across unrelated entities. Lemmatization vastly increases inclusion counts but reduces precision, as morphological normalization over-generalizes matching.

Flat Training (NEREL-outerflat)
Training	Prompt	Micro F1 (%)			Macro F1 (%)
Training	Prompt	Overall	Inner	Outer	Overall	Inner	Outer
	Keyword	76.59 $\pm$ 0.23	3.84 $\pm$ 0.77	83.28 $\pm$ 0.23	76.67 $\pm$ 0.17	3.30 $\pm$ 0.58	82.94 $\pm$ 0.27
	Definition	76.43 $\pm$ 0.17	4.42 $\pm$ 0.52	82.96 $\pm$ 0.31	76.57 $\pm$ 0.17	3.76 $\pm$ 0.79	82.66 $\pm$ 0.32
	Most-freq	76.60 $\pm$ 0.21	3.66 $\pm$ 0.46	83.08 $\pm$ 0.56	76.67 $\pm$ 0.24	3.23 $\pm$ 0.53	82.80 $\pm$ 0.55
	Context	75.32 $\pm$ 0.17	3.32 $\pm$ 0.72	81.38 $\pm$ 0.23	75.42 $\pm$ 0.12	2.79 $\pm$ 0.69	81.05 $\pm$ 0.30

Appendix

Table 7: Full prompt comparison on NEREL test set. Six prompt strategies tested: Keyword (entity type names), Definition (natural language definitions), Most-freq (most frequent entity example per type), Context (sentence contexts), Lex-all-out (full sentences with entity markers), Struct-nest (structural nesting information).

Digits Corruption
Corruption	Position	Micro F1 (%)			Macro F1 (%)
Corruption	Position	Overall	Inner	Outer	Overall	Inner	Outer
Early	random	77.98 $\pm$ 0.04	21.23 $\pm$ 2.16	82.74 $\pm$ 0.01	78.15 $\pm$ 0.01	17.41 $\pm$ 1.86	82.53 $\pm$ 0.05
	start	77.74 $\pm$ 0.10	23.32 $\pm$ 0.45	82.68 $\pm$ 0.25	77.76 $\pm$ 0.11	19.38 $\pm$ 0.67	82.33 $\pm$ 0.29
	end	77.85 $\pm$ 0.21	23.96 $\pm$ 0.73	82.61 $\pm$ 0.30	77.95 $\pm$ 0.24	20.35 $\pm$ 0.82	82.33 $\pm$ 0.22
	middle	78.12 $\pm$ 0.16	22.07 $\pm$ 1.77	82.45 $\pm$ 0.12	78.29 $\pm$ 0.12	18.45 $\pm$ 1.23	82.24 $\pm$ 0.07

Appendix

Table 8: Full entity corruption results on NEREL test set. “Early” (early damage) refers to training on corrupted data and predicting on clean data; “Late” (late damage) refers to training on clean data and predicting on corrupted data. Positions: random, start, end, middle, syntax (syntactic root).

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 106 papers

Assessment of Generative Named Entity Recognition in the Era of Large Language Models
Jan 25, 2026Qi Zhan, Yile Wang, Hui HuangNamed Entity Recognition LLM Information Extraction
College of Computer Science and Software Engineering, Shenzhen University
Named Entity Recognition using Sliding Window Approach
Aug 30, 2026Hariom Ingle, Ronit Ghode, Ishwari Gondkar +3Named Entity Recognition
Department of Information Technology, PICT, Pune, India · L3Cube Labs, Pune, India · Indian Institute of Technology Madras, Chennai, India
DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models
Aug 1, 2026Xuankang Zhang, Jiangming LiuNamed Entity Recognition Zero-Shot Learning
Yunnan University

Jan 25, 2026J/K move · Enter open · S save

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Qi Zhan, Yile Wang, Hui Huang

College of Computer Science and Software Engineering, Shenzhen University

Named Entity Recognition LLM Information Extraction LLM Fine-Tuning LLM Evaluation

Open paper AskPDF arXiv

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.

Abstract

Named Entity Recognition

Explore similar work

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Named Entity Recognition using Sliding Window Approach

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models