Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Qi Zhan; Yile Wang; Hui Huang

cs.CLJan 25, 2026

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Authors: Qi Zhan, Yile Wang, Hui Huang

Organizations: College of Computer Science and Software Engineering, Shenzhen University

Abstract

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.

Figures & tables

Figure 1: Comparison between traditional NER based on pre-trained models (top) and generative NER based on large language models (bottom).

Types	Examples of Labeled Output
Inline Bracketed	This is the first demonstration of a specific interaction with [PU.1 \| Protein ]
Inline Bracketed	on a [myeloid [PU.1 \| Protein ] binding site \| DNA ].
Inline XML	This is the first demonstration of a specific interaction with < Protein > PU.1 </ Protein >
Inline XML	on a < DNA > myeloid < Protein > PU.1 </ Protein > binding site </ DNA > .
Category-grouped JSON	{ “ DNA ” : [“myeloid PU.1 binding site”], “ Protein ” : [“PU.1”, “PU.1”],
Category-grouped JSON	“ Cell_line ”: [], “ Cell_type ”: [], “ RNA ”: []}

Table 1: Five types of output formats and examples for both flat and nested named entities.

Datasets	Types	Domain	# Labels	# Train	# Dev	# Test
CoNLL2003	Flat	General	4	14,041	3,250	3,453
OntoNotes5.0	Flat	General	18	59,924	8,528	8,262
ACE2005	Nested	General	7	7,299	971	1,060
GENIA	Nested	Biomedical	5	14,835	1,854	1,854

Table 2: Statistics of four NER datasets we used.

Panel of Table 3 — Table 3: Results of pre-trained models, GPT-3, and open-source LLMs on flat NER datasets. The best results are shown in bold and the second-best results are shown in underlined . Averages are highlighted in blue .

Panel of Table 4 — Table 4: Results of pre-trained models, GPT-3, and open-source LLMs on nested NER datasets.

Instruction I. Symbol-based Label with Explanations (CoNLL2003-SE)

Instruction : Your task is to do sequence labeling with labels A, B, C, and D.

A: A collective entity such as a company, institution, brand, political or governmental body…(Explanation of ORG )

B: A named individual, including humans, animals, fictional characters, and their aliases. (Explanation of PER )

C: A geographical or spatial entity, including natural features, built structures, regions… (Explanation of LOC )

D: Named entities that are not persons, organizations, or locations, including derived… (Explanation of MISC )

Instruction II. Symbol-based Label Only (CoNLL2003-SO)

Table 5: Examples of two variants of CoNLL2003 with symbol-based labels. The first contains explanations for each symbolic label (CoNLL2003-SE), while the second does not offer explanations in each sample (CoNLL2003-SO).

Panel of Figure 2 — Figure 2: Investigation of entity-label memorization. (a) and (b) compare few-shot prompting with LoRA SFT on CoNLL2003 (left) and CleanCoNLL (right). (c) and (d) report F1 changes under two variants (SE: with explanations; SO: symbol only) relative to the original dataset.

	Error Types	Examples of Incorrectly Labeled Output
Type	OOD Types	[Inter] ${}_{\textsc{ORG}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ defender [Joceyln Angloma] ${}_{\textsc{PLAYER}}$
Type	Wrong Types	[Inter] ${}_{\textsc{LOC}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ defender [Joceyln Angloma] ${}_{\textsc{PER}}$
Boundary	Contain Gold	[Inter] ${}_{\textsc{ORG}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ [defender Joceyln Angloma] ${}_{\textsc{PER}}$
	Contained by Gold	[Inter] ${}_{\textsc{ORG}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ defender Joceyln [Angloma] ${}_{\textsc{PER}}$
	Overlap with Gold	[Inter] ${}_{\textsc{ORG}}$ will be without suspended French [defender Joceyln] ${}_{\textsc{PER}}$ Angloma
Others	Completely-O	[Inter] ${}_{\textsc{ORG}}$ will be without [suspended] ${}_{\textsc{MISC}}$ [French] ${}_{\textsc{MISC}}$ defender [Joceyln Angloma] ${}_{\textsc{PER}}$

Table 6: Error types based on the sentence: “[Inter]

{}_{\textsc{ORG}}

will be without suspended [French]

{}_{\textsc{MISC}}

defender [Joceyln Angloma]

{}_{\textsc{PER}}

.” Underlined entities denote correct predictions, while blue bold spans highlight specific errors.

Panel of Figure 3 — Figure 3: Distribution of error types for BERT-MRC and LLaMA3.1-8B on CoNLL2003.

Error Types	Input Sentence	Ground Truth	BERT-MRC	LLaMA3.1-8B
Omitted Mentions	Italy recalled Marcello Cuttitta .	[ M. Cuttitta ] ${}_{\textsc{PER}}$	O (Missed) (45.5%)	[ M. Cuttitta ] ${}_{\textsc{PER}}$ (13.7%)
Wrong Types	In congruence with futures, Permian cash prices for the weekend fell more…	[ Permian ] ${}_{\textsc{MISC}}$	[ Permian ] ${}_{\textsc{MISC}}$ (29.0%)	[ Permian ] ${}_{\textsc{LOC}}$ (38.2%)
Completely-O	I’ve also got a contract to play for New South Wales in the Super 12 next year.	O	O (11.1%)	[ Super 12 ] ${}_{\textsc{MISC}}$ (25.2%)

Table 7: Case study of errors from BERT-MRC and LLaMA3.1-8B. Text in blue indicates incorrect predictions. The number in parentheses indicates the share of each error type among all errors.

Models	TruthfulQA	MMLU	GSM8K	HellaSwag	DROP	Avg.
Models	MC1	Acc	EM	Acc_norm	F1	Avg.
LLaMA3.1-8B	40.15	68.76	84.69	72.53	33.20	59.87
w/ NER LoRA Fine-tuning (XML)	36.96 ( $\downarrow$ 3.19)	67.08 ( $\downarrow$ 1.68)	80.36 ( $\downarrow$ 4.33)	70.66 ( $\downarrow$ 1.87)	58.70 ( $\uparrow$ 25.50)	62.75 ( $\uparrow$ 2.88)
Qwen2.5-7B	47.49	73.42	76.57	65.43	15.74	55.73
w/ NER LoRA Fine-tuning (XML)	45.04 ( $\downarrow$ 2.45)	72.75 ( $\downarrow$ 0.67)	79.61 ( $\uparrow$ 3.04)	68.47 ( $\uparrow$ 3.04)	61.06 ( $\uparrow$ 45.32)	65.39 ( $\uparrow$ 9.66)

Table 8: Performance comparison of LLMs on benchmarks before and after generative NER fine-tuning.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 9 — Table 9: Checkpoints of open-source models used in our experiments.

Panel of Figure 4 — Figure 4: The template and example of constructing instructions for generative NER.

Output Format	Task Description
Inline Bracketed	Your task is to identify all named entities in the input sentence and rewrite the sentence by enclosing each entity using the format [Entity Text \| LABEL]. Use only the label tags defined in the Label Set below.
Inline XML	Your task is to identify all named entities in the input sentence and rewrite the sentence by enclosing each entity using the format <LABEL>Entity Text</LABEL>. Use only the label tags defined in the Label Set below.
Category-grouped JSON	Your task is to identify all named entities in the input sentence and extract them into a JSON object grouped by label. The keys of the JSON object should be the label tags defined in the Label Set below, and the values should be lists of extracted entities. If a label has no corresponding entities, use an empty list.
Occurrence-based JSON	Your task is to identify all named entities in the input sentence and extract them into a JSON list. Output a JSON list where each object contains: 1. “text”: the entity string. 2. “label”: the label tags defined in the Label Set below. 3. “occurrence_index”: a 1-based integer indicating the order of appearance of this text in the sentence (e.g., 1 for the first appearance, 2 for the second). If no entities are found, return an empty list.
Offset-based JSON	Your task is to identify all named entities in the input sentence and extract them with precise character offsets. Output a JSON list where each object contains: 1. “text”: the entity string. 2. “label”: the label tags defined in the Label Set below. 3. “start”: the 0-based index of the entity’s first character. 4. “end”: the 0-based index of the first character after the entity (exclusive).

Appendix

Table 10: Task descriptions for different output formats.

Model	Fine-tuning	CoNLL2003	GENIA
LLaMA3.1-8B	LoRA	93.85	79.18
LLaMA3.1-8B	Full	93.61 ( $\downarrow$ 0.24)	78.70 ( $\downarrow$ 0.48)
Qwen2.5-7B	LoRA	93.73	78.67
Qwen2.5-7B	Full	93.58 ( $\downarrow$ 0.15)	78.53 ( $\downarrow$ 0.14)

Appendix

Table 11: Comparison between LoRA fine-tuning and full fine-tuning on CoNLL2003 and GENIA. Numbers in parentheses denote the difference compared with LoRA fine-tuning.

Figure 5: Performance on flat and nested entity mentions in the two nested NER datasets, averaged across eight LLMs for each output format. Downward arrows report the F1 decrease from flat to nested entities.

Model	CoNLL2003		OntoNotes5.0		ACE2005		GENIA
Model	F1	$F_{1}^{\text{w/o offset}}$	F1	$F_{1}^{\text{w/o offset}}$	F1	$F_{1}^{\text{w/o offset}}$	F1	$F_{1}^{\text{w/o offset}}$
LLaMA3.1-8B	63.22	92.64 ( $\uparrow$ 29.42)	53.31	87.60 ( $\uparrow$ 34.29)	35.85	81.60 ( $\uparrow$ 45.75)	34.62	77.96 ( $\uparrow$ 43.34)
LLaMA3.2-3B	56.13	91.91 ( $\uparrow$ 35.78)	31.20	84.71 ( $\uparrow$ 53.51)	27.31	79.54 ( $\uparrow$ 52.23)	25.79	76.63 ( $\uparrow$ 50.84)
Qwen2.5-7B	60.84	92.74 ( $\uparrow$ 31.90)	50.92	87.59 ( $\uparrow$ 36.67)	32.59	80.61 ( $\uparrow$ 48.02)	32.02	76.45 ( $\uparrow$ 44.43)
Qwen3-4B	61.18	91.97 ( $\uparrow$ 30.79)	38.34	84.76 ( $\uparrow$ 46.42)	35.41	80.07 ( $\uparrow$ 44.66)	36.81	77.33 ( $\uparrow$ 40.52)
Granite3.3-8B	64.79	92.90 ( $\uparrow$ 28.11)	44.08	87.05 ( $\uparrow$ 42.97)	36.90	81.86 ( $\uparrow$ 44.96)	39.54	77.50 ( $\uparrow$ 37.96)

Appendix

Table 12: Impact of character-offset errors on Offset-based JSON performance.

F_{1}^{\text{w/o offset}}

removes the offset-matching requirement, revealing that models identify entities accurately but struggle with precise positional alignment. Green values indicate the absolute F1 improvement from ignoring offsets.

Figure 6: Distribution of character-offset errors by boundary displacement

d

for two representative models across four datasets.

Metric	LLaMA3.1-8B	BERT-MRC
Training Time (h)	2.40	2.31
Trainable Parameters (M)	671	335
Peak GPU Memory (GiB)	27.89	21.62
F1	93.85	93.04

Appendix

Table 13: Cost-benefit comparison between LLaMA3.1-8B and BERT-MRC on CoNLL2003.

Labels	Definition
ORG	A collective entity such as a company, institution, brand, political or governmental body, publication, or any organized group of people acting as a unit.
PER	A named individual, including humans, animals, fictional characters, and their aliases.
LOC	A geographical or spatial entity, including natural features, built structures, regions, public or commercial places, assorted buildings, and abstract or metaphorical places.
MISC	Named entities that are not persons, organizations, or locations, including derived adjectives, religions, ideologies, nationalities, languages, events, programs, wars, titles of works, slogans, eras, and types of objects.

Appendix

Table 14: Label definitions for CoNLL2003 dataset.

Labels	Definition
PERSON	People, including fictional.
NORP	Nationalities or religious or political groups.
FAC	Buildings, airports, highways, bridges, etc..
ORG	Companies, agencies, institutions, etc..
GPE	Countries, cities, states.
LOC	Non-GPE locations, mountain ranges, bodies of water.

Appendix

Table 15: Label definitions for OntoNotes5.0 dataset.

Labels	Definition
DNA	Deoxyribonucleic acid sequences or molecules, including genes, gene fragments, promoters, enhancers, and other genomic regions that encode or regulate biological information.
RNA	Ribonucleic acid molecules transcribed from DNA, encompassing messenger RNA, non-coding RNAs, and RNA transcripts involved in gene expression and regulation.
Protein	Functional biological macromolecules composed of amino acids, including enzymes, receptors, transcription factors, and protein complexes produced by translation of RNA.
Cell_line	A population of cells derived from a single source and maintained in vitro through continuous culture, often genetically stable and used for experimental research.
Cell_type	A class of cells defined by shared morphological, functional, and molecular characteristics within an organism, such as immune cells, epithelial cells, or neurons.

Appendix

Table 16: Label definitions for GENIA dataset.

Labels	Definition
PER	Person entities are limited to humans. A person may be a single individual or a group.
ORG	Organization entities are limited to corporations, agencies, and other groups of people defined by an established organizational structure.
GPE	GPE entities are geographical regions defined by political and/or social groups. A GPE entity subsumes and does not distinguish between a nation, its region, its government, or its people.
LOC	Location entities are limited to geographical entities such as geographical areas and landmasses, bodies of water, and geological formations.
FAC	Facility entities are limited to buildings and other permanent man-made structures and real estate improvements.
VEH	A vehicle entity is a physical device primarily designed to move an object from one location to another, by (for example) carrying, pulling, or pushing the transported object. Vehicle entities may or may not have their own power source.

Appendix

Table 17: Label definitions for ACE2005 dataset.

Models	CoNLL2003			OntoNotes5.0			Avg ${}^{\text{Flat}}_{\text{F1}}$	ACE2005			GENIA			Avg ${}^{\text{Nested}}_{\text{F1}}$
Models	P	R	F1	P	R	F1	Avg ${}^{\text{Flat}}_{\text{F1}}$	P	R	F1	P	R	F1	Avg ${}^{\text{Nested}}_{\text{F1}}$
Pre-Trained NER Models
BERT-Tagger	91.20	92.30	91.70	90.01	88.35	89.16	90.43	-	-	-	-	-	-	-
GNN-SL	93.02	93.40	93.20	91.48	91.29	91.39	92.30	-	-	-	-	-	-	-
ACE + Fine-tune	93.07	94.21	93.64	91.48	91.29	91.39	92.52	-	-	-	-	-	-	-
BERT-MRC+DSC	93.41	93.25	93.33	91.59	92.56	92.07	92.70	-	-	-	-	-	-	-

Appendix

Table 18: Complete Precision, Recall, and F1 results of pre-trained models, GPT-3, and open-source LLMs on flat and nested NER datasets.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 106 papers

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models
Aug 1, 2026Xuankang Zhang, Jiangming LiuNamed Entity Recognition Zero-Shot Learning
Yunnan University
Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam
Aug 30, 2026Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani +1Named Entity Recognition Multilingual Language Model Evaluation
Rajiv Gandhi University of Knowledge Technologies, Basar, Telangana, India
Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels
Aug 31, 2026Elena Merdjanovska, Jonas Golde, Alan AkbikNamed Entity Recognition LLM-Assisted Annotation
Humboldt-Universität zu Berlin · Science of Intelligence

Aug 1, 2026J/K move · Enter open · S save

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models

Xuankang Zhang, Jiangming Liu

Yunnan University

Named Entity Recognition Zero-Shot Learning LLM Prompting LLM Information Extraction

Open paper AskPDF arXiv

Recent advancements of zero-shot Named Entity Recognition (NER) establish strong baselines by formulating sequence labeling into question answering where Large Language Models (LLMs) can be naturally adopted. However, existing LLM-based zero-shot NER methods suffer from the limitations of prompt and demonstration engineering. To address these issues with minimal human interventions, we introduce DE-NER, a dialogue elicitation framework which elicits the chatting ability of LLMs to fully extract the knowledge encoded in LLMs. Our experiments demonstrate that the proposed method outperform the competitive baselines in zero-shot settings across multiple benchmarks, with an average improvement of 3.75% F1 points. Codes are released in https://github.com/kkkenshi/DE-NER.

Abstract

Named Entity Recognition

Explore similar work

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels