Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Qi Zhan; Yile Wang; Hui Huang

cs.CLJan 25, 2026

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Authors: Qi Zhan, Yile Wang, Hui Huang

Organizations: College of Computer Science and Software Engineering, Shenzhen University

Abstract

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.

Figures & tables

Figure 1: Comparison between traditional NER based on pre-trained models (top) and generative NER based on large language models (bottom).

Types	Examples of Labeled Output
Inline Bracketed	This is the first demonstration of a specific interaction with [PU.1 \| Protein ]
Inline Bracketed	on a [myeloid [PU.1 \| Protein ] binding site \| DNA ].
Inline XML	This is the first demonstration of a specific interaction with < Protein > PU.1 </ Protein >
Inline XML	on a < DNA > myeloid < Protein > PU.1 </ Protein > binding site </ DNA > .
Category-grouped JSON	{ “ DNA ” : [“myeloid PU.1 binding site”], “ Protein ” : [“PU.1”, “PU.1”],
Category-grouped JSON	“ Cell_line ”: [], “ Cell_type ”: [], “ RNA ”: []}

Table 1: Five types of output formats and examples for both flat and nested named entities.

Datasets	Types	Domain	# Labels	# Train	# Dev	# Test
CoNLL2003	Flat	General	4	14,041	3,250	3,453
OntoNotes5.0	Flat	General	18	59,924	8,528	8,262
ACE2005	Nested	General	7	7,299	971	1,060
GENIA	Nested	Biomedical	5	14,835	1,854	1,854

Table 2: Statistics of four NER datasets we used.

Panel of Table 3 — Table 3: Results of pre-trained models, GPT-3, and open-source LLMs on flat NER datasets. The best results are shown in bold and the second-best results are shown in underlined . Averages are highlighted in blue .

Panel of Table 4 — Table 4: Results of pre-trained models, GPT-3, and open-source LLMs on nested NER datasets.

Instruction I. Symbol-based Label with Explanations (CoNLL2003-SE)

Instruction : Your task is to do sequence labeling with labels A, B, C, and D.

A: A collective entity such as a company, institution, brand, political or governmental body…(Explanation of ORG )

B: A named individual, including humans, animals, fictional characters, and their aliases. (Explanation of PER )

C: A geographical or spatial entity, including natural features, built structures, regions… (Explanation of LOC )

D: Named entities that are not persons, organizations, or locations, including derived… (Explanation of MISC )

Instruction II. Symbol-based Label Only (CoNLL2003-SO)

Table 5: Examples of two variants of CoNLL2003 with symbol-based labels. The first contains explanations for each symbolic label (CoNLL2003-SE), while the second does not offer explanations in each sample (CoNLL2003-SO).

Panel of Figure 2 — Figure 2: Investigation of entity-label memorization. (a) and (b) compare few-shot prompting with LoRA SFT on CoNLL2003 (left) and CleanCoNLL (right). (c) and (d) report F1 changes under two variants (SE: with explanations; SO: symbol only) relative to the original dataset.

	Error Types	Examples of Incorrectly Labeled Output
Type	OOD Types	[Inter] ${}_{\textsc{ORG}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ defender [Joceyln Angloma] ${}_{\textsc{PLAYER}}$
Type	Wrong Types	[Inter] ${}_{\textsc{LOC}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ defender [Joceyln Angloma] ${}_{\textsc{PER}}$
Boundary	Contain Gold	[Inter] ${}_{\textsc{ORG}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ [defender Joceyln Angloma] ${}_{\textsc{PER}}$
	Contained by Gold	[Inter] ${}_{\textsc{ORG}}$ will be without suspended [French] ${}_{\textsc{MISC}}$ defender Joceyln [Angloma] ${}_{\textsc{PER}}$
	Overlap with Gold	[Inter] ${}_{\textsc{ORG}}$ will be without suspended French [defender Joceyln] ${}_{\textsc{PER}}$ Angloma
Others	Completely-O	[Inter] ${}_{\textsc{ORG}}$ will be without [suspended] ${}_{\textsc{MISC}}$ [French] ${}_{\textsc{MISC}}$ defender [Joceyln Angloma] ${}_{\textsc{PER}}$

Table 6: Error types based on the sentence: “[Inter]

{}_{\textsc{ORG}}

will be without suspended [French]

{}_{\textsc{MISC}}

defender [Joceyln Angloma]

{}_{\textsc{PER}}

.” Underlined entities denote correct predictions, while blue bold spans highlight specific errors.

Panel of Figure 3 — Figure 3: Distribution of error types for BERT-MRC and LLaMA3.1-8B on CoNLL2003.

Error Types	Input Sentence	Ground Truth	BERT-MRC	LLaMA3.1-8B
Omitted Mentions	Italy recalled Marcello Cuttitta .	[ M. Cuttitta ] ${}_{\textsc{PER}}$	O (Missed) (45.5%)	[ M. Cuttitta ] ${}_{\textsc{PER}}$ (13.7%)
Wrong Types	In congruence with futures, Permian cash prices for the weekend fell more…	[ Permian ] ${}_{\textsc{MISC}}$	[ Permian ] ${}_{\textsc{MISC}}$ (29.0%)	[ Permian ] ${}_{\textsc{LOC}}$ (38.2%)
Completely-O	I’ve also got a contract to play for New South Wales in the Super 12 next year.	O	O (11.1%)	[ Super 12 ] ${}_{\textsc{MISC}}$ (25.2%)

Table 7: Case study of errors from BERT-MRC and LLaMA3.1-8B. Text in blue indicates incorrect predictions. The number in parentheses indicates the share of each error type among all errors.

Models	TruthfulQA	MMLU	GSM8K	HellaSwag	DROP	Avg.
Models	MC1	Acc	EM	Acc_norm	F1	Avg.
LLaMA3.1-8B	40.15	68.76	84.69	72.53	33.20	59.87
w/ NER LoRA Fine-tuning (XML)	36.96 ( $\downarrow$ 3.19)	67.08 ( $\downarrow$ 1.68)	80.36 ( $\downarrow$ 4.33)	70.66 ( $\downarrow$ 1.87)	58.70 ( $\uparrow$ 25.50)	62.75 ( $\uparrow$ 2.88)
Qwen2.5-7B	47.49	73.42	76.57	65.43	15.74	55.73
w/ NER LoRA Fine-tuning (XML)	45.04 ( $\downarrow$ 2.45)	72.75 ( $\downarrow$ 0.67)	79.61 ( $\uparrow$ 3.04)	68.47 ( $\uparrow$ 3.04)	61.06 ( $\uparrow$ 45.32)	65.39 ( $\uparrow$ 9.66)

Table 8: Performance comparison of LLMs on benchmarks before and after generative NER fine-tuning.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 9 — Table 9: Checkpoints of open-source models used in our experiments.

Panel of Figure 4 — Figure 4: The template and example of constructing instructions for generative NER.

Output Format	Task Description
Inline Bracketed	Your task is to identify all named entities in the input sentence and rewrite the sentence by enclosing each entity using the format [Entity Text \| LABEL]. Use only the label tags defined in the Label Set below.
Inline XML	Your task is to identify all named entities in the input sentence and rewrite the sentence by enclosing each entity using the format <LABEL>Entity Text</LABEL>. Use only the label tags defined in the Label Set below.
Category-grouped JSON	Your task is to identify all named entities in the input sentence and extract them into a JSON object grouped by label. The keys of the JSON object should be the label tags defined in the Label Set below, and the values should be lists of extracted entities. If a label has no corresponding entities, use an empty list.
Occurrence-based JSON	Your task is to identify all named entities in the input sentence and extract them into a JSON list. Output a JSON list where each object contains: 1. “text”: the entity string. 2. “label”: the label tags defined in the Label Set below. 3. “occurrence_index”: a 1-based integer indicating the order of appearance of this text in the sentence (e.g., 1 for the first appearance, 2 for the second). If no entities are found, return an empty list.
Offset-based JSON	Your task is to identify all named entities in the input sentence and extract them with precise character offsets. Output a JSON list where each object contains: 1. “text”: the entity string. 2. “label”: the label tags defined in the Label Set below. 3. “start”: the 0-based index of the entity’s first character. 4. “end”: the 0-based index of the first character after the entity (exclusive).

Appendix

Table 10: Task descriptions for different output formats.

Model	Fine-tuning	CoNLL2003	GENIA
LLaMA3.1-8B	LoRA	93.85	79.18
LLaMA3.1-8B	Full	93.61 ( $\downarrow$ 0.24)	78.70 ( $\downarrow$ 0.48)
Qwen2.5-7B	LoRA	93.73	78.67
Qwen2.5-7B	Full	93.58 ( $\downarrow$ 0.15)	78.53 ( $\downarrow$ 0.14)

Appendix

Table 11: Comparison between LoRA fine-tuning and full fine-tuning on CoNLL2003 and GENIA. Numbers in parentheses denote the difference compared with LoRA fine-tuning.

Figure 5: Performance on flat and nested entity mentions in the two nested NER datasets, averaged across eight LLMs for each output format. Downward arrows report the F1 decrease from flat to nested entities.

Model	CoNLL2003		OntoNotes5.0		ACE2005		GENIA
Model	F1	$F_{1}^{\text{w/o offset}}$	F1	$F_{1}^{\text{w/o offset}}$	F1	$F_{1}^{\text{w/o offset}}$	F1	$F_{1}^{\text{w/o offset}}$
LLaMA3.1-8B	63.22	92.64 ( $\uparrow$ 29.42)	53.31	87.60 ( $\uparrow$ 34.29)	35.85	81.60 ( $\uparrow$ 45.75)	34.62	77.96 ( $\uparrow$ 43.34)
LLaMA3.2-3B	56.13	91.91 ( $\uparrow$ 35.78)	31.20	84.71 ( $\uparrow$ 53.51)	27.31	79.54 ( $\uparrow$ 52.23)	25.79	76.63 ( $\uparrow$ 50.84)
Qwen2.5-7B	60.84	92.74 ( $\uparrow$ 31.90)	50.92	87.59 ( $\uparrow$ 36.67)	32.59	80.61 ( $\uparrow$ 48.02)	32.02	76.45 ( $\uparrow$ 44.43)
Qwen3-4B	61.18	91.97 ( $\uparrow$ 30.79)	38.34	84.76 ( $\uparrow$ 46.42)	35.41	80.07 ( $\uparrow$ 44.66)	36.81	77.33 ( $\uparrow$ 40.52)
Granite3.3-8B	64.79	92.90 ( $\uparrow$ 28.11)	44.08	87.05 ( $\uparrow$ 42.97)	36.90	81.86 ( $\uparrow$ 44.96)	39.54	77.50 ( $\uparrow$ 37.96)

Appendix

Table 12: Impact of character-offset errors on Offset-based JSON performance.

F_{1}^{\text{w/o offset}}

removes the offset-matching requirement, revealing that models identify entities accurately but struggle with precise positional alignment. Green values indicate the absolute F1 improvement from ignoring offsets.

Figure 6: Distribution of character-offset errors by boundary displacement

d

for two representative models across four datasets.

Metric	LLaMA3.1-8B	BERT-MRC
Training Time (h)	2.40	2.31
Trainable Parameters (M)	671	335
Peak GPU Memory (GiB)	27.89	21.62
F1	93.85	93.04

Appendix

Table 13: Cost-benefit comparison between LLaMA3.1-8B and BERT-MRC on CoNLL2003.

Labels	Definition
ORG	A collective entity such as a company, institution, brand, political or governmental body, publication, or any organized group of people acting as a unit.
PER	A named individual, including humans, animals, fictional characters, and their aliases.
LOC	A geographical or spatial entity, including natural features, built structures, regions, public or commercial places, assorted buildings, and abstract or metaphorical places.
MISC	Named entities that are not persons, organizations, or locations, including derived adjectives, religions, ideologies, nationalities, languages, events, programs, wars, titles of works, slogans, eras, and types of objects.

Appendix

Table 14: Label definitions for CoNLL2003 dataset.

Labels	Definition
PERSON	People, including fictional.
NORP	Nationalities or religious or political groups.
FAC	Buildings, airports, highways, bridges, etc..
ORG	Companies, agencies, institutions, etc..
GPE	Countries, cities, states.
LOC	Non-GPE locations, mountain ranges, bodies of water.

Appendix

Table 15: Label definitions for OntoNotes5.0 dataset.

Labels	Definition
DNA	Deoxyribonucleic acid sequences or molecules, including genes, gene fragments, promoters, enhancers, and other genomic regions that encode or regulate biological information.
RNA	Ribonucleic acid molecules transcribed from DNA, encompassing messenger RNA, non-coding RNAs, and RNA transcripts involved in gene expression and regulation.
Protein	Functional biological macromolecules composed of amino acids, including enzymes, receptors, transcription factors, and protein complexes produced by translation of RNA.
Cell_line	A population of cells derived from a single source and maintained in vitro through continuous culture, often genetically stable and used for experimental research.
Cell_type	A class of cells defined by shared morphological, functional, and molecular characteristics within an organism, such as immune cells, epithelial cells, or neurons.

Appendix

Table 16: Label definitions for GENIA dataset.

Labels	Definition
PER	Person entities are limited to humans. A person may be a single individual or a group.
ORG	Organization entities are limited to corporations, agencies, and other groups of people defined by an established organizational structure.
GPE	GPE entities are geographical regions defined by political and/or social groups. A GPE entity subsumes and does not distinguish between a nation, its region, its government, or its people.
LOC	Location entities are limited to geographical entities such as geographical areas and landmasses, bodies of water, and geological formations.
FAC	Facility entities are limited to buildings and other permanent man-made structures and real estate improvements.
VEH	A vehicle entity is a physical device primarily designed to move an object from one location to another, by (for example) carrying, pulling, or pushing the transported object. Vehicle entities may or may not have their own power source.

Appendix

Table 17: Label definitions for ACE2005 dataset.

Models	CoNLL2003			OntoNotes5.0			Avg ${}^{\text{Flat}}_{\text{F1}}$	ACE2005			GENIA			Avg ${}^{\text{Nested}}_{\text{F1}}$
Models	P	R	F1	P	R	F1	Avg ${}^{\text{Flat}}_{\text{F1}}$	P	R	F1	P	R	F1	Avg ${}^{\text{Nested}}_{\text{F1}}$
Pre-Trained NER Models
BERT-Tagger	91.20	92.30	91.70	90.01	88.35	89.16	90.43	-	-	-	-	-	-	-
GNN-SL	93.02	93.40	93.20	91.48	91.29	91.39	92.30	-	-	-	-	-	-	-
ACE + Fine-tune	93.07	94.21	93.64	91.48	91.29	91.39	92.52	-	-	-	-	-	-	-
BERT-MRC+DSC	93.41	93.25	93.33	91.59	92.56	92.07	92.70	-	-	-	-	-	-	-

Appendix

Table 18: Complete Precision, Recall, and F1 results of pre-trained models, GPT-3, and open-source LLMs on flat and nested NER datasets.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 106 papers

Aug 1, 2026

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models

Recent advancements of zero-shot Named Entity Recognition (NER) establish strong baselines by formulating sequence labeling into question answering where Large Language Models (LLMs) can be naturally adopted. However, existing LLM-based zero-shot NER methods suffer from the limitations of prompt and demonstration engineering. To address these issues with minimal human interventions, we introduce DE-NER, a dialogue elicitation framework which elicits the chatting ability of LLMs to fully extract the knowledge encoded in LLMs. Our experiments demonstrate that the proposed method outperform the competitive baselines in zero-shot settings across multiple benchmarks, with an average improvement of 3.75% F1 points. Codes are released in https://github.com/kkkenshi/DE-NER.

Xuankang Zhang, Jiangming Liu

Yunnan University

Named Entity Recognition Zero-Shot Learning LLM Prompting LLM Information Extraction

Ask PDF

Aug 30, 2026

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

Language is humanity's most consequential technology, yet for over a billion speakers across India's twenty-two constitutionally recognised languages, its digital layer remains structurally incomplete. Named Entity Recognition (NER), the foundational step in transforming raw text into machine-interpretable knowledge, has been studied exhaustively for English but remains largely unsolved across most Indic languages. This paper presents a rigorous comparative study of generative and encoder-based neural architectures for NER on all eleven languages of the Naamapadam benchmark. We evaluate five classic model families spanning sequence-to-sequence transformers and multilingual encoders; four decoder-only large language models (LLMs) fine-tuned with LoRA and 4-bit NF4 quantisation; and nine generative models in zero-to-5-shot inference. Under strict CoNLL span-level evaluation, encoder-based models (mBERT and XLM-R, both F1=0.675 on Hindi) substantially outperform every generative architecture in ten of eleven languages, with gaps of 7.5-40 percentage points against the strongest competitor (Gemma-2-2B: avg F1=0.427). The best few-shot result reaches only 28% of the encoder baseline. We identify three language clusters--encoder-dominant, partial-coverage, and failure-zone; and provide actionable deployment guidelines grounded in transfer learning and low-resource NLP principles.

Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani +1

Rajiv Gandhi University of Knowledge Technologies, Basar, Telangana, India

Named Entity Recognition Multilingual Language Model Evaluation Decoder-Only Language Models Low-Resource Language Processing

Ask PDF

Aug 31, 2026

Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels

Large language models are increasingly used to annotate datasets for training smaller, task-specialized models such as named entity recognition. While this method yields effective models, it assumes that the synthetic dataset is correctly annotated. In this work, we find that (i) current fine-tuning processes simply ignore LLM-introduced annotation noise, resulting in degraded performance and (ii) existing noise-robust losses are not transferable to sequence labeling because annotation noise in named entity recognition is heterogeneous: for example, missing mentions and type errors affect the training signal in different ways. Treating all noisy tokens equally in noise-robust losses and applying a single reweighing criterion for all may therefore remove useful supervision or reinforce incorrect labels. To address this limitation, we propose error-type-aware loss reweighting for NER, which introduces separate reweighing rules for different types of potentially erroneous tokens. Our approach is simple and efficient, does not require additional training resources, and improves F1 by 0.8 - 2.0 percentage points on dataset-level average for noise levels between 15% and 40%, with a maximum improvement of 4.6 percentage points with 24.1% noise on Wikigold.

Elena Merdjanovska, Jonas Golde, Alan Akbik

Humboldt-Universität zu Berlin · Science of Intelligence

Named Entity Recognition LLM-Assisted Annotation Noisy-Label Learning Robust Loss Functions Adaptive Loss Weighting

Ask PDF

Abstract

Named Entity Recognition

Explore similar work

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels