ENEIDE: A High Quality Silver Standard Dataset for Named Entity Recognition and Linking in Historical Italian
Authors: Cristian Santini, Sebastian Barzaghi, Paolo Sernani, Emanuele Frontoni, Laura Melosi, Mehwish Alam
Organizations: University of Macerata, Macerata, Italy · Telecom Paris, Polytechnic Institute of Paris, Palaiseau, France · University of Bologna, Bologna, Italy
This paper introduces ENEIDE (Extracting Named Entities from Italian Digital Editions), a silver standard dataset for Named Entity Recognition and Linking (NERL) in historical Italian texts. The corpus comprises 2,111 documents with over 8,000 entity annotations semi-automatically extracted from two scholarly digital editions: Digital Zibaldone, the philosophical diary of the Italian poet Giacomo Leopardi (1798--1837), and Aldo Moro Digitale, the complete works of the Italian politician Aldo Moro (1916--1978). Annotations cover multiple entity types (person, location, organization, literary work) linked to Wikidata identifiers, including NIL entities that cannot be mapped to the knowledge graph. To the best of our knowledge, ENEIDE represents the first multi-domain, publicly available NERL dataset for historical Italian with training, development, and test splits. We present a methodology for semi-automatic annotations extraction from manually curated scholarly digital editions, including quality control and annotation enhancement procedures. Baseline experiments using state-of-the-art models demonstrate the dataset's challenge for NERL and the gap between zero-shot approaches and fine-tuned models. The dataset's diachronic coverage spanning two centuries makes it particularly suitable for temporal entity disambiguation and cross-domain evaluation. ENEIDE is released under a CC BY-NC-SA 4.0 license.
Figures & tables
Figure 1: Example of markup with entities annotated for a note in Digital Zibaldone.
Figure 2: Example of entities annotated with HTML and RDFa in a document of the Edition of Aldo Moro’s works.
Statistic
DZ
AMD
Annotations in samples
403
215
Wrong annotations
13
18
Missing annotations
10
64
Precision
96.8
91.6
Recall
94.4
70.6
F1
95.6
79.8
Table 1: Quality assessment statistics for DZ and AMD samples.
DZ
AMD
Documents (train)
735
743
Documents (dev)
157
159
Documents (test)
158
160
Tokens per Doc (train)
246.6
125.9
Tokens per Doc (dev)
253.2
119.3
Tokens per Doc (test)
248.7
119
Table 2: Number of documents, average tokens, total annotations, and Wikidata entity overlap percentages in ENEIDE.
DZ
AMD
Split
PER
LOC
WORK
NIL
IDs
PER
LOC
ORG
NIL
IDs
train
1,661
488
786
182
623
759
940
1,067
64
583
dev
375
149
203
74
276
158
226
206
13
203
test
318
130
169
42
241
194
190
205
9
238
Table 3: Annotation counts by entity type (PER=person, LOC=location, WORK=literary work, ORG=organization), NIL entities, and unique Wikidata identifiers (IDs) in each split.
Dataset
Model
Precision
Recall
F1
DZ
LLaMA 3.1-8B
0.438
0.376
0.405
Minerva-7B
0.531
0.028
0.052
Ministral-8B
0.351
0.329
0.340
GLiNER (fine-tuned)
0.851
0.723
0.782
AMD
LLaMA 3.1-8B
0.719
0.603
0.656
Minerva-7B
0.659
0.138
0.228
Table 4: NER results on DZ and AMD test sets using zero-shot instruction-tuned LLMs.
Dataset
Model
Accuracy
DZ
mGENRE
0.579
BLINK-ita
0.502
BELA
0.598
AMD
mGENRE
0.689
BLINK-ita
0.643
BELA
0.676
Table 5: Entity disambiguation results on DZ and AMD test sets using EL systems trained on Wikipedia.
Figure 3: Temporal distribution of named entities in the DZ corpus, showing references spanning from classical antiquity to the early modern period.
Figure 4: Temporal distribution of named entities in the AMD corpus, showing references concentrated in the 20th-century Italian history.
International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token-level named entity recognition (NER) resources. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold-annotated sentences and 8,094 entity spans from International Court of Justice (ICJ) decisions, UN Security Council resolutions, and European Court of Human Rights (ECtHR) judgments, annotated with seven institution-specific entity types. We construct IntLawNER with a cost-effective hybrid algorithmic-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM-based vetting, and human review, with 89.6% of gold spans accepted unchanged from the silver layer. However, the silver-to-gold analysis reveals that human-machine aggregate agreement metrics can be misleading in domain-specific NER: Cohen's kappa=0.964 on boundary-matched spans masks a macro-F1 of 0.753 when missing entities, boundary errors, and label corrections are included. The benchmark shows that zero-shot span-based GLiNER collapses on entity types dependent on institutional function rather than surface form (0.243 micro-F1), while fine-tuned transformers struggle on rare labels. Carefully selected few-shot examples that demonstrate label contrasts improve every LLM over zero-shot prompting, with Claude Opus 4.6 reaching the best score of 0.873 micro-F1. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts.
Genis Skura, Roland Bouffanais, Didier Wernli
Department of Computer Science, Faculty of Science Global Studies Institute University of Geneva, Switzerland
Existing scholarly information extraction (SIE) datasets focus on scientific papers and overlook implementation-level details in code repositories. README files describe datasets, source code, and other implementation-level artifacts, however, their free-form Markdown offers little semantic structure, making automatic information extraction difficult. To address this gap, NERdME is introduced: 200 manually annotated README files with over 10,000 labeled spans and 10 entity types. Baseline results using large language models and fine-tuned transformers show clear differences between paperlevel and implementation-level entities, indicating the value of extending SIE benchmarks with entity types available in README files. A downstream entity-linking experiment was conducted to demonstrate that entities derived from READMEs can support artifact discovery and metadata integration.
We present BullingerDB, a large-scale benchmark dataset for historical document analysis based on the correspondence of Heinrich Bullinger (1504-1575). The corpus comprises 20,898 pages and 499,222 text lines written by 796 writers over six decades, featuring stylistic variation, multilingual content (mostly Latin and Early New High German) as well as meta-information such as writer identity and time. We evaluate BullingerDB on text recognition and writer retrieval. TrOCR, the best performing model, achieves a CER of 9.1%. For writer retrieval, we introduce a temporal nDCG metric to assess time-aware retrieval. While temporally coherent retrieval is achievable, mAP (78.3%) scores indicate challenges due to long-term stylistic variation. With BullingerDB, we aim to establish a new benchmark for multilingual historical text recognition and temporally-aware writer analysis.
Marco Peer, Anna-Scius Bertrand, Patricia Scheurer +1
AIBEX Group, University of Fribourg, Switzerland · iCoSys Institute, University of Applied Sciences and Arts Western Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland