Existing scholarly information extraction (SIE) datasets focus on scientific papers and overlook implementation-level details in code repositories. README files describe datasets, source code, and other implementation-level artifacts, however, their free-form Markdown offers little semantic structure, making automatic information extraction difficult. To address this gap, NERdME is introduced: 200 manually annotated README files with over 10,000 labeled spans and 10 entity types. Baseline results using large language models and fine-tuned transformers show clear differences between paperlevel and implementation-level entities, indicating the value of extending SIE benchmarks with entity types available in README files. A downstream entity-linking experiment was conducted to demonstrate that entities derived from READMEs can support artifact discovery and metadata integration.
Figures & tables
Dataset
Source
#ET
#Doc
#Mention
Manual
ILE
PLE
Span
SciERC ( Luan et al., 2018 )
Paper abstract
6
500
8,089
✓
✗
✓
✓
SciREX ( Jain et al., 2020 )
Full papers
4
438
156,931
✓
✗
✓
✓
SciDMT ( Pan et al., 2024 )
Full papers
3
48k
1,830,886
✗
✗
✓
✓
GSAP-NER ( Otto et al., 2023 )
Full papers
10
100
54,598
✓
✗
✓
✓
SoMeSci ( Schindler et al., 2021 )
Full papers
9
1367
7,237
✓
✓
✗
✓
Hidden Entity ( Gan et al., 2025 )
GitHub READMEs
4
811
1,439
✓
✓
✗
✗
Table 1. Comparison of NERdME with existing SIE datasets.
Figure 1. Illustration of an annotated README file in the dataset. Note that unlabeled sentences are NOT negative samples.
Figure 2. Span statistics in NERdME. Bars show the proportion of spans across train, validation, and test splits for each entity type; numbers in parentheses indicate total spans per type.
Group
UniqueRatio
FormLength
PPL
EditDist
Paper
0.56
10.00
1,326
0.92
Implementation
0.29
6.00
2,509
0.94
Table 2. Linguistic comparison of entity groups.
Approach
CONFERENCE
DATASET
EVALMETRIC
LICENSE
ONTOLOGY
PROGLANG
PROJECT
PUBLICATION
SOFTWARE
WORKSHOP
F1-P.
Zero-shot LLM
77.38
50.98
37.07
87.34
45.98
34.41
56.23
70.42
29.96
20.68
Fine-tuned
82.29
69.94
60.18
86.62
79.43
82.72
47.36
90.16
77.25
13.27
F1-E.
Zero-shot LLM
47.63
45.30
33.94
86.15
18.52
31.69
53.72
64.95
23.12
20.68
Fine-tuned
58.38
63.94
44.78
78.38
73.13
79.96
34.22
82.12
72.46
0.00
Table 3. Best results of various language models for individual entity type on NERdME under partial and exact match F1 score. F1-P. and F1-E. represent partial and exact match F1 score, respectively. The complete scores for all models and entity types are available in GitHub repository.
Method
P
R
F1
Hits@1
Hits@3
MRR
Fuzzy
26.84
13.17
17.67
15.24
15.29
15.27
Semantic
65.18
38.68
47.75
37.68
37.72
37.69
Table 4. Entity linking performance on NERdME entities under different linking methods. Best results are marked in bold.