Ontology-Guided Knowledge Graph Construction from Structured Metadata and Climate Science Publications

Aayush Acharya; Qi Zhang; Eduard Dragut; Longin Jan Latecki

cs.CLOct 23, 2026

Ontology-Guided Knowledge Graph Construction from Structured Metadata and Climate Science Publications

Authors: Aayush Acharya, Qi Zhang, Eduard Dragut, Longin Jan Latecki

Organizations: Temple University

Abstract

The rapid expansion of scientific literature far outpaces the speed of manual curation, creating a fundamental bottleneck in extracting structured knowledge from scientific texts. To address this, we introduce an agentic, ontology-guided framework for knowledge graph (KG) construction. Our framework utilizes specialized agents for document understanding, information extraction, ontology conformance, and entity linking. First, an LLM extracts factual knowledge in the form of triples with explicit textual evidence from research papers. Then the triplets undergo a multistage verification process to ground the triplets in the ontology. By enforcing schema constraints and requiring evidence in the form of text fragments, we ensure the data is both semantically consistent and traceable. In an agreement study over 1,000 extracted triplets, Gemini-2.5-flash achieves Cohen's $κ$ scores of 0.640--0.700 against three human annotators, matching the human--human agreement range of 0.616--0.689. We applied this framework to 2,000 scientific publications and construct a novel, ontology-grounded KG for climate science with 133,990 nodes and 755,308 relations; of the 23,327 triplets that the publications contribute after validation, 98.96% are judged correct. The pipeline code and KG is publicly available https://github.com/aayushacharya/climatekg.

Figures & tables

Figure 1: Ontology-guided, evidence-based KG construction pipeline. An LLM extracts candidate triples and evidence from scientific PDFs, which are validated for ontology conformance and textual grounding before final entity linking and acceptance.

Figure 2: Hybrid entity linking pipeline showing how embedding distance fails in cases where the similarity score is low and LLM selection helps.

Paper Interval	New Entities / Paper	Entity Novelty (%)	New Mentions / Paper	Mention Novelty (%)	Relation Novelty (%)
1–500	16.89	61.9	19.88	43.3	84.4
501–1,000	12.31	43.7	13.98	26.9	77.0
1,001–1,500	10.58	46.1	12.40	28.9	77.5
1,501–2,000	8.81	40.3	10.10	24.1	73.1

Table 1: Knowledge saturation over the 2,000-publication entity-linking run. Novelty is computed within each paper interval as the proportion of unique items that had not appeared in earlier papers. Relation novelty is computed over unique source–relation–target assertions.

Figure 3: Knowledge saturation as papers are added to the KG. Left: novelty rates are computed over non-overlapping 100-paper windows. Right: schema coverage is the cumulative fraction of final entity or relation types observed after each paper.

Comparison	Obs. Agreement	Cohen’s $\kappa$
Ann. 1 vs. Ann. 2	0.946	0.616
Ann. 1 vs. Ann. 3	0.956	0.667
Ann. 2 vs. Ann. 3	0.960	0.689
Gemini vs. Ann. 1	0.950	0.640
Gemini vs. Ann. 2	0.952	0.645
Gemini vs. Ann. 3	0.962	0.700

Table 2: Agreement among human annotators and Gemini-2.5-flash on 1,000 triplet validation decisions.

Stage	Input Triplets	Filtered	Output Triplets	$C_{\text{triplet}}$ (%)
Triplet and evidence extractor agent	–	–	39,964	94.05
Ontology conformance agent	39,964	11,050	28,914	94.29
Evidence validator agent	28,914	1,014	27,900	94.72
Validator agent	27,900	4,573	23,327	98.96

Table 3: Per-stage attrition over the full 2,000-publication corpus, evaluated by Gemini-2.5-flash. Each row reports the output of the cumulative pipeline through that stage.

Stages applied	Triplets kept	$C_{\text{triplet}}$ (%)
none (extractor)	39,964	94.05
O	28,914	94.29
E	38,386	94.38
V	32,436	98.86
O+E	27,900	94.72
O+V	24,172	98.96

Table 4: Stage ablation over the same 39,964 candidate triplets. O = ontology conformance agent, E = evidence validator agent, V = validator agent. Every triplet stores the verdict of every stage, so all eight variants are evaluated post hoc with the same

C_{\text{triplet}}

judge.

Statistic	Value
Scientific publications	2,000
KG nodes	133,990
from initialization	60,707
added from publications	73,283
KG relations	755,308
from initialization	456,563

Table 5: Aggregate statistics of the constructed climate KG after enrichment with 2,000 scientific publications. The split rows separate what the deterministic initialization of Section 3.4 contributes from what the extraction pipeline adds, and are computed from the graph build files, which hold two relations more than the loaded database.

Metric	KARMA $\dagger$	Ours
	Lu and Wang (2025)	(after validator)
Output Triplets	3,820	17,141
Valid	3,525	16,981
Invalid	295	160
$C_{\text{triplet}}$ (%)	92.277	99.067

Table 6: Comparison with KARMA over the first 1,007 scientific publications after final validation

Metric	ClimateIE	Ours (with
	Pan et al. (2025a)	ClimateIE ontology)
Output Triplets	5,729	637
Valid	4,297	592
Invalid	1,432	45
$C_{\text{triplet}}$ (%)	75.004	92.936

Table 7: Comparison with ClimateIE over the 25 papers in the ClimateIE human corpus, using the ClimateIE ontology for both systems and Gemini-2.5-flash validation.

Entity type	Gold	Extracted	Recall (%)
Source	797	774	97.11
Experiment	196	162	82.65
Frequency	85	57	67.06
Activity	55	30	54.55
Micro avg.	1,133	1,023	90.29

Table 8: Entity-level recall against CMIP Publication Hub metadata for 139 papers, computed over gold entities whose surface form is recoverable from the parsed paper text.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Pipeline for selection of relevant scientific publications. The pipeline starts with a manual seed of highly relevant scientific publications, followed by iterative citation mining and LLM-based filtering to build a high-quality corpus.

String Matching Path	Embedding Similarity Path
Strict rigid identifiers	Semantic or contextual concepts
Exact name matches; standardized taxonomy terms; known domain abbreviations; official model names (e.g., CESM2); institution names.	Semantic variations of variables; context-dependent entities; geographic locations; weather phenomena descriptions; abstract research concepts.

Appendix

Table 9: Categorization strategy for routing extracted entities to the appropriate linking mechanism.

Score Bin	Matched	Rejected	Total	Rate
$[0.50,\;0.55)$	186	185	371	50.1%
$[0.55,\;0.60)$	281	205	486	57.8%
$[0.60,\;0.65)$	406	283	689	58.9%
$[0.65,\;0.70)$	287	232	519	55.3%
$[0.70,\;0.75)$	261	170	431	60.6%
$[0.75,\;0.80)$	370	144	514	72.0%

Appendix

Table 10: Entity-linking sensitivity analysis by top-1 embedding similarity score bin.

Stage	Mechanism	Cost (2,000 papers)	Ablation effect
Paper screening	Llama-3.3-70B, local	no API cost, GPU time	corpus selection, not evaluated
PDF parsing	olmOCR, local	no API cost, GPU time	prerequisite for evidence grounding
Triplet + evidence extraction	Gemini-2.5-pro, $\approx$ 5 calls/paper	$\approx$ USD 400, 2–3 h per 1,000 papers	produces 39,964 triplets at 94.05%
Ontology conformance (O)	deterministic set membership	no API cost, CPU seconds	w/o: 31,095 at 98.88%, 7,768 untypeable kept
Evidence validation (E)	deterministic fuzzy match	no API cost, CPU minutes	w/o: 24,172 at 98.96%, 845 ungrounded kept
Validator agent (V)	Gemini-2.5-flash, 1 call/triplet	$\approx$ USD 3 (27,900 calls)	w/o: 27,900 at 94.72%, $-4.24$ points

Appendix

Table 11: Per-stage cost, runtime, and ablation effect for the full 2,000-publication run. Ablation effect reports the result of leaving that stage out of the full pipeline, which itself yields 23,327 triplets at 98.96%

C_{\text{triplet}}

Experimental setting	Threshold	Rate (%)
Evidence span	80%	7.741
Evidence span	75%	6.793
Evidence span	70%	6.137
Evidence span without table/figure interpretations	80%	1.443
Evidence span without table/figure interpretations	75%	1.035
Evidence span without table/figure interpretations	70%	0.626

Appendix

Table 12: Hallucination analysis of an LLM-based KG extractor agent with evidence.

Figure 5: Overview of the ontology-guided, evidence-based KG construction pipeline. The workflow proceeds in three phases: (1) Paper Screening and Feature Extraction , where documents are filtered for empirical relevance (Step 1–3); (2) Extraction and Verification , where an LLM extracts entity-relation triples supported by explicit textual evidence and verifies ontology conformance (Step 4–5); and (3) Hybrid Entity Linking , which routes entities to either deterministic string matching or semantic embedding similarity (Step 6) to determine whether to discard the entity or integrate it into the Updated Climate KG.

Figure 6: A small subsection of structured ESGF metadata converted to KG based on ontology specifications mapped with geolocation data from Geonames.

Figure 7: Decision flow for entity linking. Entities are linked via exact string matching when strict naming conventions apply; otherwise, embedding-based similarity with adaptive thresholds and LLM verification is used to link, add, or discard entities from the KG.

Entity Type	Count	Entity Type	Count
City	30,062	Ensemble	578
Result	16,207	Experiment	481
Author	10,437	Source	394
EvaluationMetric	8,430	Venue	373
No_Country_Region	6,965	Country	252
PhysicalScheme	6,568	Forcing	204

Appendix

Table 13: Distribution of entity types in the constructed KG.

Relation Type	Count
PAPER_MENTIONS	96,100
RESULT_HAS_METRIC	35,522
PAPER_HAS_RESULT	29,879
METHOD_HAS_RESULT	29,879
PAPER_HAS_KEYWORD	22,644
PAPER_AUTHORED_BY	14,853

Appendix

Table 14: Distribution of scientific paper relation types in the constructed KG.

Relation Type	Count
COVERS_REGION	354,837
IN_COUNTRY	33,923
PRODUCES_VARIABLE	31,024
ASSOCIATED_WITH_MEMBER	11,903
HAS_SPATIAL_RESOLUTION	6,708
APPLIES_TO_REALM	4,031

Appendix

Table 15: Distribution of climate relation types in the constructed KG.

KG Construction

Knowledge graph construction turns text, tables, images, and databases into structured entities and relationships that machines can query and reason over.

See all 195 papers

Agents-K1: Towards Agent-native Knowledge Orchestration
Jun 11, 2026Zongsheng Cao, Bihao Zhan, Jinxin Shi +23KG Construction AI for Science
Shanghai Artificial Intelligence Laboratory · East China Normal University · Fudan University
HGNet: Scalable Foundation Model for Automated Knowledge Graph Generation from Scientific Literature
Mar 24, 2026Devvrat Joshi, Islem RekikKG Construction Relation Extraction
BASIRA Lab, Imperial-X (I-X) and Department of Computing Imperial College London, London, United Kingdom
Enabling Knowledge Graph Understanding at Scale with the EXplore Your Graphs ENgine (EXYGEN)
Sep 10, 2026Harshdeep Singh, Yurui Zhu, Giovanni Colavizza +1Graph Retrieval-Augmented Generation Knowledge Graph Question Answering
Odoma Ltd.

Jun 11, 2026J/K move · Enter open · S save

Agents-K1: Towards Agent-native Knowledge Orchestration

Zongsheng Cao, Bihao Zhan, Jinxin Shi, Jiong Wang+22

Shanghai Artificial Intelligence Laboratory · East China Normal University · Fudan University

KG Construction AI for Science Document Information Extraction Knowledge Representation LLM Agents

Open paper AskPDF arXiv

Current LLM-based research agents have advanced through agent orchestration, yet largely overlook scientific knowledge orchestration. Existing works often reduce papers to abstracts, surface mentions, and flat cites edges, omitting key entities, claims, evidence, mechanisms, and method lineages essential for scientific reasoning. To this end, we introduce Agents-K1, an end-to-end knowledge orchestration pipeline that converts raw documents into agent-native scientific knowledge graphs. Agents-K1 integrates three components under a unifying theoretical foundation: a multimodal parser whose five-module schema captures entities, multimodal evidence, citations, and typed inter-entity relations across the full paper rather than abstracts alone; a 4B information-extraction backbone trained with GRPO under a rule-based reward; and a graphanything CLI, a tri-source agent interface that unifies web search, multimodal graph retrieval, and cross-document traversal. On top of this, we process 2.46 million scientific papers across six subjects to produce Scholar-KG, of which we release a one-million-paper subset, and the full Scholar-KG is accessible via the SCP link below. The same pipeline can be extended to general-domain corpora and to schema-conformant data synthesis. Extensive experiments demonstrate that Agents-K1 achieves superior performance in scientific information extraction, knowledge graph construction, and multi-hop scientific reasoning.

Abstract

KG Construction

Explore similar work

Agents-K1: Towards Agent-native Knowledge Orchestration

HGNet: Scalable Foundation Model for Automated Knowledge Graph Generation from Scientific Literature

Enabling Knowledge Graph Understanding at Scale with the EXplore Your Graphs ENgine (EXYGEN)