What Does an Ontology Catch? Curated, LLM-Generated and Data-Derived Ontologies as Checkers of LLM Agents over Knowledge Graphs

Michal Orzechowski; Rund Tawfiq; Tomasz Kosciolek

cs.AIOct 27, 2026

What Does an Ontology Catch? Curated, LLM-Generated and Data-Derived Ontologies as Checkers of LLM Agents over Knowledge Graphs

Authors: Michal Orzechowski, Rund Tawfiq, Tomasz Kosciolek

Organizations: ACK Cyfronet AGH, Krakow, Poland · AGH University of Krakow, Krakow, Poland · Sano Centre for Computational Medicine, Krakow, Poland

Abstract

LLM agents that answer questions over Web knowledge graphs are increasingly wrapped in checks derived from the graph's ontology, on the assumption that a formal schema makes a reliable judge. Yet a checker can be wrong as well: it can miss errors, and it can flag correct answers and push the agent into "fixing" them. Whether a curated ontology is worth its cost as such a judge, and how it compares with an ontology an LLM writes or a schema mined from the data, is unknown. We make the checker itself the object of study. Across curated, LLM-generated and data-derived ontologies, used as prompt context, as a checker, or both, and across open-weight and proprietary agents on public graphs and a private graph of nearly $10^{9}$ triples, we account for what ontology-derived checks catch and what they wrongly flag. We show how each source trades caught errors against false alarms, how a curated schema used as published can contradict its own data, which errors structural checks miss, and whether combining sources or giving the agent the checker's specific feedback pays off. On UniProt, the curated ontology flags more correct answers than either an ontology derived from the data or one an LLM derives from a summary of the data with masked identifiers. On real agent answers, its flags are only marginally more precise than random flags.

Figures & tables

Panel of Figure 1 — Figure 1. The setting. An LLM agent answers a question with a SPARQL query; a check derived from an ontology passes the answer or flags it, and a flag triggers one retry. The same ontology can also be the agent’s context. A flow diagram: a question goes to an LLM agent, which sends a SPARQL query to a knowledge graph; a check derived from an ontology reads the query and its results and either passes the answer or flags it, which sends the agent back for one retry. Below, an ontology from one of three sources (curated, written by an LLM, derived from the data) feeds the agent as context and the check.

Curated ontology (excerpt)

ex:birthDate rdfs:domain ex:Person .

ex:Person owl:disjointWith ex:City .

ex:birthPlace rdfs:domain ex:Person .

ex:birthPlace rdfs:range ex:Place .

ex:population is used by the data but not declared.

Data (excerpt)

Figure 2. Three answers checked against one curated ontology, in an invented vocabulary. Three SPARQL queries over an invented vocabulary: one wrong query is caught by a domain and disjointness check, one correct query is flagged because a property is undeclared, and one wrong query passes every check.

Graph	Triples	Curated ontology declares	Role
UniProt	$4.0\times 10^{8}$	domain, range, disjointness	confirmatory
Rhea	$5.5\times 10^{6}$	class hierarchy only	exploratory
Rhea, masked	$5.5\times 10^{6}$	as Rhea	exploratory
Corporate	$2.7\times 10^{4}$	domain, range; no disjointness	exploratory
Private	${\sim}10^{9}$	not described	rates only

Table 1. The graphs: size, what the curated ontology declares, and role.

Check from	catch	false alarm	unknown	catch, all	catch, rows returned	false alarm	precision at 25% wrong	fixed / corrupted
	Real answers			Artificial faults
Retry on empty result (reference)	2.5	0.1	—	79.2	—	0.0	100	15.4 / 0.0
Curated ontology, specific message	18.9	16.8	1.8	60.1	54.3	15.0	57.1	28.3 / 1.0
same flags, generic message	18.9	16.8	1.8	60.1	54.3	15.0	57.1	20.2 / 0.2
class checks only	3.1	0.6	—	17.0	—	1.1	83.4	—
class, domain and range	8.0	8.8	—	26.8	—	6.1	59.5	—

Table 2. Each ontology as a check on UniProt, in %: real answers of seven models, and artificial faults. Bold, per column: the highest catch rate, precision and share fixed; the lowest false-alarm rate, share unknown and share corrupted.

RQ1: as checks, on artificial faults
Comparison	Estimate (95% CI)	Margin	$p^{*}$
C2: false alarms on controls, curated $-$ data-derived	$+8.6$ (6.0, 11.2)	5	0.013
type-swap catches, curated $-$ data-derived	$-$ 22.3 ( $-$ 23.5, $-$ 21.0)	5
C3: false alarms on controls, curated $-$ masked LLM-generated	15.7 (13.7, 17.9)	5	0.025
RQ3: accuracy, and net detection for C4
C1a: context, curated $-$ data-derived	$-$ 2.1 ( $-$ 3.4, $-$ 0.9)	—	0.003

Table 3. Registered claims and exploratory comparisons on UniProt, in points, with 95% intervals; for the claims, the margin and the adjusted

p

(

p^{*}

). Bold: established.

Class	$n$	no rows	cur.	data	LLM-m	rand.
Invented identifier	2,986	80.5	77.4	77.5	74.5	48.1
Type swap	6,413	80.0	55.3	78.4	49.9	48.0
Same-type constant	1,554	46.8	3.1	5.8	0.0	47.9
Change of meaning	245	41.2	0.0	0.0	0.0	44.9
Correct items (false alarm)	3,630	0.0	15.0	9.7	1.2–1.3	—

Table 4. Catch rates by class of artificial fault on UniProt, in %. LLM-m: masked LLM-generated, mean of three; rand.: a random flag at the curated check’s rate. Bold: a random flag catches more than every ontology.

Graph	Disagreement
UniProt	Contradictions: up:maximum is declared xsd:float , but all 139 of its values are integers; 5 data properties declared functional have two distinct values for some subjects (82 entries for one property, 3,859 citations for four others). No instance of two disjoint classes.
Rhea	Omissions: 4 properties the data uses (between 15,032 and 90,295 times each) are not declared.
Private	94.2% of triples use an identifier other than the ontology’s own, mostly renamings of declared terms; 19.9% use a term with no counterpart in it; 1.46% have a subject typed outside the property’s declared domain (a mismatch).

Table 5. Where a curated ontology, as published, disagrees with its own data.

False alarm	undeclared term	closed world	data clash	schema too narrow	checker defect	other
Curated	17	11	0	0	0	0

Table 6. Why checks go wrong, in a blind audit of a sample: the curated check’s false alarms by judged cause (top) and missed wrong answers by kind of error (bottom), counted where both annotators gave the same code, out of 32 and 64 items; Cohen’s

\kappa

0.77 and 0.54.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Reads	Unknown when
Term exists	the query	—
Class membership	query; bindings	a value has no type
Domain and range	query; bindings	a value has no type
Disjointness	query; bindings	a value has no type
Datatype range	bindings	—
Numbers traceable	answer and results	—

Appendix

Table 7. The checks: what each reads, and when it reports unknown.

	Setup	Per answer checked	Per answer flagged
Check from	(US$)	triples read	tokens	time (s)
Curated	—	25 (277)	65,976 (435,066)	46.0 (420.0)
same flags, generic message	—	—	60,927 (308,426)	49.2 (309.7)
Data-derived	—	25 (269)	28,076 (202,405)	17.2 (112.3)
LLM-generated, draft 1	0.43	24 (267)	42,027 (181,818)	22.4 (170.5)
LLM-generated, draft 2	0.47	25 (269)	58,719 (455,836)	29.5 (456.1)

Appendix

Table 8. What checking costs on UniProt, per kind of ontology, pooled over the seven models (median, with the 90th percentile in parentheses).

	Accuracy, ontology in the prompt				Curated check
Model	none	curated	data-derived	LLM-generated	net change	message effect
Claude Sonnet 5.5	95.6	95.2	96.0	97.5	0.3	0.9
Claude Haiku 5.5	95.7	93.8	93.0	—	0.2	1.0
Qwen3.8-27B	80.8	79.7	80.8	81.9	0.8	0.0
Qwen3-Coder-30B-A3B	1.6	5.9	9.0	—	$-$ 0.2	$-$ 7.1
GLM-5.3-Flash	82.4	86.5	87.1	—	0.6	$-$ 1.8

Appendix

Table 9. Per model, on UniProt: accuracy with each ontology in the prompt, in %, and the curated check’s net change after its retry and the specific message’s effect per flagged answer (C5), in points.

Model	Curated	Data-derived	LLM-generated, masked
Claude Sonnet 5.5	7.1 / 17.4	0.0 / 1.7	0.0 / 1.2
Claude Haiku 5.5	14.8 / 15.9	0.0 / 5.0	0.0 / 2.2
Qwen3.8-27B	13.2 / 16.7	0.8 / 3.5	0.0 / 2.0
Qwen3-Coder-30B-A3B	32.4 / 30.0	35.3 / 10.0	14.7 / 0.0
GLM-5.3-Flash	14.3 / 18.1	1.9 / 5.6	1.9 / 2.7
Gemma-4-31B	30.2 / 19.2	24.0 / 6.1	16.7 / 5.1

Appendix

Table 10. Each model’s checks on UniProt, on its answers with no ontology in the prompt: catch / false alarm, in %. The three masked LLM-generated ontologies give identical rates.

	Real answers					Artificial faults
Check	catch	false al.	prec.	vs rand.	fix / corr.	catch	false al.	prec.
Retry on empty	2.5	0.1	81.3	$+65.8$	2 / 0	80.4	0.0	100
Curated	18.9	16.8	17.1	$+1.6$	28 / 5	59.7	13.2	60.1
Data-derived	8.2	3.5	29.9	$+14.4$	3 / 1	75.5	9.3	73.0
Combined	19.5	17.8	16.7	$+1.3$	30 / 5	—	—	—
LLM judge	40.3	16.2	31.3	$+15.8$	40 / 83	98.7 / 98.0	11.4 / 11.6	74.2 / 73.8

Appendix

Table 11. Comparators on UniProt, in %, on real answers and on artificial faults with a question text. Vs rand.: points by which precision beats a random flag at the same rate; fix / corr.: answers fixed and corrupted after one retry; the judge on faults: Sonnet / Haiku.

	Sonnet, UniProt	Sonnet, Rhea	Qwen3.8, Rhea	Qwen3.8, UniProt
Questions	170	101	101	170
Accuracy	11.8	11.9	9.2	9.0
Curated	34.3 / 11.7	41.8 / 41.7	35.3 / 46.4	35.3 / 4.3
vs rand.	$+8.7$	0.0	$-3.2$	$+10.7$
Data-derived	0.8 / 0.0	2.8 / 0.0	1.0 / 0.0	5.6 / 0.0
LLM-generated	2.4–18.4 / 5.0–28.3	3.3–6.6 / 8.3	4.8–8.7 / 7.1	0.9–18.0 / 6.5–19.6

Appendix

Table 12. Human-written questions (SIB) on UniProt and Rhea, no ontology in the prompt: accuracy and each check’s catch / false alarm, in %. Vs rand.: points by which precision beats a random flag; LLM-generated: range over three.

	Artificial faults (caught / false alarms)				Real answers (caught / false alarms)
Ontology	C	C+DR	+DJ (J)	full	C	C+DR	+DJ (J)	full
Curated	26.5 / 0.0	41.3 / 7.1	41.3 / 7.1	56.3 / 15.7	3.1 / 0.6	8.0 / 8.8	8.0 / 8.8	18.9 / 16.8
Data-derived	46.8 / 0.0	73.7 / 0.0	78.5 / 7.1	78.5 / 7.1	1.0 / 0.2	1.0 / 0.3	4.8 / 2.1	8.2 / 3.5
LLM-generated, masked	31.9 / 0.0	50.4 / 0.0	50.4 / 0.0	50.4 / 0.0	1.0 / 0.2	1.0 / 0.3	1.0 / 0.3	5.0 / 1.8
LLM-generated, ordinary	30.4 / 0.0	46.8 / 0.0	46.8 / 0.1	55.3 / 10.3	—	—	—	—

Appendix

Table 13. The checks layer by layer on UniProt, caught / false alarms in %: type swaps and untouched correct queries (artificial faults, per question), and real answers. C: class; DR: domain and range; +DJ: plus disjointness (J: the data-derived co-occurrence patterns); LLM-generated: mean of three.

				LLM-generated
Graph	accuracy	Curated	Data-derived	drafts 1–2	draft 3
Rhea, Sonnet	96.0	8.0 / 63.0	0.0 / 2.1	0.0 / 3.0–3.3	36.0 / 9.8
Rhea, Qwen3.8	92.1	20.4 / 39.1	0.0 / 3.3	0.0 / 4.5–4.7	16.3 / 11.4
Masked Rhea, Sonnet	68.3	58.8 / 57.0	0.0 / 0.0	0.5–4.0 / 0.0–3.7
Masked Rhea, Qwen3.8	30.0	25.9 / 20.6	1.9 / 2.6	0.9–2.8 / 2.6–3.7
Corporate, Sonnet	99.4	50.0 † / 24.8	0.0 † / 0.5	0.0 † / 0.0

Appendix

Table 14. The central comparison on the exploratory graphs, real answers of Sonnet and Qwen3.8: accuracy, and each check’s catch / false alarm, in %. LLM-generated: one range on the masked copy; identical on the corporate and private graphs. † Very few wrong answers.

Model		UniProt	Rhea	masked Rhea	corporate	masked $-$ Rhea
	a: UniProt	b: query from memory				agent accuracy
Sonnet	4.9	53.7	46.0	1.1	46.3	$-27.8$ ( $-32.6$ , $-22.9$ )
Haiku	1.4	39.0	24.8	1.3	0.0	$-35.2$ ( $-40.6$ , $-29.8$ )
Qwen3.8	0.8	0.0	4.9	0.0	26.7	$-62.1$ ( $-68.2$ , $-55.8$ )
Qwen3-Coder	0.0	1.0	0.0	1.1	0.0	$-2.1$ ( $-3.4$ , $-0.9$ )
GLM	1.7	26.7	21.7	0.0	37.5	$-52.9$ ( $-58.0$ , $-47.6$ )

Appendix

Table 15. Closed-book probes, exact match in %: an answer (a) and a query (b) written from memory; and the agents’ accuracy with no ontology, masked Rhea minus Rhea, in points with 95% intervals.

LLM Answer Verification

Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.

See all 151 papers

Agent-OM: Leveraging LLM Agents for Ontology Matching
Dec 1, 2023Zhangcheng Qiang, Weiqing Wang, Kerry TaylorSemantic Mapping
Australian National University, Canberra, ACT, Australia · Monash University, Melbourne, VIC, Australia
Toward Effective and Reliable LLM Agents via Dynamic Ontology
Aug 24, 2026Xiaohui Zhang, Zequn Sun, Chengyuan Yang +3LLM Agents LLM Agent Reliability
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Computer Science, Nanjing University of Information Science and Technology, China · School of Intelligence Science and Technology, Nanjing University, China +1
Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach
Apr 25, 2026Abid Talukder, Maruf Ahmed Mridul, Oshani SeneviratneMulti-Agent LLM Systems LLM Agent Orchestration
Rensselaer Polytechnic Institute, Troy, New York, United States

Dec 1, 2023J/K move · Enter open · S save

Agent-OM: Leveraging LLM Agents for Ontology Matching

Zhangcheng Qiang, Weiqing Wang, Kerry Taylor

Australian National University, Canberra, ACT, Australia · Monash University, Melbourne, VIC, Australia

Semantic Mapping

Open paper AskPDF arXiv

Ontology matching (OM) enables semantic interoperability between different ontologies and resolves their conceptual heterogeneity by aligning related entities. OM systems currently have two prevailing design paradigms: conventional knowledge-based expert systems and newer machine learning-based predictive systems. While large language models (LLMs) and LLM agents have revolutionised data engineering and have been applied creatively in many domains, their potential for OM remains underexplored. This study introduces a novel agent-powered LLM-based design paradigm for OM systems. With consideration of several specific challenges in leveraging LLM agents for OM, we propose a generic framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching, with a set of OM tools. Our framework is implemented in a proof-of-concept system. Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks over state-of-the-art OM systems show that our system can achieve results very close to the long-standing best performance on simple OM tasks and can significantly improve the performance on complex and few-shot OM tasks.

Abstract

LLM Answer Verification

Explore similar work

Agent-OM: Leveraging LLM Agents for Ontology Matching

Toward Effective and Reliable LLM Agents via Dynamic Ontology

Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach