LLM agents that answer questions over Web knowledge graphs are increasingly wrapped in checks derived from the graph's ontology, on the assumption that a formal schema makes a reliable judge. Yet a checker can be wrong as well: it can miss errors, and it can flag correct answers and push the agent into "fixing" them. Whether a curated ontology is worth its cost as such a judge, and how it compares with an ontology an LLM writes or a schema mined from the data, is unknown. We make the checker itself the object of study. Across curated, LLM-generated and data-derived ontologies, used as prompt context, as a checker, or both, and across open-weight and proprietary agents on public graphs and a private graph of nearly 109 triples, we account for what ontology-derived checks catch and what they wrongly flag. We show how each source trades caught errors against false alarms, how a curated schema used as published can contradict its own data, which errors structural checks miss, and whether combining sources or giving the agent the checker's specific feedback pays off. On UniProt, the curated ontology flags more correct answers than either an ontology derived from the data or one an LLM derives from a summary of the data with masked identifiers. On real agent answers, its flags are only marginally more precise than random flags.
Figures & tables
Figure 1. The setting. An LLM agent answers a question with a SPARQL query; a check derived from an ontology passes the answer or flags it, and a flag triggers one retry. The same ontology can also be the agent’s context. A flow diagram: a question goes to an LLM agent, which sends a SPARQL query to a knowledge graph; a check derived from an ontology reads the query and its results and either passes the answer or flags it, which sends the agent back for one retry. Below, an ontology from one of three sources (curated, written by an LLM, derived from the data) feeds the agent as context and the check.
Curated ontology (excerpt)
ex:birthDate rdfs:domain ex:Person .
ex:Person owl:disjointWith ex:City .
ex:birthPlace rdfs:domain ex:Person .
ex:birthPlace rdfs:range ex:Place .
ex:population is used by the data but not declared.
Data (excerpt)
Figure 2. Three answers checked against one curated ontology, in an invented vocabulary. Three SPARQL queries over an invented vocabulary: one wrong query is caught by a domain and disjointness check, one correct query is flagged because a property is undeclared, and one wrong query passes every check.
Graph
Triples
Curated ontology declares
Role
UniProt
4.0×108
domain, range, disjointness
confirmatory
Rhea
5.5×106
class hierarchy only
exploratory
Rhea, masked
5.5×106
as Rhea
exploratory
Corporate
2.7×104
domain, range; no disjointness
exploratory
Private
∼109
not described
rates only
Table 1. The graphs: size, what the curated ontology declares, and role.
Real answers
Artificial faults
Check from
catch
false alarm
unknown
catch, all
catch, rows returned
false alarm
precision at 25% wrong
fixed / corrupted
Retry on empty result (reference)
2.5
0.1
—
79.2
—
0.0
100
15.4 / 0.0
Curated ontology, specific message
18.9
16.8
1.8
60.1
54.3
15.0
57.1
28.3 / 1.0
same flags, generic message
18.9
16.8
1.8
60.1
54.3
15.0
57.1
20.2 / 0.2
class checks only
3.1
0.6
—
17.0
—
1.1
83.4
—
class, domain and range
8.0
8.8
—
26.8
—
6.1
59.5
—
Table 2. Each ontology as a check on UniProt, in %: real answers of seven models, and artificial faults. Bold, per column: the highest catch rate, precision and share fixed; the lowest false-alarm rate, share unknown and share corrupted.
Comparison
Estimate (95% CI)
Margin
p∗
RQ1: as checks, on artificial faults
C2: false alarms on controls, curated − data-derived
+8.6 (6.0, 11.2)
5
0.013
type-swap catches, curated − data-derived
− 22.3 ( − 23.5, − 21.0)
5
C3: false alarms on controls, curated − masked LLM-generated
15.7 (13.7, 17.9)
5
0.025
RQ3: accuracy, and net detection for C4
C1a: context, curated − data-derived
− 2.1 ( − 3.4, − 0.9)
—
0.003
Table 3. Registered claims and exploratory comparisons on UniProt, in points, with 95% intervals; for the claims, the margin and the adjusted p ( p∗ ). Bold: established.
Class
n
no rows
cur.
data
LLM-m
rand.
Invented identifier
2,986
80.5
77.4
77.5
74.5
48.1
Type swap
6,413
80.0
55.3
78.4
49.9
48.0
Same-type constant
1,554
46.8
3.1
5.8
0.0
47.9
Change of meaning
245
41.2
0.0
0.0
0.0
44.9
Correct items (false alarm)
3,630
0.0
15.0
9.7
1.2–1.3
—
Table 4. Catch rates by class of artificial fault on UniProt, in %. LLM-m: masked LLM-generated, mean of three; rand.: a random flag at the curated check’s rate. Bold: a random flag catches more than every ontology.
Graph
Disagreement
UniProt
Contradictions: up:maximum is declared xsd:float , but all 139 of its values are integers; 5 data properties declared functional have two distinct values for some subjects (82 entries for one property, 3,859 citations for four others). No instance of two disjoint classes.
Rhea
Omissions: 4 properties the data uses (between 15,032 and 90,295 times each) are not declared.
Private
94.2% of triples use an identifier other than the ontology’s own, mostly renamings of declared terms; 19.9% use a term with no counterpart in it; 1.46% have a subject typed outside the property’s declared domain (a mismatch).
Table 5. Where a curated ontology, as published, disagrees with its own data.
False alarm
undeclared term
closed world
data clash
schema too narrow
checker defect
other
Curated
17
11
0
0
0
0
Table 6. Why checks go wrong, in a blind audit of a sample: the curated check’s false alarms by judged cause (top) and missed wrong answers by kind of error (bottom), counted where both annotators gave the same code, out of 32 and 64 items; Cohen’s κ 0.77 and 0.54.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Check
Reads
Unknown when
Term exists
the query
—
Class membership
query; bindings
a value has no type
Domain and range
query; bindings
a value has no type
Disjointness
query; bindings
a value has no type
Datatype range
bindings
—
Numbers traceable
answer and results
—
Appendix
Table 7. The checks: what each reads, and when it reports unknown.
Setup
Per answer checked
Per answer flagged
Check from
(US$)
triples read
tokens
time (s)
Curated
—
25 (277)
65,976 (435,066)
46.0 (420.0)
same flags, generic message
—
—
60,927 (308,426)
49.2 (309.7)
Data-derived
—
25 (269)
28,076 (202,405)
17.2 (112.3)
LLM-generated, draft 1
0.43
24 (267)
42,027 (181,818)
22.4 (170.5)
LLM-generated, draft 2
0.47
25 (269)
58,719 (455,836)
29.5 (456.1)
Appendix
Table 8. What checking costs on UniProt, per kind of ontology, pooled over the seven models (median, with the 90th percentile in parentheses).
Accuracy, ontology in the prompt
Curated check
Model
none
curated
data-derived
LLM-generated
net change
message effect
Claude Sonnet 5.5
95.6
95.2
96.0
97.5
0.3
0.9
Claude Haiku 5.5
95.7
93.8
93.0
—
0.2
1.0
Qwen3.8-27B
80.8
79.7
80.8
81.9
0.8
0.0
Qwen3-Coder-30B-A3B
1.6
5.9
9.0
—
− 0.2
− 7.1
GLM-5.3-Flash
82.4
86.5
87.1
—
0.6
− 1.8
Appendix
Table 9. Per model, on UniProt: accuracy with each ontology in the prompt, in %, and the curated check’s net change after its retry and the specific message’s effect per flagged answer (C5), in points.
Model
Curated
Data-derived
LLM-generated, masked
Claude Sonnet 5.5
7.1 / 17.4
0.0 / 1.7
0.0 / 1.2
Claude Haiku 5.5
14.8 / 15.9
0.0 / 5.0
0.0 / 2.2
Qwen3.8-27B
13.2 / 16.7
0.8 / 3.5
0.0 / 2.0
Qwen3-Coder-30B-A3B
32.4 / 30.0
35.3 / 10.0
14.7 / 0.0
GLM-5.3-Flash
14.3 / 18.1
1.9 / 5.6
1.9 / 2.7
Gemma-4-31B
30.2 / 19.2
24.0 / 6.1
16.7 / 5.1
Appendix
Table 10. Each model’s checks on UniProt, on its answers with no ontology in the prompt: catch / false alarm, in %. The three masked LLM-generated ontologies give identical rates.
Real answers
Artificial faults
Check
catch
false al.
prec.
vs rand.
fix / corr.
catch
false al.
prec.
Retry on empty
2.5
0.1
81.3
+65.8
2 / 0
80.4
0.0
100
Curated
18.9
16.8
17.1
+1.6
28 / 5
59.7
13.2
60.1
Data-derived
8.2
3.5
29.9
+14.4
3 / 1
75.5
9.3
73.0
Combined
19.5
17.8
16.7
+1.3
30 / 5
—
—
—
LLM judge
40.3
16.2
31.3
+15.8
40 / 83
98.7 / 98.0
11.4 / 11.6
74.2 / 73.8
Appendix
Table 11. Comparators on UniProt, in %, on real answers and on artificial faults with a question text. Vs rand.: points by which precision beats a random flag at the same rate; fix / corr.: answers fixed and corrupted after one retry; the judge on faults: Sonnet / Haiku.
Sonnet, UniProt
Sonnet, Rhea
Qwen3.8, Rhea
Qwen3.8, UniProt
Questions
170
101
101
170
Accuracy
11.8
11.9
9.2
9.0
Curated
34.3 / 11.7
41.8 / 41.7
35.3 / 46.4
35.3 / 4.3
vs rand.
+8.7
0.0
−3.2
+10.7
Data-derived
0.8 / 0.0
2.8 / 0.0
1.0 / 0.0
5.6 / 0.0
LLM-generated
2.4–18.4 / 5.0–28.3
3.3–6.6 / 8.3
4.8–8.7 / 7.1
0.9–18.0 / 6.5–19.6
Appendix
Table 12. Human-written questions (SIB) on UniProt and Rhea, no ontology in the prompt: accuracy and each check’s catch / false alarm, in %. Vs rand.: points by which precision beats a random flag; LLM-generated: range over three.
Artificial faults (caught / false alarms)
Real answers (caught / false alarms)
Ontology
C
C+DR
+DJ (J)
full
C
C+DR
+DJ (J)
full
Curated
26.5 / 0.0
41.3 / 7.1
41.3 / 7.1
56.3 / 15.7
3.1 / 0.6
8.0 / 8.8
8.0 / 8.8
18.9 / 16.8
Data-derived
46.8 / 0.0
73.7 / 0.0
78.5 / 7.1
78.5 / 7.1
1.0 / 0.2
1.0 / 0.3
4.8 / 2.1
8.2 / 3.5
LLM-generated, masked
31.9 / 0.0
50.4 / 0.0
50.4 / 0.0
50.4 / 0.0
1.0 / 0.2
1.0 / 0.3
1.0 / 0.3
5.0 / 1.8
LLM-generated, ordinary
30.4 / 0.0
46.8 / 0.0
46.8 / 0.1
55.3 / 10.3
—
—
—
—
Appendix
Table 13. The checks layer by layer on UniProt, caught / false alarms in %: type swaps and untouched correct queries (artificial faults, per question), and real answers. C: class; DR: domain and range; +DJ: plus disjointness (J: the data-derived co-occurrence patterns); LLM-generated: mean of three.
LLM-generated
Graph
accuracy
Curated
Data-derived
drafts 1–2
draft 3
Rhea, Sonnet
96.0
8.0 / 63.0
0.0 / 2.1
0.0 / 3.0–3.3
36.0 / 9.8
Rhea, Qwen3.8
92.1
20.4 / 39.1
0.0 / 3.3
0.0 / 4.5–4.7
16.3 / 11.4
Masked Rhea, Sonnet
68.3
58.8 / 57.0
0.0 / 0.0
0.5–4.0 / 0.0–3.7
Masked Rhea, Qwen3.8
30.0
25.9 / 20.6
1.9 / 2.6
0.9–2.8 / 2.6–3.7
Corporate, Sonnet
99.4
50.0 † / 24.8
0.0 † / 0.5
0.0 † / 0.0
Appendix
Table 14. The central comparison on the exploratory graphs, real answers of Sonnet and Qwen3.8: accuracy, and each check’s catch / false alarm, in %. LLM-generated: one range on the masked copy; identical on the corporate and private graphs. † Very few wrong answers.
a: UniProt
b: query from memory
agent accuracy
Model
UniProt
Rhea
masked Rhea
corporate
masked − Rhea
Sonnet
4.9
53.7
46.0
1.1
46.3
−27.8 ( −32.6 , −22.9 )
Haiku
1.4
39.0
24.8
1.3
0.0
−35.2 ( −40.6 , −29.8 )
Qwen3.8
0.8
0.0
4.9
0.0
26.7
−62.1 ( −68.2 , −55.8 )
Qwen3-Coder
0.0
1.0
0.0
1.1
0.0
−2.1 ( −3.4 , −0.9 )
GLM
1.7
26.7
21.7
0.0
37.5
−52.9 ( −58.0 , −47.6 )
Appendix
Table 15. Closed-book probes, exact match in %: an answer (a) and a query (b) written from memory; and the agents’ accuracy with no ontology, masked Rhea minus Rhea, in points with 95% intervals.
Ontology matching (OM) enables semantic interoperability between different ontologies and resolves their conceptual heterogeneity by aligning related entities. OM systems currently have two prevailing design paradigms: conventional knowledge-based expert systems and newer machine learning-based predictive systems. While large language models (LLMs) and LLM agents have revolutionised data engineering and have been applied creatively in many domains, their potential for OM remains underexplored. This study introduces a novel agent-powered LLM-based design paradigm for OM systems. With consideration of several specific challenges in leveraging LLM agents for OM, we propose a generic framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching, with a set of OM tools. Our framework is implemented in a proof-of-concept system. Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks over state-of-the-art OM systems show that our system can achieve results very close to the long-standing best performance on simple OM tasks and can significantly improve the performance on complex and few-shot OM tasks.
Zhangcheng Qiang, Weiqing Wang, Kerry Taylor
Australian National University, Canberra, ACT, Australia · Monash University, Melbourne, VIC, Australia
Large language model (LLM) agents rely heavily on knowledge encoded in model parameters or presented as unstructured context. In domain-specific tasks, this leaves important semantic connections implicit. This often results in incomplete evidence use and brittle multi-step decisions. Ontologies offer a way to externalize domain concepts and relations as machine-interpretable structures, but constructing task-usable ontologies traditionally requires substantial effort from domain experts and is difficult to scale. Automatic construction is also challenging: an ontology that appears semantically plausible may not contain the relational structures needed for actual decision making. We present OaK, an ontology-as-a-kernel framework that dynamically constructs and refines task-oriented ontologies for LLM agents. Given task requirements and training data, OaK constructs an ontology and its knowledge graph, generates task-adaptation functions for graph reasoning, and uses judge feedback to iteratively refine both. By making relevant concepts and relations explicit, the ontology grounds knowledge retrieval and multi-step decision making. We evaluate OaK on TravelPlanner, CRMArenaPro, and ToolQA. Results show that OaK improves standard LLM agents, strengthens evidence grounding, and boosts the reliability of multi-step reasoning.
Xiaohui Zhang, Zequn Sun, Chengyuan Yang +3
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Computer Science, Nanjing University of Information Science and Technology, China · School of Intelligence Science and Technology, Nanjing University, China +1
Automatically generating formal ontologies from unstructured natural language remains a central challenge in knowledge engineering. While large language models (LLMs) show promise, it remains unclear which architectural design choices drive generation quality and why current approaches fail. We present a controlled experimental study using domain-specific insurance contracts to investigate these questions. We first establish a single-agent LLM baseline, identifying key failure modes such as poor Ontology Design Pattern compliance, structural redundancy, and ineffective iterative repair. We then introduce a multi-agent architecture that decomposes ontology construction into four artifact-driven roles: Domain Expert, Manager, Coder, and Quality Assurer. We evaluate performance across architectural quality (via a panel of heterogeneous LLM judges) and functional usability (via competency question driven SPARQL evaluation with complementary retrieval augmented generation based assessment). Results show that the multi-agent approach significantly improves structural quality and modestly enhances queryability, with gains driven primarily by front-loaded planning. These findings highlight planning-first, artifact-driven generation as a promising and more auditable path toward scalable automated ontology engineering.
Abid Talukder, Maruf Ahmed Mridul, Oshani Seneviratne
Rensselaer Polytechnic Institute, Troy, New York, United States