Toward Effective and Reliable LLM Agents via Dynamic Ontology

Xiaohui Zhang; Zequn Sun; Chengyuan Yang; Yuanning Cui; Lingbing Guo; Wei Hu

cs.AIAug 24, 2026

Toward Effective and Reliable LLM Agents via Dynamic Ontology

Authors: Xiaohui Zhang, Zequn Sun, Chengyuan Yang, Yuanning Cui, Lingbing Guo, Wei Hu

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Computer Science, Nanjing University of Information Science and Technology, China · School of Intelligence Science and Technology, Nanjing University, China · National Institute of Healthcare Data Science, Nanjing University, China

Abstract

Large language model (LLM) agents rely heavily on knowledge encoded in model parameters or presented as unstructured context. In domain-specific tasks, this leaves important semantic connections implicit. This often results in incomplete evidence use and brittle multi-step decisions. Ontologies offer a way to externalize domain concepts and relations as machine-interpretable structures, but constructing task-usable ontologies traditionally requires substantial effort from domain experts and is difficult to scale. Automatic construction is also challenging: an ontology that appears semantically plausible may not contain the relational structures needed for actual decision making. We present OaK, an ontology-as-a-kernel framework that dynamically constructs and refines task-oriented ontologies for LLM agents. Given task requirements and training data, OaK constructs an ontology and its knowledge graph, generates task-adaptation functions for graph reasoning, and uses judge feedback to iteratively refine both. By making relevant concepts and relations explicit, the ontology grounds knowledge retrieval and multi-step decision making. We evaluate OaK on TravelPlanner, CRMArenaPro, and ToolQA. Results show that OaK improves standard LLM agents, strengthens evidence grounding, and boosts the reliability of multi-step reasoning.

Figures & tables

Figure 1: Overview of OaK. The construction stage (left) runs a refinement loop over the ontology kernel. The inference stage (right) freezes the kernel and lets a ReAct agent solve unseen queries by calling functions as tools.

LLM	Method	CS		HC		Final
		Micro	Macro	Micro	Macro
DeepSeek-v4-flash	ReAct	81.60	19.10	44.29	36.40	15.30
	AFlow	79.31	31.70	43.60	42.70	29.10
	MemP	82.78	55.50	63.91	59.50	51.50
	ReCode	86.60	50.90	48.67	37.60	48.20
	AgentSquare	79.10	30.40	49.40	47.80	27.70

Table 1: TravelPlanner results on DeepSeek-v4-flash and gpt-4o-mini. All entries are percentages (%). CS = Commonsense Constraint Pass Rate; HC = Hard Constraint Pass Rate; Final = Final Pass Rate. The best value in each column is in bold and the second best is underlined .

LLM	Method	B2B					B2C
		Workflow	Policy	Text	Database	Avg.	Workflow	Policy	Text	Database	Avg.
DeepSeek-v4-flash	ReAct	92.50	47.50	31.57	71.56	60.78	88.75	45.62	43.47	65.62	60.87
	AFlow	96.25	46.25	33.87	48.44	56.20	93.75	47.50	41.84	33.75	54.21
	MemP	96.25	55.62	35.74	78.12	66.44	95.00	61.25	43.77	69.06	67.27
	ReCode	26.25	39.38	10.13	24.06	24.96	27.50	34.38	11.71	24.06	24.41
	AgentSquare	95.00	46.88	31.10	50.94	55.98	88.75	56.25	36.37	44.06	56.36

Table 2: CRMArenaPro results on DeepSeek-v4-flash and gpt-4o-mini. All entries are percentages (%); Workflow, Policy, Text, and Database are task-category scores. Avg. is the equal-weight mean of Workflow, Policy, Text, and Database. The best value in each column is in bold and the second best is underlined .

LLM	Method	Flight	Coffee	Airbnb	DBLP	Yelp	Avg.
DeepSeek-v4-flash	AFlow	57.78	49.28	53.89	35.56	70.00	53.18
	AgentSquare	46.67	41.06	50.56	28.89	63.89	46.06
	ReAct	37.78	48.79	48.33	43.18	82.20	51.96
	ReCode	28.95	18.40	46.15	12.50	57.14	32.21
	MemP	55.56	48.79	54.44	35.00	82.22	55.02
	OaK	66.67	64.25	58.33	44.94	88.76	64.58

Table 3: ToolQA results on the five subsets, using DeepSeek-v4-flash and gpt-4o-mini as backbones. All entries are percentages (%) and report exact-match accuracy. Avg. is the weighted average across subsets. The best value in each column is in bold and the second best is underlined .

Figure 2: Ablation results on TravelPlanner.

Figure 3: Ablation results on CRMArenaPro.

Figure 4: Ablation results on ToolQA.

Figure 5: Loop progress on TravelPlanner, CRMArenaPro, and ToolQA using DeepSeek-v4-flash.

Figure 8: Resource cost versus final pass rate on TravelPlanner using DeepSeek-v4-flash.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Seed	CS		HC		Final
	Micro	Macro	Micro	Macro
1	83.50	55.90	58.31	62.40	54.00
2	88.77	60.40	60.46	61.90	58.20
3	86.06	59.50	59.16	60.40	55.50
Avg. $\pm$ Std.	$86.11\pm 2.64$	$58.60\pm 2.38$	$59.31\pm 1.08$	$61.57\pm 1.04$	$55.90\pm 2.13$

Appendix

Table 4: Multi-seed robustness of OaK on TravelPlanner using DeepSeek-v4-flash. All entries are percentages (%).

Figure 9: Ablation results on TravelPlanner using gpt-4o-mini.

Figure 10: Ablation results on CRMArenaPro using gpt-4o-mini.

Figure 11: Ablation results on ToolQA using gpt-4o-mini.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 681 papers

Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach
Apr 25, 2026Abid Talukder, Maruf Ahmed Mridul, Oshani SeneviratneMulti-Agent LLM Systems LLM Agent Orchestration
Rensselaer Polytechnic Institute, Troy, New York, United States
Agent-OM: Leveraging LLM Agents for Ontology Matching
Dec 1, 2023Zhangcheng Qiang, Weiqing Wang, Kerry TaylorSemantic Mapping
Australian National University Canberra, ACT, Australia · Monash University Melbourne, VIC, Australia
Automatic Ontology Construction Using LLMs as an External Layer of Memory, Verification, and Planning for Hybrid Intelligent Systems
Apr 22, 2026Pavel Salovskii, Iuliia GorshkovaKG Construction Memory-Augmented Language Models
Partenit.io, San Francisco, CA, USA · Partenit.io San Francisco, CA, USA

Apr 25, 2026J/K move · Enter open · S save

Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach

Abid Talukder, Maruf Ahmed Mridul, Oshani Seneviratne

Rensselaer Polytechnic Institute, Troy, New York, United States

Multi-Agent LLM Systems LLM Agent Orchestration Ontology Learning

Open paper AskPDF arXiv

Automatically generating formal ontologies from unstructured natural language remains a central challenge in knowledge engineering. While large language models (LLMs) show promise, it remains unclear which architectural design choices drive generation quality and why current approaches fail. We present a controlled experimental study using domain-specific insurance contracts to investigate these questions. We first establish a single-agent LLM baseline, identifying key failure modes such as poor Ontology Design Pattern compliance, structural redundancy, and ineffective iterative repair. We then introduce a multi-agent architecture that decomposes ontology construction into four artifact-driven roles: Domain Expert, Manager, Coder, and Quality Assurer. We evaluate performance across architectural quality (via a panel of heterogeneous LLM judges) and functional usability (via competency question driven SPARQL evaluation with complementary retrieval augmented generation based assessment). Results show that the multi-agent approach significantly improves structural quality and modestly enhances queryability, with gains driven primarily by front-loaded planning. These findings highlight planning-first, artifact-driven generation as a promising and more auditable path toward scalable automated ontology engineering.

Abstract

LLM Agents

Explore similar work

Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach

Agent-OM: Leveraging LLM Agents for Ontology Matching

Automatic Ontology Construction Using LLMs as an External Layer of Memory, Verification, and Planning for Hybrid Intelligent Systems