From Fluent Text to Ontology-Compatible Definitions: A Quality-Control Agentic Workflow
Authors: Kun Feng, Chenchen Chu, Yan Liu, Haojie Sun, Jijun Tang, Chen Tian, Shuolin Xiao, Jianping He, Zhao Li
Organizations: Nanjing University, Nanjing, 210023, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, 518107, China · Materials Artificial Intelligence Center, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China · University of Chinese Academy of Sciences, 100049, Beijing, China · Ambulatory Operations, Baylor College of Medicine, Houston, Texas, 77030, USA · Ralph O’Connor Sustainable Energy Institute, Johns Hopkins University, Baltimore, Maryland, 21218, USA · Beijing Foreign Studies University Artificial Intelligence and Human Languages Lab, China
Biomedical ontologies are structured knowledge systems that must evolve continuously to represent newly emerging knowledge, but their manual maintenance remains time-consuming and laborintensive. Textual definitions are especially costly because they must be readable to humans and compatible with controlled entities, relations, and logical constraints. Large language models (LLMs) offer a promising way to assist ontology engineering tasks, including definition generation. However, textual plausibility does not guarantee knowledge-system compatibility. A generated definition may be fluent and close to a reference definition, yet still fail to ground its implied entities or violate ontology-level constraints. Directly inserting such definitions into ontology libraries may therefore introduce entity-level hallucinations and logical inconsistencies. To address this problem, we propose two structured-knowledge incompatibility metrics, Entity Error Rate (EER) and Logical Inconsistency Rate (LIR), to quantify entity grounding and logical consistency for LLM-generated ontology definitions. We further develop an agentic quality-control (QC) workflow that treats definition generation as an iterative process rather than a direct decoding step: it generates definitions, extracts axioms, checks entity existence and logical consistency, and uses the resulting diagnostics to drive repair loops. Comprehensive experiments on Disease Ontology, Uberon Ontology, and Uberon-ConstraintStress demonstrate that EER and LIR complement conventional reference-based metrics by evaluating whether generated content can be grounded and checked inside the target ontology. The workflow effectively decreases EER while maintaining strong conventional text-generation performance. LIR further exposes sparse ontology-level logical failures that are invisible to text similarity metrics, although their repair remains limited.
Figures & tables
Figure 1: Evaluation framework for LLM-generated ontology definitions. Text-generation metrics measure similarity to reference definitions. Knowledge-compatible evaluation, instantiated here for ontologies, also checks entity grounding and logical consistency.
Figure 2: Axiom-based quality-control workflow. The existence loop retries definition generation and axiom extraction for records with unresolved predicates or objects. The logic loop keeps the generated definition fixed and revises only the extracted axioms for records that become unsatisfiable after ROBOT reasoning; it does not trigger definition regeneration, because allowing both definition-level and axiom-level repair in the logic loop could create many additional iterations even with per-loop retry limits.
Figure 3: Definition coverage and remaining definition gaps in DO and Uberon. The 2024 and 2025 segments combine newly added active terms with definitions and previously active terms that first received definitions in the newer release.
Ontology
Test setting
Definition train+val
Axiom train+val
Test size
DO
2024
9,093
4,810
188
DO
2025
9,093
4,810
236
Uberon
2024
9,557
9,557
96
Uberon
2025
9,557
9,557
112
Uberon
Constraint-Stress
9,557
9,557
973
Table 1: Training and evaluation settings used in the main experiments.
Test setting
Model
Loop setting
Token F1 ↑
ROUGE-L ↑
BLEU-2 ↑
BERT-F1 ↑
Cosine ↑
EER ↓
DO 2024
Qwen2.5-14B
Base, no loop
0.283
0.222
0.107
0.608
0.664
0.670
DO 2024
Qwen2.5-14B
Base, existence loop
0.293
0.236
0.118
0.619
0.676
0.378
DO 2024
Qwen2.5-14B
LoRA, no loop
0.588
0.579
0.380
0.731
0.766
0.016
DO 2024
Qwen2.5-14B
LoRA, existence loop
0.588
0.579
0.381
0.731
0.766
0.000
DO 2024
Llama-3.1-8B
Base, no loop
0.310
0.240
0.115
0.593
0.642
0.697
DO 2024
Llama-3.1-8B
Base, existence loop
0.320
0.244
0.123
0.608
0.656
0.457
Table 2: Conventional text-generation scores and Entity Error Rate (EER) for the same model-loop settings shown in Figure 4 . Darker cells indicate better values: higher conventional scores and lower EER. Arrows in column headers indicate the preferred metric direction.
Figure 4: Entity Error Rate (EER) before and after entity-existence repair. Lower is better. The repair loop substantially reduces unresolved-entity failures for base models and for difficult Uberon settings.
Test setting
Model
Loop setting
LIR ↓
Real count
Uberon 2024
Qwen2.5-14B
Base, existence loop
1.05%
1/95
Uberon 2024
Qwen2.5-14B
Base, all loops
1.05%
1/95
Uberon 2024
Qwen2.5-14B
LoRA, existence loop
2.08%
2/96
Uberon 2024
Qwen2.5-14B
LoRA, all loops
2.08%
2/96
Uberon 2024
Llama-3.1-8B
Base, existence loop
0.00%
0/92
Uberon 2024
Llama-3.1-8B
Base, all loops
0.00%
0/92
Table 3: Logical Inconsistency Rate (LIR) after entity-existence repair. The comparison is between existence-loop and all-loop settings; all-loop adds the logical repair loop on top of the existence-loop setting, so both share the same entity-grounded candidate pool. Real count reports the number of inconsistent candidates over the number of entity-grounded candidates. Bold all-loop values indicate a lower LIR than the paired existence-loop setting.
Test set
Model
Knowledge cutoff
Token F1
ROUGE-L
BLEU-2
BERT-F1
Cosine
DO 2024
Qwen2.5-7B-Instruct
not reported; card Jan. 2025
0.280
0.184
0.084
0.587
0.705
DO 2024
Llama-3.1-8B
2023.12
0.325
0.238
0.124
0.603
0.663
DO 2024
Llama-3.1-8B-Instruct
2023.12
0.297
0.180
0.085
0.565
0.696
DO 2024
Qwen2.5-14B-Instruct
not reported; card Jan. 2025
0.281
0.192
0.089
0.597
0.704
DO 2024
gpt-oss-20b
not reported; card Aug. 2025
0.239
0.144
0.063
0.525
0.536
DO 2024
DeepSeek-R1-Distill-Llama-70B
not reported; card Jan. 2025
0.267
0.163
0.077
0.569
0.626
Table 4: Definition-generation model selection on DO release-based test sets. FT-prefixed models are evaluated after fine-tuning; other models are evaluated by direct base-model inference. The knowledge-cutoff column is included as a data-leakage reference; bold marks the best score within each test set.
Test set
Input setting
Input fields
Token F1
ROUGE-L
BLEU-2
BERT-F1
Cosine
DO 2024
Full fields
id, name, synonyms, parents, subsets
0.602
0.586
0.386
0.808
0.786
DO 2024
No ID
name, synonyms, parents, subsets
0.601
0.585
0.381
0.807
0.791
DO 2024
No ID/synonyms
name, parents, subsets
0.596
0.579
0.379
0.803
0.776
DO 2024
No ID/parents
name, synonyms, subsets
0.576
0.555
0.348
0.795
0.756
DO 2024
No ID/subsets
name, synonyms, parents
0.613
0.597
0.396
0.813
0.790
DO 2025
Full fields
id, name, synonyms, parents, subsets
0.597
0.589
0.373
0.808
0.758
Table 5: Input-field ablation for DO definition generation using the fine-tuned Qwen2.5-14B-Instruct model. Full fields include id, name, synonyms, parents, and subsets; each ablation removes the listed field or fields from the input. Bold marks the best score within each test set.
Ontology
Setting
Model
Extractor
n
Parse
Exact
Pair P
Pair R
Pair F1
Pred. Acc.
Obj. Acc.
Hit
DO
2024
Qwen2.5-14B
Base + few-shot
188
1.000
0.074
0.103
0.164
0.127
0.137
0.350
0.170
DO
2024
Qwen2.5-14B
LoRA
188
1.000
0.777
0.630
0.831
0.717
0.628
0.604
0.894
DO
2024
Llama-3.1-8B
Base + few-shot
188
1.000
0.005
0.021
0.075
0.033
0.044
0.156
0.080
DO
2024
Llama-3.1-8B
LoRA
188
1.000
0.777
0.679
0.845
0.753
0.686
0.664
0.894
DO
2025
Qwen2.5-14B
Base + few-shot
236
1.000
0.199
0.186
0.305
0.231
0.202
0.347
0.369
DO
2025
Qwen2.5-14B
LoRA
236
1.000
0.669
0.691
0.761
0.725
0.637
0.620
0.877
Table 6: Axiom-extractor performance on test-set no-loop workflow outputs. Base rows use five-shot prompting, whereas LoRA rows use the fine-tuned axiom extractor. Predicate and object strings are normalized before comparison; bold marks the higher score within each Base/LoRA pair.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Parameter
Value
LoRA/SFT
Base models
Qwen2.5-14B-Instruct; Llama-3.1-8B-Instruct
LoRA/SFT
LoRA rank r
64
LoRA/SFT
LoRA alpha
128
LoRA/SFT
LoRA dropout
0.05
LoRA/SFT
Target modules
q, k, v, o, gate, up, down projections
LoRA/SFT
Rank-stabilized LoRA (RS-LoRA)
enabled
Appendix
Table A.1: Fine-tuning, greedy definition-generation comparison, and random-sampling repair settings used in the experiments. The Low-Rank Adaptation (LoRA) and supervised fine-tuning (SFT) settings are shared by the definition generator and axiom extractor unless otherwise noted.
Test setting
Model
Loop setting
Token F1 ↑
ROUGE-L ↑
BLEU-2 ↑
BERT-F1 ↑
Cosine ↑
EER ↓
DO 2024
Qwen2.5-14B
Base, no loop
0.283
0.222
0.107
0.608
0.664
0.670
DO 2024
Qwen2.5-14B
Base, all loops
0.293
0.236
0.118
0.619
0.676
0.378
DO 2024
Qwen2.5-14B
LoRA, no loop
0.588
0.579
0.380
0.731
0.766
0.016
DO 2024
Qwen2.5-14B
LoRA, all loops
0.588
0.579
0.381
0.731
0.766
0.000
DO 2024
Llama-3.1-8B
Base, no loop
0.310
0.240
0.115
0.593
0.642
0.697
DO 2024
Llama-3.1-8B
Base, all loops
0.320
0.244
0.123
0.608
0.656
0.457
Appendix
Table A.2 : Conventional text-generation scores and Entity Error Rate (EER) comparing direct generation with the full QC workflow. Darker cells indicate better values: higher conventional scores and lower EER. Arrows in column headers indicate the preferred metric direction.
Ontology
Setting
Model
Extractor
n
Parse
Exact
Pair P
Pair R
Pair F1
Pred. Acc.
Obj. Acc.
Hit
Uberon
Uberon eval
Qwen2.5-14B
LoRA
1912
1.000
0.348
0.423
0.490
0.454
0.520
0.394
0.745
Uberon
Uberon eval
Llama-3.1-8B
LoRA
1912
1.000
0.300
0.306
0.487
0.375
0.413
0.302
0.745
DO
DO eval
Qwen2.5-14B
LoRA
962
1.000
0.686
0.518
0.810
0.632
0.536
0.505
0.923
DO
DO eval
Llama-3.1-8B
LoRA
962
1.000
0.682
0.606
0.797
0.689
0.635
0.588
0.915
Appendix
Table A.3 : Axiom-extractor performance on validation splits used during LoRA training. The table evaluates whether the fine-tuned extractor learned to produce parseable and reference-compatible relationship axioms before being used in the workflow.
Automatically generating formal ontologies from unstructured natural language remains a central challenge in knowledge engineering. While large language models (LLMs) show promise, it remains unclear which architectural design choices drive generation quality and why current approaches fail. We present a controlled experimental study using domain-specific insurance contracts to investigate these questions. We first establish a single-agent LLM baseline, identifying key failure modes such as poor Ontology Design Pattern compliance, structural redundancy, and ineffective iterative repair. We then introduce a multi-agent architecture that decomposes ontology construction into four artifact-driven roles: Domain Expert, Manager, Coder, and Quality Assurer. We evaluate performance across architectural quality (via a panel of heterogeneous LLM judges) and functional usability (via competency question driven SPARQL evaluation with complementary retrieval augmented generation based assessment). Results show that the multi-agent approach significantly improves structural quality and modestly enhances queryability, with gains driven primarily by front-loaded planning. These findings highlight planning-first, artifact-driven generation as a promising and more auditable path toward scalable automated ontology engineering.
Abid Talukder, Maruf Ahmed Mridul, Oshani Seneviratne
Rensselaer Polytechnic Institute, Troy, New York, United States
Ontology extension refers to the process of enriching an existing ontology in response to emerging requirements, making it more complete. This task is a resource-intensive and error-prone process. Large Language Models (LLMs) have shown promising performance on generating ontologies from scratch, but current approaches rarely tie ontology extension explicitly to requirements or reusable core models, and offer limited, systematic evaluation of LLM outputs. This paper introduces OntoExtend, a requirements-driven framework for ontology extension with LLMs. It uses retrieval-augmented generation (RAG) over relevant input ontologies and requirements in the form of competency questions to propose grounded extensions. We evaluate OntoExtend on 39 CQs from two use cases: a public EU-project ontology, Onto-DESIDE, and an industrial ontology from Bosch. The generated fragments show few structural issues, satisfy all functional evaluation tests, and are rated by ontology engineers as requiring minor to moderate revision before integration. These results suggest that OntoExtend is useful as a drafting assistant for requirement-driven ontology extension in real world scenarios, while remaining sensitive to CQ specificity and modelling profile.
Anna Sofia Lippolis, Mohammad Javad Saeedizade, Stefan Schmid +5
Linköping University, Sweden · University of Bologna, Italy · ISTC-CNR, Italy +1
Ontology generation from Competency Questions (CQs) is a central yet labor-intensive phase of Ontology Engineering. While large language models (LLMs) offer promising automation capabilities, current evaluations remain fragmented. Task formulations are heterogeneous, gold standards often lack fine-grained CQ provenance, metrics conflate lexical overlap with structural and logical adequacy, and reference ontologies are not always explicitly designed around the evaluation CQs. Here, we address these limitations with CQ4OE, a benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from CQs. For each ontology in the benchmark, we build a CQ-driven gold OWL ontology with explicit provenance linking each CQ to the classes, properties, and axioms required to answer it. From this resource, we define two complementary evaluation tasks. CQ2Term supports term-level evaluation of CQ-specific class and property prediction over 99 CQs, and CQ2Onto supports ontology-level evaluation over 118 CQs, including hierarchy, property modeling, and axiom-level structure. We demonstrate CQ4OE with experiments using nine LLMs under zero-shot, iterative, and multi-agent generation strategies, showing that LLMs recover explicit vocabulary terms more reliably than creating ontologies, particularly in property modeling, hierarchy construction, and axiom generation.
Jiayi Li, Ziyuan Wang, Daniel Garijo +1
Ontology Engineering Group, Universidad Politécnica de Madrid