IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineering

Elliott Watkiss-Leek; Reham Alharbi; Harry Rostron; Andrew Ng; Ewan Johnson; Andrew Mitchell; Terry R. Payne; Valentina Tamma

cs.AIApr 1, 2026

IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineering

Authors: Elliott Watkiss-Leek, Reham Alharbi, Harry Rostron, Andrew Ng, Ewan Johnson, Andrew Mitchell, Terry R. Payne, Valentina Tamma, +1 more

Organizations: School of Computer Science and Informatics, University of Liverpool, Brownlow Hill, Liverpool, L69 7ZX, UK · College of Computer Science and Engineering, Taibah University, Saudi Arabia · Unilever Plc., Materials Innovation Factory, University of Liverpool, 51 Oxford Street, Liverpool, L7 3NY, UK

Abstract

Competency question (CQ) elicitation represents a critical but resource-intensive bottleneck in ontology engineering. This foundational phase is often hampered by the communication gap between domain experts, who possess the necessary knowledge, and ontology engineers, who formalise it. This paper introduces IDEA2, a novel, semi-automated workflow that integrates Large Language Models (LLMs) within a collaborative, expert-in-the-loop process to address this challenge. The methodology is characterised by a core iterative loop: an initial LLM-based extraction of CQs from requirement documents, a co-creational review and feedback phase by domain experts on an accessible collaborative platform, and an iterative, feedback-driven reformulation of rejected CQs by an LLM until consensus is achieved. To ensure transparency and reproducibility, the entire lifecycle of each CQ is tracked using a provenance model that captures the full lineage of edits, anonymised feedback, and generation parameters. The workflow was validated in 2 real-world scenarios (scientific data, cultural heritage), demonstrating that IDEA2 can accelerate the requirements engineering process, improve the acceptance and relevance of the resulting CQs, and exhibit high usability and effectiveness among domain experts. We release all code and experiments at https://github.com/KE-UniLiv/IDEA2

Figures & tables

Approach	CQ Generation	Expert Validation	Iterative Reformulation
IDEA [ 14 ]	$\times$	$\checkmark$	$\times$
Retrofit-CQ [ 4 ]	$\checkmark$	$\times$	$\times$
RevOnt [ 12 ]	$\checkmark$	$\times$	$\times$
OntoChat [ 45 ]	$\checkmark$	$\checkmark$	$\times$
IDEA2	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison of approaches for competency question elicitation.

Figure 1: Overview of the IDEA2 architecture: (1) CQ Extraction generates candidate requirements from heterogeneous sources; (2) these are published to a collaborative dashboard for Expert Validation ; (3) feedback drives an Iterative Reformulation loop where rejected CQs are refined by the LLM based on the feedback; and (4) the process terminates upon meeting defined stopping criteria, producing a CQ export.

Figure 2: Screenshot of the IDEA2 collaborative dashboard on Notion. The interface shows the CQ Pool (left) and the detailed validation view (right) for a specific requirement (ID 109). In this example, a split vote (Score 0) and qualitative comments from domain experts highlight a domain-specific ambiguity, providing the necessary context to trigger and guide the reformulation loop.

Metric	Scenario 1: AnIML	Scenario 2: CH
Focus	Full Workflow	CQ Reformulation
Source Type	XML Schema	Personas, User Story
No. of Experts	4	3
Initial CQs	103 (IDEA2-generated)	23 (Manual+LLM CQs)
Iterations	3	2
Final Acceptance Rate	92.7% (102/110)	73.3% (22/30)

Table 2: Comparative summary of key metrics from validation scenarios. Note: Final Acceptance Rate is calculated as the total number of accepted requirements divided by the total number of validation tasks (initial + reformulations).

Scenario	Iteration	# CQs	Avg. Time per CQ	Acceptance Rate
AnIML	It. 1 (Initial)	103	0.58 min	95.1%
	It. 2 (Reformulation)	5	2.93 min	60.0%
	It. 3 (Reformulation)	2	1.75 min	50.0%
CH	It. 1 (Reformulation)	23	2.26 min	69.6%
CH	It. 2 (Reformulation)	7	3.19 min	85.7%

Table 3: Detailed breakdown of validation metrics per iteration, showing the number of CQs reviewed in each cycle. The decreased volume in later iterations corresponds to the subset of requirements requiring reformulation.

Metric	AnIML (N=4)	CH (N=3))
Observed agreement (%)	89.8	90.0
Krippendorff’s $\alpha$ [ 26 ]	0.177	0.843
Fleiss’ $\kappa$ [ 20 ]	0.176	0.841
Gwet’s AC1 [ 23 ]	0.884	0.885
PABAK [ 9 ]	0.797	0.867
Pairwise Cohen’s $\kappa$ [ 13 ] range	0.017 to 0.422	0.772 to 1.000

Table 4: Comparison of inter-annotator agreement metrics across the two studies.

Panel of Figure 3 — Figure 3: Usage metrics results derived from the expert evaluation. The Likert scale charts illustrate the domain experts’ agreement regarding the tool’s consistency, collaborative capabilities, expressiveness, and intuitiveness across (a) the AnIML scenario ( $N=4$ ) and (b) the Cultural Heritage scenario ( $N=3$ ).

Explore similar work

Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models
Apr 17, 2026Reham Alharbi, Valentina Tamma, Terry R. Payne +1LLM Evaluation
Taibah University, Madinah, Saudi Arabia · University of Liverpool, Liverpool, UK
CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions
Sep 22, 2026Jiayi Li, Ziyuan Wang, Daniel Garijo +1LLM Evaluation Ontology Learning
Ontology Engineering Group, Universidad Politécnica de Madrid
OntoExtend: A Framework for Requirement-driven and Scalable Ontology Extension with LLMs
Jul 20, 2026Anna Sofia Lippolis, Mohammad Javad Saeedizade, Stefan Schmid +5Ontology Learning
Linköping University, Sweden · University of Bologna, Italy · ISTC-CNR, Italy +1

Apr 17, 2026J/K move · Enter open · S save

Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

Reham Alharbi, Valentina Tamma, Terry R. Payne, Jacopo de Berardinis

Taibah University, Madinah, Saudi Arabia · University of Liverpool, Liverpool, UK

LLM Evaluation

Open paper AskPDF arXiv

Competency Questions (CQs) are a cornerstone of requirement elicitation in ontology engineering. CQs represent requirements as a set of natural language questions that an ontology should satisfy; they are traditionally modelled by ontology engineers together with domain experts as part of a human-centred, manual elicitation process. The use of Generative AI automates CQ creation at scale, therefore democratising the process of generation, widening stakeholder engagement, and ultimately broadening access to ontology engineering. However, given the large and heterogeneous landscape of LLMs, varying in dimensions such as parameter scale, task and domain specialisation, and accessibility, it is crucial to characterise and understand the intrinsic, observable properties of the CQs they produce (e.g., readability, structural complexity) through a systematic, cross-domain analysis. This paper introduces a set of quantitative measures for the systematic comparison of CQs across multiple dimensions. Using CQs generated from well defined use cases and scenarios, we identify their salient properties, including readability, relevance with respect to the input text and structural complexity of the generated questions. We conduct our experiments over a set of use cases and requirements using a range of LLMs, including both open (KimiK2-1T, LLama3.1-8B, LLama3.2-3B) and closed models (Gemini 2.5 Pro, GPT 4.1). Our analysis demonstrates that LLM performance reflects distinct generation profiles shaped by the use case.

Abstract

Explore similar work

Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions

OntoExtend: A Framework for Requirement-driven and Scalable Ontology Extension with LLMs