Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs

Hussein Abdallah; Ibrahim Abdelaziz; Panos Kalnis; Essam Mansour

cs.CLApr 15, 2026

Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs

Authors: Hussein Abdallah, Ibrahim Abdelaziz, Panos Kalnis, Essam Mansour

Organizations: Concordia University · IBM · KAUST

Abstract

Open-world Question Answering (OW-QA) over knowledge graphs (KGs) aims to answer questions over incomplete or evolving KGs. Traditional KGQA assumes a closed world where answers must exist in the KG, limiting real-world applicability. In contrast, open-world QA requires inferring missing knowledge based on graph structure and context. Large language models (LLMs) excel at language understanding but lack structured reasoning. Graph neural networks (GNNs) model graph topology but struggle with semantic interpretation. Existing systems integrate LLMs with GNNs or graph retrievers. Some support open-world QA but rely on structural embeddings without semantic grounding. Most assume observed paths or complete graphs, making them unreliable under missing links or multi-hop reasoning. We present GLOW, a hybrid system that combines a pre-trained GNN and an LLM for open-world KGQA. The GNN predicts top-k candidate answers from the graph structure. These, along with relevant KG facts, are serialized into a structured prompt (e.g., triples and candidates) to guide the LLM's reasoning. This enables joint reasoning over symbolic and semantic signals, without relying on retrieval or fine-tuning. To evaluate generalization, we introduce GLOW-BENCH, a 1,000-question benchmark over incomplete KGs across diverse domains. GLOW outperforms existing LLM-GNN systems on standard benchmarks and GLOW-BENCH, achieving up to 53.3% and an average 38% improvement. GitHub code and data are available.

Figures & tables

Method	Existing Datasets	GLOW-Bench (ours)
Method	1-Hop	1-Hop	2-Hop
LLM ${}_{\text{Only}}$	25.7	30.0	18.9
GCR	7.8	34.1	15.3
GoG	44.6	48.3	29.5
AskGNN	53.7	79.4	34.3
GLOW-GN	71.7	83.3	42.4

Table 1: Average accuracy (%) on OW-QA benchmarks by reasoning depth. GLOW-GN leads on both 1- and 2-hop questions, while GCR drops sharply under OWA. All use Qwen3-8B; scores for existing datasets are averaged over arxiv2023, ogbn-arxiv, and ogbn-products.

Figure 1: Overview of GLOW’s four stages: question understanding, retrieval, augmentation, and generation. GLOW builds GNN-guided prompts using KG context and/or GNN predictions, with three variants: (a) KG facts (GLOW-G), (b) GNN predictions (GLOW-N), and (c) both (GLOW-GN).

KG	KD	#OW-QT	Target ET(s)	RH	#Class	MCC	Label Types
YAGO4	G	10	Person & Creative Work	1–2	3–102	2–32+	Nationality, Publisher, Occupation
BioKG	DS	6	Drug & Protein	1–2	2–29	2–32	Kingdom, Class, SPECIES, R.Keyword
LinkedMDB	E	5	Film	1–2	7–39	4–32+	Language, Producer, Genre
CrunchBase	DS	4	Investor	1	6–14	4–16	Country, InvestRegion, Company

Table 2: Summary of Open-World Question Template (OW-QT) across four KG-based features: (1) Knowledge Domain (KD): Domain-Specific (DS), Entertainment (E), or Generic (G); (2) Target Entity Type (ET); (3) Reasoning Hops (RH); and (4) Multiple Choice Count (MCC).

	LLM-Model	Dataset	RH	LLM-Only	AskGNN	GCR	GoG	GLOW-G	GLOW-N	GLOW-GN
Open-Weight LLMs	Qwen3-8B Yang et al. (2025)	Arxiv2023	1	31	62	4	53	64	66	81
		ogbn-arxiv	1	12	70	14	57	71	60	77
		ogbn-product	1	34	29	5	44	45	51	57
		GLOW-Bench	1	30	79	34	48	60	80	83
		GLOW-Bench	2	19	34	15	29	44	37	42
	DeepSeek-R1- Distill-Qwen-7B DeepSeek-AI (2025)	Arxiv2023	1	24	48	7	28	55	59	67

Table 3: Exact Match Accuracy (%) of GLOW vs. baselines across four OWA-QA datasets. GLOW-GN consistently outperforms AskGNN, GCR, GoG, and LLM-only setups across all LLMs and datasets, showing strong generalization and effective use of both textual and structural signals, even with smaller LLMs.

Figure 2: Average Exact and Hierarchical Match Accuracy (%) across datasets using different methods and LLMs. Dark bars show Exact Match; lighter segments show added Hierarchical Match gains. GCR and AskGNN are inapplicable to commercial LLMs.

Figure 3: Effect of GLOW-Bench characteristics on the GLOW answer accuracy(%) with Qwen3-8B. The effects are grounded by A) Knowledge Domain (KD), B) Knowledge Graph (KG) , and C) Multiple Choice Count (MCC).

Figure 4: The GNN error propagation using GraphSAINT(GS) and ShaDowGNN(SH) models on questions with accuracy score

<

50%. GLOW boosts accuracy by up to 12% over weak GNNs using textual cues. AskGNN closely follows GNN performance.

Figure 5: Impact of GNN top-K answer count on GLOW-GN’s average accuracy (%) across different LLMs. GLOW-GN achieves the best performance at top-K=3, while higher values tend to mislead the LLMs.

	Training Time (H)	Tokens Count (K)	Answer Time (Sec)
AskGNN	23.4	0.79	11.2
GCR	6	0.84	13.4
GoG	N/A	1.5	15.7
GLOW-G	N/A	0.78	12.6
GLOW-N	1.7	0.43	8.2
GLOW-GN	1.7	0.65	11.5

Table 4: The average training time in Hours, Answer tokens count per question in (K-Tokens) and answer time per question in Seconds using Qwen3-8B LLM.

Figure 6: Human evaluation results versus Exact-match and Hierarchical-match. Hierarchical matching closely aligns with human evaluation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

OW-QT	KG	KD	Target ET	RH	#Class	MCC	Label Type	Reasoning Path
1	BioKG	DS	Drug	1	2	2–4	Kingdom	Kingdom
2		DS	Drug	1	13	8–16	Superclass	Superclass
3		DS	Drug	2	29	16–32	Class	RelatedPubMed $\rightarrow$ Class
4		DS	Protein	1	18	16–32	SPECIES	SPECIES
5		DS	Protein	2	15	8–16	R.Keyword	RelatedPubMed $\rightarrow$ R.Keyword
6		DS	Protein	1	4	2–4	Family	Family

Appendix

Table 5: Our GLOW-Bench benchmark characterizes each Open-World Question Template (OW-QT) across four KG-based features: (1) Knowledge Domain (KD): Domain-Specific (DS), Entertainment (E), or Generic (G); (2) Target Entity Type (ET); (3) Reasoning Hops (RH); and (4) Multiple Choice Count (MCC).

LLM-Model	Dataset	RH	AskGNN	GraphSAINT	LLM-Only	GCR	GLOW-L	GLOW-G	GLOW-N	GLOW-GN
	Arxiv2023	1	62	41	31	4	59	64	66	81
	ogbn-arxiv	1	70	59	12	14	63	71	60	77
	ogbn-product	1	29	45	34	5	41	44	51	57
	drug-superclass	1	88	84	29	3	50	77	85	92
	drug-kingdom	1	98	96	88	94	91	98	97	96
	protein-SPECIES	1	75	89	8	16	19	71	91	92

Appendix

Table 6: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use Qwen3-8B as the underlying LLM. Best results are marked in bold .

LLM-Model	Dataset	RH	AskGNN	GraphSAINT	LLM-Only	GLOW-L	GLOW-G	GLOW-N	GLOW-GN
	Arxiv2023	1	N/A	41	35	58	59	48	62
	ogbn-arxiv	1	N/A	41	35	58	59	48	67
	ogbn-product	1	N/A	45	N/A	N/A	N/A	N/A	N/A
	drug-superclass	1	N/A	84	8	47	57	73	87
	drug-kingdom	1	N/A	96	31	100	100	98	99
	protein-SPECIES	1	N/A	89	6	10	77	92	93

Appendix

Table 7: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use GPT-4o-Mini OpenAI et al. (2024) as the underlying LLM. Best results are marked in bold .

LLM-Model	Dataset	RH	AskGNN	GraphSAINT	LLM-Only	GLOW-L	GLOW-G	GLOW-N	GLOW-GN
	Arxiv2023	1	N/A	41	17	50	57	61	73
	ogbn-arxiv	1	N/A	59	58	60	62	63	65
	ogbn-product	1	N/A	45	N/A	N/A	N/A	N/A	N/A
	drug-superclass	1	N/A	84	3	32	48	73	86
	drug-kingdom	1	N/A	96	9	88	100	98	96
	protein-SPECIES	1	N/A	89	2	8	79	92	93

Appendix

Table 8: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use DeepSeek-V3 DeepSeek-AI (2024) as the underlying LLM. Best results are marked in bold .

LLM-Model	Dataset	RH	AskGNN	GraphSAINT	LLM-Only	GCR	GLOW-L	GLOW-G	GLOW-N	GLOW-GN
	Arxiv2023	1	48	41	24	7	44	55	59	67
	ogbn-arxiv	1	50	59	13	15	30	30	61	63
	ogbn-product	1	19	45	27	12	36	38	43	51
	drug-superclass	1	53	84	20	5	20	21	69	68
	drug-kingdom	1	98	96	60	90	90	100	98	100
	protein-SPECIES	1	71	89	1	12	4	62	88	90

Appendix

Table 9: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use DeepSeek-R1-Distill-Qwen:7B DeepSeek-AI (2025) as the underlying LLM. Best results are marked in bold .

LLM-Model	Dataset	RH	AskGNN	GraphSAINT	LLM-Only	GCR	GLOW-G	GLOW-N	GLOW-GN
	Arxiv2	1	63	41	14	14	32	63	77
	ogbn-arxiv	1	66	59	5	21	31	55	73
	ogbn-product	1	30	45	22	13	45	52	54
	drug-superclass	1	86	84	0	9	22	73	71
	drug-kingdom	1	97	96	8	98	95	98	99
	protein-SPECIES	1	75	89	2	18	45	93	90

Appendix

Table 10: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use IBM Granite-3.3-8B-instruct Saon et al. (2025) as the underlying LLM. Best results are marked in bold .

Explore similar work

Explore-over-Graph: Hybrid Embedding-LLM Reasoning for Knowledge Graph Question Answering under Incompleteness
Sep 30, 2026Ola El Khatib, Djellel DifallahKnowledge Graph Question Answering Multi-Hop QA
New York University Abu Dhabi, UAE
Foresight-over-Graph: Reasoning Beyond Local Horizons for Knowledge Base Question Answering
Oct 6, 2026Yang Hong, Yajun Yang, Xin Wang +2Multi-Hop QA LLM Reasoning with Graphs
Tianjin University, Tianjin, China · Beijing Jiaotong University, Beijing, China
Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting
Apr 14, 2026Shuai Wang, Xixi Wang, Yinan YuKnowledge Graph Question Answering Multi-Hop QA
Chalmers University of Technology and University of Gothenburg, Sweden · Technical University of Denmark, Kgs. Lyngby, Denmark

Sep 30, 2026J/K move · Enter open · S save

Explore-over-Graph: Hybrid Embedding-LLM Reasoning for Knowledge Graph Question Answering under Incompleteness

Ola El Khatib, Djellel Difallah

New York University Abu Dhabi, UAE

Knowledge Graph Question Answering Multi-Hop QA LLM Reasoning with Graphs

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly combined with knowledge graphs (KGs) to ground reasoning in structured evidence. However, most LLM-based KGQA methods rely on traversing existing graph edges and become unreliable when reasoning paths are broken by missing facts. Alternatives that ask LLMs to generate missing knowledge risk introducing hallucinated evidence. We introduce XoG (eXplore-over-Graph), a framework for multi-hop question answering over incomplete KGs that recovers missing reasoning paths from learned graph structure rather than LLM parametric knowledge. XoG combines type-level entity-relation statistics to identify candidate relations with KG embeddings to retrieve plausible missing entities, using the LLM as a semantic selector and reasoner. These mechanisms are integrated into an iterative planning-exploration-reasoning process. Experiments on WebQSP, CWQ, and the Wikidata-based BRINK benchmark show that XoG remains competitive on complete KGs and consistently outperforms comparable methods without task-specific KGQA training under KG incompleteness. These gains persist across multiple LLM backbones, indicating that stronger LLMs alone do not resolve missing graph evidence. XoG also reduces LLM token consumption by up to 33% compared with a closely related planning-based approach.

Abstract

Explore similar work

Explore-over-Graph: Hybrid Embedding-LLM Reasoning for Knowledge Graph Question Answering under Incompleteness

Foresight-over-Graph: Reasoning Beyond Local Horizons for Knowledge Base Question Answering

Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting