Open-world Question Answering (OW-QA) over knowledge graphs (KGs) aims to answer questions over incomplete or evolving KGs. Traditional KGQA assumes a closed world where answers must exist in the KG, limiting real-world applicability. In contrast, open-world QA requires inferring missing knowledge based on graph structure and context. Large language models (LLMs) excel at language understanding but lack structured reasoning. Graph neural networks (GNNs) model graph topology but struggle with semantic interpretation. Existing systems integrate LLMs with GNNs or graph retrievers. Some support open-world QA but rely on structural embeddings without semantic grounding. Most assume observed paths or complete graphs, making them unreliable under missing links or multi-hop reasoning. We present GLOW, a hybrid system that combines a pre-trained GNN and an LLM for open-world KGQA. The GNN predicts top-k candidate answers from the graph structure. These, along with relevant KG facts, are serialized into a structured prompt (e.g., triples and candidates) to guide the LLM's reasoning. This enables joint reasoning over symbolic and semantic signals, without relying on retrieval or fine-tuning. To evaluate generalization, we introduce GLOW-BENCH, a 1,000-question benchmark over incomplete KGs across diverse domains. GLOW outperforms existing LLM-GNN systems on standard benchmarks and GLOW-BENCH, achieving up to 53.3% and an average 38% improvement. GitHub code and data are available.
Figures & tables
Method
Existing Datasets
GLOW-Bench (ours)
1-Hop
1-Hop
2-Hop
LLM Only
25.7
30.0
18.9
GCR
7.8
34.1
15.3
GoG
44.6
48.3
29.5
AskGNN
53.7
79.4
34.3
GLOW-GN
71.7
83.3
42.4
Table 1: Average accuracy (%) on OW-QA benchmarks by reasoning depth. GLOW-GN leads on both 1- and 2-hop questions, while GCR drops sharply under OWA. All use Qwen3-8B; scores for existing datasets are averaged over arxiv2023, ogbn-arxiv, and ogbn-products.
Figure 1: Overview of GLOW’s four stages: question understanding, retrieval, augmentation, and generation. GLOW builds GNN-guided prompts using KG context and/or GNN predictions, with three variants: (a) KG facts (GLOW-G), (b) GNN predictions (GLOW-N), and (c) both (GLOW-GN).
KG
KD
#OW-QT
Target ET(s)
RH
#Class
MCC
Label Types
YAGO4
G
10
Person & Creative Work
1–2
3–102
2–32+
Nationality, Publisher, Occupation
BioKG
DS
6
Drug & Protein
1–2
2–29
2–32
Kingdom, Class, SPECIES, R.Keyword
LinkedMDB
E
5
Film
1–2
7–39
4–32+
Language, Producer, Genre
CrunchBase
DS
4
Investor
1
6–14
4–16
Country, InvestRegion, Company
Table 2: Summary of Open-World Question Template (OW-QT) across four KG-based features: (1) Knowledge Domain (KD): Domain-Specific (DS), Entertainment (E), or Generic (G); (2) Target Entity Type (ET); (3) Reasoning Hops (RH); and (4) Multiple Choice Count (MCC).
LLM-Model
Dataset
RH
LLM-Only
AskGNN
GCR
GoG
GLOW-G
GLOW-N
GLOW-GN
Open-Weight LLMs
Qwen3-8B Yang et al. (2025)
Arxiv2023
1
31
62
4
53
64
66
81
ogbn-arxiv
1
12
70
14
57
71
60
77
ogbn-product
1
34
29
5
44
45
51
57
GLOW-Bench
1
30
79
34
48
60
80
83
GLOW-Bench
2
19
34
15
29
44
37
42
DeepSeek-R1- Distill-Qwen-7B DeepSeek-AI (2025)
Arxiv2023
1
24
48
7
28
55
59
67
Table 3: Exact Match Accuracy (%) of GLOW vs. baselines across four OWA-QA datasets. GLOW-GN consistently outperforms AskGNN, GCR, GoG, and LLM-only setups across all LLMs and datasets, showing strong generalization and effective use of both textual and structural signals, even with smaller LLMs.
Figure 2: Average Exact and Hierarchical Match Accuracy (%) across datasets using different methods and LLMs. Dark bars show Exact Match; lighter segments show added Hierarchical Match gains. GCR and AskGNN are inapplicable to commercial LLMs.
Figure 3: Effect of GLOW-Bench characteristics on the GLOW answer accuracy(%) with Qwen3-8B. The effects are grounded by A) Knowledge Domain (KD), B) Knowledge Graph (KG) , and C) Multiple Choice Count (MCC).
Figure 4: The GNN error propagation using GraphSAINT(GS) and ShaDowGNN(SH) models on questions with accuracy score < 50%. GLOW boosts accuracy by up to 12% over weak GNNs using textual cues. AskGNN closely follows GNN performance.
Figure 5: Impact of GNN top-K answer count on GLOW-GN’s average accuracy (%) across different LLMs. GLOW-GN achieves the best performance at top-K=3, while higher values tend to mislead the LLMs.
Training Time (H)
Tokens Count (K)
Answer Time (Sec)
AskGNN
23.4
0.79
11.2
GCR
6
0.84
13.4
GoG
N/A
1.5
15.7
GLOW-G
N/A
0.78
12.6
GLOW-N
1.7
0.43
8.2
GLOW-GN
1.7
0.65
11.5
Table 4: The average training time in Hours, Answer tokens count per question in (K-Tokens) and answer time per question in Seconds using Qwen3-8B LLM.
Figure 6: Human evaluation results versus Exact-match and Hierarchical-match. Hierarchical matching closely aligns with human evaluation.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
OW-QT
KG
KD
Target ET
RH
#Class
MCC
Label Type
Reasoning Path
1
BioKG
DS
Drug
1
2
2–4
Kingdom
Kingdom
2
DS
Drug
1
13
8–16
Superclass
Superclass
3
DS
Drug
2
29
16–32
Class
RelatedPubMed → Class
4
DS
Protein
1
18
16–32
SPECIES
SPECIES
5
DS
Protein
2
15
8–16
R.Keyword
RelatedPubMed → R.Keyword
6
DS
Protein
1
4
2–4
Family
Family
Appendix
Table 5: Our GLOW-Bench benchmark characterizes each Open-World Question Template (OW-QT) across four KG-based features: (1) Knowledge Domain (KD): Domain-Specific (DS), Entertainment (E), or Generic (G); (2) Target Entity Type (ET); (3) Reasoning Hops (RH); and (4) Multiple Choice Count (MCC).
LLM-Model
Dataset
RH
AskGNN
GraphSAINT
LLM-Only
GCR
GLOW-L
GLOW-G
GLOW-N
GLOW-GN
Arxiv2023
1
62
41
31
4
59
64
66
81
ogbn-arxiv
1
70
59
12
14
63
71
60
77
ogbn-product
1
29
45
34
5
41
44
51
57
drug-superclass
1
88
84
29
3
50
77
85
92
drug-kingdom
1
98
96
88
94
91
98
97
96
protein-SPECIES
1
75
89
8
16
19
71
91
92
Appendix
Table 6: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use Qwen3-8B as the underlying LLM. Best results are marked in bold .
LLM-Model
Dataset
RH
AskGNN
GraphSAINT
LLM-Only
GLOW-L
GLOW-G
GLOW-N
GLOW-GN
Arxiv2023
1
N/A
41
35
58
59
48
62
ogbn-arxiv
1
N/A
41
35
58
59
48
67
ogbn-product
1
N/A
45
N/A
N/A
N/A
N/A
N/A
drug-superclass
1
N/A
84
8
47
57
73
87
drug-kingdom
1
N/A
96
31
100
100
98
99
protein-SPECIES
1
N/A
89
6
10
77
92
93
Appendix
Table 7: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use GPT-4o-Mini OpenAI et al. (2024) as the underlying LLM. Best results are marked in bold .
LLM-Model
Dataset
RH
AskGNN
GraphSAINT
LLM-Only
GLOW-L
GLOW-G
GLOW-N
GLOW-GN
Arxiv2023
1
N/A
41
17
50
57
61
73
ogbn-arxiv
1
N/A
59
58
60
62
63
65
ogbn-product
1
N/A
45
N/A
N/A
N/A
N/A
N/A
drug-superclass
1
N/A
84
3
32
48
73
86
drug-kingdom
1
N/A
96
9
88
100
98
96
protein-SPECIES
1
N/A
89
2
8
79
92
93
Appendix
Table 8: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use DeepSeek-V3 DeepSeek-AI (2024) as the underlying LLM. Best results are marked in bold .
LLM-Model
Dataset
RH
AskGNN
GraphSAINT
LLM-Only
GCR
GLOW-L
GLOW-G
GLOW-N
GLOW-GN
Arxiv2023
1
48
41
24
7
44
55
59
67
ogbn-arxiv
1
50
59
13
15
30
30
61
63
ogbn-product
1
19
45
27
12
36
38
43
51
drug-superclass
1
53
84
20
5
20
21
69
68
drug-kingdom
1
98
96
60
90
90
100
98
100
protein-SPECIES
1
71
89
1
12
4
62
88
90
Appendix
Table 9: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use DeepSeek-R1-Distill-Qwen:7B DeepSeek-AI (2025) as the underlying LLM. Best results are marked in bold .
LLM-Model
Dataset
RH
AskGNN
GraphSAINT
LLM-Only
GCR
GLOW-G
GLOW-N
GLOW-GN
Arxiv2
1
63
41
14
14
32
63
77
ogbn-arxiv
1
66
59
5
21
31
55
73
ogbn-product
1
30
45
22
13
45
52
54
drug-superclass
1
86
84
0
9
22
73
71
drug-kingdom
1
97
96
8
98
95
98
99
protein-SPECIES
1
75
89
2
18
45
93
90
Appendix
Table 10: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from Hu et al. (2024) and our developed benchmark GLOW-Bench. All results below use IBM Granite-3.3-8B-instruct Saon et al. (2025) as the underlying LLM. Best results are marked in bold .
Large language models (LLMs) are increasingly combined with knowledge graphs (KGs) to ground reasoning in structured evidence. However, most LLM-based KGQA methods rely on traversing existing graph edges and become unreliable when reasoning paths are broken by missing facts. Alternatives that ask LLMs to generate missing knowledge risk introducing hallucinated evidence. We introduce XoG (eXplore-over-Graph), a framework for multi-hop question answering over incomplete KGs that recovers missing reasoning paths from learned graph structure rather than LLM parametric knowledge. XoG combines type-level entity-relation statistics to identify candidate relations with KG embeddings to retrieve plausible missing entities, using the LLM as a semantic selector and reasoner. These mechanisms are integrated into an iterative planning-exploration-reasoning process. Experiments on WebQSP, CWQ, and the Wikidata-based BRINK benchmark show that XoG remains competitive on complete KGs and consistently outperforms comparable methods without task-specific KGQA training under KG incompleteness. These gains persist across multiple LLM backbones, indicating that stronger LLMs alone do not resolve missing graph evidence. XoG also reduces LLM token consumption by up to 33% compared with a closely related planning-based approach.
Large language models (LLMs) have demonstrated strong capabilities in question answering, yet they still frequently suffer from hallucinations on knowledge-intensive tasks. Knowledge graphs (KGs) provide LLMs with structured, interpretable, and updatable factual grounding, making them a promising external knowledge source for reliable reasoning. However, existing LLM-guided graph reasoning methods typically rely on hop-wise greedy or beam-style pruning during evidence retrieval. Such local decision processes are inherently myopic: evidence that appears weak near the source may become crucial only after deeper graph context is explored, causing answer-critical branches to be discarded prematurely and making the reasoning chain difficult to recover. To address this limitation, we propose Foresight-over-Graph (FoG), a foresight-aware evidence retrieval framework for knowledge base question answering (KBQA). FoG iteratively constructs a question-relevant evidence subgraph and uses far-to-near feedback to guide path exploration, and maintains a compact memory subgraph to support continued exploration. Extensive experiments on widely used KBQA benchmarks demonstrate that FoG achieves state-of-the-art performance, with a particularly large improvement of 16.58% in Hit on CWQ, while also reducing LLM calls and token usage. Our code is available at https://github.com/yhong7/FoG .
Yang Hong, Yajun Yang, Xin Wang +2
Tianjin University, Tianjin, China · Beijing Jiaotong University, Beijing, China
Large Language Models (LLMs) have shown remarkable capabilities across various tasks but remain prone to hallucinations in knowledge-intensive scenarios. Knowledge Base Question Answering (KBQA) mitigates this by grounding generation in Knowledge Graphs (KGs). However, most multi-hop KBQA methods rely on explicit edge traversal, making them fragile to KG incompleteness. In this paper, we proposed a novel graph-based soft prompting framework that shifts the reasoning paradigm from node-level path traversal to subgraph-level reasoning. Specifically, we employ a Graph Neural Network (GNN) to encode extracted structural subgraphs into soft prompts, enabling LLM to reason over richer structural context and identify relevant entities beyond immediate graph neighbors, thereby reducing sensitivity to missing edges. Furthermore, we introduce a two-stage paradigm that reduces computational cost while preserving good performance: a lightweight LLM first leverages the soft prompts to identify question-relevant entities and relations, followed by a more powerful LLM for evidence-aware answer generation. Experiments on four multi-hop KBQA benchmarks show that our approach achieves state-of-the-art performance on three of them, demonstrating its effectiveness. Code is available at the repository: https://github.com/Wangshuaiia/GraSP.
Shuai Wang, Xixi Wang, Yinan Yu
Chalmers University of Technology and University of Gothenburg, Sweden · Technical University of Denmark, Kgs. Lyngby, Denmark