Knowledge graph question answering (KGQA) is a promising approach for mitigating LLM hallucination by grounding reasoning in structured and verifiable knowledge graphs. Existing approaches fall into two paradigms: retrieval-based methods utilize small specialized models, which are efficient but often produce unreachable paths and miss implicit constraints, while agent-based methods utilize large general models, which achieve stronger structural grounding at substantially higher cost. We propose RouterKGQA, a framework for specialized--general model collaboration, in which a specialized model generates reasoning paths and a general model performs KG-guided repair only when needed, improving performance at minimal cost. We further equip the specialized with constraint-aware answer filtering, which reduces redundant answers. In addition, we design a more efficient general agent workflow, further lowering inference cost. Experimental results show that RouterKGQA outperforms the previous best by 3.57 points in F1 and 0.49 points in Hits@1 on average across benchmarks, while requiring only 1.15 average LLM calls per question. Codes and models are available at https://github.com/Oldcircle/RouterKGQA.
Figures & tables
Figure 1: Overview of the RouterKGQA framework.
Category
Method
WebQSP
CWQ
Hits@1
F1
Hits@1
F1
LLM-only
Llama-2-7B ( Touvron et al., 2023 )
53.87
35.67
22.74
18.11
Llama-3.1-8B ( Team, 2024 )
55.96
31.61
30.81
21.83
GPT-4o-mini
64.86
48.59
40.89
40.89
IR-based
KV-Mem ( Miller et al., 2016 )
46.70
34.50
21.10
15.70
PullNet ( Sun et al., 2019 )
68.10
–
47.20
–
Table 1: Performance comparison on WebQSP and CWQ (%). Results for methods without public code are taken from the original papers; all other results are reproduced in our environment using the same Freebase snapshot. See Appendix C.2 for unified metric definitions and Appendix D for reproduction settings and implementation details.
Category
Method
Hits@1
F1
Avg. # LLM Calls
Avg. # Tokens
Avg. Cost (USD) / 10K Qs
Retrieval-based
GNN-RAG
90.73
73.49
2.00
778.17
0.56
RoG
86.18
70.24
2.00
592.55
0.45
ChatKBQA
86.33
82.13
1.00
208.08
0.46
GCR
91.68
75.27
2.00
725.57
0.89
MemQ
88.74
77.90
1.00
175.62
0.21
Agent-based
ToG
77.20
50.04
10.91
8,426.59
19.35
Table 2: Efficiency and performance comparison on WebQSP (%). RouterKGQA (Full) achieves the best F1 at a cost comparable to retrieval-based methods, while RouterKGQA (Stage 2+3), i.e., the agent-based-only configuration, remains substantially cheaper than representative agent-based baselines. Costs are estimated using Llama-2-7B pricing (0.05/1Minput,0.25/1M output) and Llama-3.1-8B pricing (0.10/1Minput,0.10/1M output) from Artificial Analysis, and official GPT-4o-mini pricing (0.15/1Minput,0.60/1M output) from OpenAI, all accessed 2026-02-14.
Variants
WebQSP
CWQ
Hits@1
F1
Hits@1
F1
RouterKGQA
91.15
86.38
89.66
83.78
Stage-level ablations
w/o Stage 1 (Stage 2+3)
86.07
70.68
70.67
59.30
w/o Stage 2 (Stage 1+3)
89.02
84.68
88.05
82.12
Stopping criterion variants
Table 3: Ablation study of RouterKGQA (%). Constraint ablations are conducted on Stage 1+3. Stage 1: retrieval-based CRP generation; Stage 2: agent-based KG-guided repair; Stage 3: progressive query execution.
Variants
WebQSP
CWQ
Hits@1
F1
Hits@1
F1
GCR (w/o LLM Filter)
92.13
57.00
74.99
49.65
GCR
91.68
75.27
74.64
62.94
GCR (w/ RouterKGQA Filter)
87.13
83.95
80.46
68.05
RouterKGQA
91.15
86.38
89.66
83.78
Table 4: Reference comparison with GCR variants (%). RouterKGQA is included for reference.
Dataset
Metric
ChatKBQA
RouterKGQA
WebQSP
Skeleton Accuracy
82.43
85.78
Exact Match
62.42
69.68
CWQ
Skeleton Accuracy
74.00
75.62
Exact Match
54.23
54.63
Table 5: First-prediction generation accuracy (%) of ChatKBQA (S-expression) and RouterKGQA (CRP), both using Llama-2-7B.
Dimension
Category
N
Full
S1+S3
Δ F1
GCR
Composition Type
Conjunction
1575
85.95
83.18
+2.77
67.52
Composition
1546
82.34
81.91
+0.43
62.39
Comparative
213
82.02
80.88
+1.14
49.62
Superlative
197
79.70
76.53
+3.17
45.05
Constraint Complexity
0
1176
85.26
84.70
+0.56
65.33
1
1511
84.90
82.94
+1.96
61.03
Table 6: CWQ performance by query complexity (F1, %). Δ F1 = Full − Stage 1+3 (i.e., the contribution of Stage 2 correction). GCR results are included for cross-method comparison.
Role
Model
WebQSP
CWQ
Hits@1
F1
Hits@1
F1
Specialized (Stage 1)
Qwen2.5-7B
88.04
84.67
88.19
81.32
Qwen3-8B
88.04
84.80
89.32
83.32
Llama-2-7B
90.73
86.43
88.96
83.72
Llama-3.1-8B
91.15
86.38
89.66
83.78
General (Stage 2)
Qwen-2.5-7B
90.79
86.25
89.49
83.63
Table 7: Impact of model choice in RouterKGQA’s specialized and general roles.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Aspect
Category
CWQ
WebQSP
#
%
#
%
Depth
1-hop
854
24.19
1,042
63.58
2-hop
1,859
52.65
592
36.12
3-hop+
818
23.17
5
0.31
Constraints
0
1,176
33.31
1,169
71.32
1
1,511
42.79
332
20.26
Appendix
Table 8: Test-set distributions of question depth and constraint complexity on WebQSP and CWQ.
Hyperparameter
WebQSP
CWQ
Stage 1: CRP Generation (Specialized Model)
Fine-tuning Type
LoRA
LoRA
Learning Rate
5e-5
5e-5
LR Scheduler
Cosine
Cosine
Train Batch Size
4
4
Gradient Accum. Steps
4
4
Appendix
Table 9: Implementation details of RouterKGQA. Stage 1 settings are shared across Llama-2-7B and Llama-3.1-8B unless otherwise noted.
Category
Method
Reprod.
Notes
LLM-only
Llama-2-7B
✓
Direct inference
Llama-3.1-8B
✓
Direct inference
GPT-4o-mini
✓
Direct inference
IR-based
KV-Mem
✗
Original paper
PullNet
✗
Original paper
EmbedKGQA
✗
Original paper
Appendix
Table 10: Baseline reproduction status. ✓: reproduced in our environment; ✗: results from the original paper. † The original ChatKBQA uses Llama-2-13B on CWQ and Llama-2-7B on WebQSP; we unify both to Llama-2-7B for fair comparison under the same model scale.
Figure 2: Sub-question consistency analysis on CWQ. (a) Distribution of sub-question set relationship categories across methods. (b) Hits@1 (solid) and F1 (dotted) by sub-question relationship category.