Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models
Authors: Yinan Liu, Dongying Lin, Sigang Luo, Xiaochun Yang, Bin Wang
Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China · National Frontiers Science Center for Industrial Intelligence and Systems optimization, Northeastern University, Shenyang, China
Knowledge Bases (KBs) play a key role in various applications. As two representative KB-related tasks, knowledge base completion (KBC) and knowledge base question answering (KBQA) are closely related and inherently complementary with each other. Thus, it will be beneficial to solve the task of joint KBC and KBQA to make them reinforce each other. However, existing studies usually rely on the small language model (SLM) to enhance them jointly, and the large language model (LLM)'s strong reasoning ability is ignored. In this paper, by combining the strengths of the LLM with the SLM, we propose a novel framework JCQL, which can make these two tasks enhance each other in an iterative manner. To make KBC enhance KBQA, we augment the LLM agent-based KBQA model's reasoning paths by incorporating an SLM-trained KBC model as an action of the agent, alleviating the LLM's hallucination and high computational costs issue in KBQA. To make KBQA enhance KBC, we incrementally fine-tune the KBC model by leveraging KBQA's reasoning paths as its supplementary training data, improving the ability of the SLM in KBC. Extensive experiments over two public benchmark data sets demonstrate that JCQL surpasses all baselines for both KBC and KBQA tasks.
Figures & tables
Figure 1: Overview of our framework JCQL.
Method
WebQSP (30% KB)
WebQSP (50% KB)
CWQ (30% KB)
CWQ (50% KB)
MRR
Hits@1
Hits@3
Hits@10
MRR
Hits@1
Hits@3
Hits@10
MRR
Hits@1
Hits@3
Hits@10
MRR
Hits@1
Hits@3
Hits@10
TransE (NeurIPS’13)
0.032
0.000
0.043
0.090
0.051
0.011
0.068
0.130
0.047
0.011
0.062
0.121
0.052
0.012
0.067
0.133
DistMult (ICLR’15)
0.112
0.079
0.125
0.176
0.166
0.118
0.182
0.260
0.100
0.070
0.111
0.161
0.153
0.107
0.169
0.243
ComplEx (ICML’16)
0.119
0.085
0.132
0.183
0.178
0.127
0.200
0.277
0.106
0.074
0.119
0.166
0.158
0.110
0.177
0.252
RotatE (ICLR’19)
0.055
0.023
0.062
0.124
0.055
0.024
0.062
0.125
0.058
0.027
0.064
0.125
0.047
0.024
0.061
0.125
SimKGC (ACL’22)
0.456
0.377
0.493
0.605
0.480
0.400
0.518
0.632
0.451
0.360
0.493
0.625
0.466
0.375
0.510
0.642
Table 1: Performance on the KBC task. The best results are indicated in bold. The second best results are underlined.
Method
WebQSP
CWQ
w/o KB
GPT-4o-mini
0.677
0.421
CoT (NeurIPS’22)
0.646
0.446
SC (ICLR’23)
0.643
0.464
w/ KB
30% KB
50% KB
30% KB
50% KB
Table 2: Performance on the KBQA task in terms of Hits@1. The best results are indicated in bold. The second best results are underlined.
Figure 2: Ablation study on the LLM’s reasoning paths for the task of KBC.
Method
WebQSP
CWQ
30% KB
50% KB
30% KB
50% KB
JCQL
0.730
0.738
0.502
0.508
JCQL w/o generate
0.706
0.722
0.482
0.490
JCQL w/o complete
0.691
0.696
0.473
0.475
Table 3: Ablation study on the SLM for the task of KBQA in terms of Hits@1.
Figure 3: Parameter study over WebQSP and CWQ under different settings.
Data set
Method
30% KB
50% KB
LLM call
Time (s)
LLM call
Time (s)
CWQ
ToG
18.5
33.7
18.2
32.7
GoG
7.9
26.3
7.7
26.1
PoG
11.7
42.6
9.7
42.8
LMP
11.4
59.4
11.3
58.8
PDRR
8.6
49.5
7.6
45.8
Table 4: Efficiency study on the task of KBQA.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Data set
Entities
Relations
Train
Valid
Test
Isolated nodes
WebQSP
749,492
1,251
1,714,822
20,000
20,000
130,179
CWQ
672,970
1,232
1,479,924
20,000
20,000
119,654
Appendix
Table 5: Statistics of the 50% KB for the two data sets.
Data set
Entities
Relations
Train
Valid
Test
Isolated nodes
WebQSP
749,492
1,251
1,217,899
20,000
20,000
267,114
CWQ
672,970
1,232
1,060,601
20,000
20,000
244,439
Appendix
Table 6: Statistics of the 30% KB for the two data sets.
Data set
Answer format
Train
Test
Coverage
Licence
WebQSP
Entity/Number
2777
3531
0.992
CC Licence
CWQ
Entity
23013
1639
0.997
-
Appendix
Table 7: Statistics of KBQA data sets. Coverage refers to the accuracy of subgraph matching.
Data set
Number path
Number test
Number test /Number path
WebQSP
8,623
27
0.31
CWQ
39,776
73
0.18
Appendix
Table 8: Statistics of the reasoning path. Number path denotes the number of triples in the inference paths. Number test denotes the number of triples belonging to inference paths within the KBC test set.
T
D
WebQSP (30% KB)
WebQSP (50% KB)
CWQ (30% KB)
CWQ (50% KB)
MRR
Hits@1
Hits@3
Hits@10
MRR
Hits@1
Hits@3
Hits@10
MRR
Hits@1
Hits@3
Hits@10
MRR
Hits@1
Hits@3
Hits@10
✓
✓
0.603
0.572
0.628
0.652
0.630
0.599
0.652
0.693
0.588
0.556
0.612
0.645
0.614
0.583
0.634
0.670
✓
✗
0.566
0.533
0.583
0.626
0.598
0.565
0.620
0.657
0.530
0.496
0.552
0.595
0.582
0.548
0.605
0.643
✗
✓
0.581
0.550
0.603
0.634
0.603
0.572
0.626
0.656
0.570
0.539
0.590
0.625
0.598
0.568
0.620
0.650
✗
✗
0.517
0.485
0.537
0.581
0.544
0.509
0.565
0.610
0.503
0.467
0.524
0.569
0.536
0.501
0.559
0.602
Appendix
Table 9: Ablation study of the effect of the input sequence form of the SLM.
Figure 4: Efficiency study of the training process on WebQSP.
Figure 5: Performance of JCQL with different sample numbers in experience replay.
input
⟨ Justin Bieber, father, ? ⟩
sequence
predict tail: Justin Bieber ∣ father entity description: Justin Bieber [Canadian singer (born 1994)] related relationship: father context: ⟨ Justin Bieber ∣ mother ∣ Pattie Mallette ⟩⟨SEP⟩⟨ Justin Bieber ∣ sibling ∣ Jazmyn Bieber ⟩⟨SEP⟩ …
output
Jeremy Bieber
Appendix
Table 10: A typical case of input sequences and corresponding output for the SLM.
Question
what type of government did soviet union have?
JCQL w/o complete
Thought 1: I need to find out what type of government the Soviet Union had. Action 1: Search[Soviet Union] Observation 1: soviet union, history of topic, history of the soviet union soviet union, legislative body, supreme soviet of the soviet union soviet union, official religion, state atheism Thought 2: The Soviet Union had a legislative body called the Supreme Soviet, which implies a type of government. I need to generate more information about the government structure of the Soviet Union based on this observation. Action 2: Generate[type of government of the Soviet Union] Observation 2: soviet union, ideological basis, Marxism-Leninism soviet union, type of government, one-party socialist republic Thought 3: The Soviet Union is characterized as a one-party socialist republic based on its ideological basis of Marxism-Leninism. Action 3: Finish[one-party socialist republic]
JCQL w/o generate
Thought 1: The question requires identifying the type of government in the Soviet Union. I should gather information regarding the political structure and governance of the Soviet Union. Action 1: Search[Soviet Union] Observation 1: Soviet Union, foundational text, Declaration of the Creation of the Union of Soviet Socialist Republics Soviet Union, foundational text, Treaty on the Creation of the Union of Soviet Socialist Republics Soviet Union, legislative body, Supreme Soviet of the Soviet Union member of the Supreme Soviet of the Russian Soviet Federative Socialist Republic, country, Soviet Union Thought 2: The current information provides some context about the Soviet Union but lacks specific details about its type of government. I need to find the governance structure of the Soviet Union. Action 2: Complete[Soviet Union | instance of] Observation 2: Soviet Union, instance of, country Soviet Union, instance of, socialist state Thought 3: The results indicate that the Soviet Union was a socialist state. This clarifies the type of government. Therefore, I can summarize the answers. Action 3: Finish[ socialist state ].
Appendix
Table 11: The comparison between generate and complete under the setting i.e., 50% KB.
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.