Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire external information, and then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA for ~10B-scale LLMs.
Figures & tables
Figure 1: LLMs fail to solve open-domain complex questions due to unrecognized entities and implicit strategies. (1) In the upper part, the LLM fails to answer the question with the one-shot generation, for there is no off-the-shelf answer or evidence to this question. However, the question can be decomposed into several sub-questions and be solved once the citizenship contradiction is identified. If the hint of ‘citizenship contradiction’ is also given, the LLM can successfully solve the question with the inner knowledge now. (2) But for the bottom case with less well-known entities, the LLM fails again due to a lack of specialized knowledge about ‘Aisin-Gioro Yizhu’ and hence rejects to answer. Moreover, the strategy of the ‘political system’ is not likely to be discovered from the question text only, unless enough knowledge is provided. ‘Citizenship contradiction’ is also a possible solution.
Figure 2: GEEK workflow: the core model, retriever and extractor collaborate to solve complex questions progressively. (Left): In each iteration, based on the question state Qt , GEEK selects an action and calls the corresponding module to execute. The execution updates the question state in turn, until a final answer \mathpzcz is derived. (Right): The detailed procedure of action selection and execution. For action selection, Qt and A are fed into the core model with the instruction for action selection, and the model outputs an action code \mathpzca . For the execution of Add Decomp, Final Answer, and Self Answer, the core model outputs corresponding responses following different instructions. At last, for Retrieve & Extract, the retriever firstly retrieves several paragraphs P from the corpus as external knowledge, and then the extractor answers the decomposition question dt based on P .
Figure 3: Full process of GEEK inference. For each round, the prompts are shown in gray, and the current question state is also given to the model as input. Model responses are shown in green and action selection is represented by a red circle to save space. On the top right corner, the question state is listed, where the historical states of sub-question and fact are gradually added during the inference (best viewed in color and numerical marks).
Method
Backbone
Retrieve
Specification
SQA
ChatGPT Qin et al. (2023)
GPT-3.5 (175B)
✗
Without CoT
59.2
ChatGPT Qin et al. (2023)
GPT-3.5 (175B)
✗
CoT
62.5
FaithfulCoT Lyu et al. (2023)
code-davinci-002 (175B)
✗
-
73.2
Xie et al. (2023)
code-davinci-002 (175B)
✗
-
77.2
Lazaridou et al. (2022)
Gopher (280B)
✓
-
66.2
Visconde Pereira et al. (2023)
text-davinci-002 (175B)
✓
CoT
69.43
Table 1: Experiment results on strategyQA dataset. GEEK achieves the SOTA accuracy for LLMs in ∼ 10B scale, and surpasses all the previous methods with backbone under 300B scale, using only 6% parameters or less.
De
RE
SE
Acc
Zero-shot
✗
✗
✗
62.01
CoT
✗
✗
✗
70.74
+De
✓
✗
✗
71.50
+RE
✓
✓
✗
75.98
Full
✓
✓
✓
78.17
Table 2: Ablation study results. The three columns denote whether an action is performed under GEEK. (‘De’: ‘AddDecomp’, ‘RE’: ‘Retrieve and Extract’, ‘SE’: ‘Strategy Exploration’)
Human
Equal
GEEK
ChatGPT
13.54%
24.02%
62.45%
Table 3: ChatGPT Assessment of prediction results.