Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire external information, and then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA for ~10B-scale LLMs.
Figures & tables
Figure 1: LLMs fail to solve open-domain complex questions due to unrecognized entities and implicit strategies. (1) In the upper part, the LLM fails to answer the question with the one-shot generation, for there is no off-the-shelf answer or evidence to this question. However, the question can be decomposed into several sub-questions and be solved once the citizenship contradiction is identified. If the hint of ‘citizenship contradiction’ is also given, the LLM can successfully solve the question with the inner knowledge now. (2) But for the bottom case with less well-known entities, the LLM fails again due to a lack of specialized knowledge about ‘Aisin-Gioro Yizhu’ and hence rejects to answer. Moreover, the strategy of the ‘political system’ is not likely to be discovered from the question text only, unless enough knowledge is provided. ‘Citizenship contradiction’ is also a possible solution.
Figure 2: GEEK workflow: the core model, retriever and extractor collaborate to solve complex questions progressively. (Left): In each iteration, based on the question state Qt , GEEK selects an action and calls the corresponding module to execute. The execution updates the question state in turn, until a final answer \mathpzcz is derived. (Right): The detailed procedure of action selection and execution. For action selection, Qt and A are fed into the core model with the instruction for action selection, and the model outputs an action code \mathpzca . For the execution of Add Decomp, Final Answer, and Self Answer, the core model outputs corresponding responses following different instructions. At last, for Retrieve & Extract, the retriever firstly retrieves several paragraphs P from the corpus as external knowledge, and then the extractor answers the decomposition question dt based on P .
Figure 3: Full process of GEEK inference. For each round, the prompts are shown in gray, and the current question state is also given to the model as input. Model responses are shown in green and action selection is represented by a red circle to save space. On the top right corner, the question state is listed, where the historical states of sub-question and fact are gradually added during the inference (best viewed in color and numerical marks).
Method
Backbone
Retrieve
Specification
SQA
ChatGPT Qin et al. (2023)
GPT-3.5 (175B)
✗
Without CoT
59.2
ChatGPT Qin et al. (2023)
GPT-3.5 (175B)
✗
CoT
62.5
FaithfulCoT Lyu et al. (2023)
code-davinci-002 (175B)
✗
-
73.2
Xie et al. (2023)
code-davinci-002 (175B)
✗
-
77.2
Lazaridou et al. (2022)
Gopher (280B)
✓
-
66.2
Visconde Pereira et al. (2023)
text-davinci-002 (175B)
✓
CoT
69.43
Table 1: Experiment results on strategyQA dataset. GEEK achieves the SOTA accuracy for LLMs in ∼ 10B scale, and surpasses all the previous methods with backbone under 300B scale, using only 6% parameters or less.
De
RE
SE
Acc
Zero-shot
✗
✗
✗
62.01
CoT
✗
✗
✗
70.74
+De
✓
✗
✗
71.50
+RE
✓
✓
✗
75.98
Full
✓
✓
✓
78.17
Table 2: Ablation study results. The three columns denote whether an action is performed under GEEK. (‘De’: ‘AddDecomp’, ‘RE’: ‘Retrieve and Extract’, ‘SE’: ‘Strategy Exploration’)
Human
Equal
GEEK
ChatGPT
13.54%
24.02%
62.45%
Table 3: ChatGPT Assessment of prediction results.
Large Language Models (LLMs) have achieved strong performance in question answering and retrieval-augmented generation (RAG), yet they implicitly assume that user queries are fully specified and answerable. In real-world settings, queries are often incomplete, ambiguous, or missing critical variables, leading models to produce overconfident or hallucinated responses. In this work, we study decision-aware query resolution under incomplete information, where a model must determine whether to Answer, Ask for clarification, or Abstain. We show that standard and enhanced RAG systems do not reliably exhibit such epistemic awareness, defaulting to answer generation even when information is insufficient. To address this, we propose PassiveQA, a three-action framework that aligns model behaviour with information sufficiency through supervised finetuning. Our approach integrates structured information-state representations, knowledge graph-grounded context, and a finetuned planner that explicitly models missing variables and decision reasoning. Experiments across multiple QA datasets show that the finetuned planner achieves significant improvements in macro F1 and abstention recall while reducing hallucination rates, under a compute-constrained training regime. These results provide strong empirical evidence that epistemic decision-making must be learned during training rather than imposed at inference time.
Although Large Language Models (LLMs) encode rich factual knowledge in their parameters, reliably recalling and verifying such knowledge remains a key bottleneck in factual question answering. Existing end-to-end methods entangle knowledge elicitation with reasoning, making it difficult to determine whether correct answers arise from parametric knowledge or the input context. To address this challenge, we propose VAKE (Verifiable Activation of Parametric KnowledgE), a two-stage reinforcement-learning framework that externalizes latent parametric knowledge through explicit Priming and transfers the acquired elicitation capability to implicit Reasoning. Given a query and an insufficient retrieved subgraph, the Priming policy explicitly inserts bridging triples as verifiable evidence, with supervision provided by rewards derived from answers generated by a separate frozen model over the augmented subgraph. Building on the policy learned during Priming, the Reasoning stage trains the model to answer from the original input, testing whether the capability acquired through explicit knowledge elicitation transfers to implicit reasoning. Experiments across seven benchmarks and models from 3B to 14B show that VAKE consistently outperforms standard baselines, including when transferring directly from HotpotQA to OOD datasets. LLM-based evaluation further shows that over 80% of the inserted triples provide factual bridging knowledge not derivable from the retrieved context, while more than half elicit knowledge inaccessible through direct prompting. These results suggest that VAKE activates latent parametric knowledge rather than copying the input context or memorizing dataset-specific associations.
Zuocheng Ying, Yang Yang, Yumou Wu +6
ByteDance · Huazhong University of Science and Technology
Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.