Organizations: Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Dr.-Luise-Herzberg-Straße 4, 90461 Nuremberg, Germany
Large Language Models (LLMs) show remarkable performance based on parametric knowledge, i.e., their ability to recall facts and associations internalized in a model's weights. However, studying this capability on established multi-hop benchmarks may be complicated by benchmark contamination, as many of these benchmarks have been publicly available long enough to plausibly appear in modern pretraining corpora. As a result, correct answers under missing evidence may reflect either general parametric knowledge or memorization of benchmark-specific instances. To study how models use internal knowledge when retrieval evidence is incomplete, we introduce FondueQA, in which every required multi-hop fact is uniquely localized to a designated evidence document. This enables controlled document-level evidence ablations to study parametric compensation under incomplete retrieval. Our empirical evaluation reveals that parametric compensation strongly depends on both entity popularity and the structural role of missing evidence. Generally, models compensate more successfully for facts involving popular entities. On FondueQA, we show that bridging evidence can be more useful than the answer-bearing document itself. Moreover, models may recall a missing fact in isolation but fail to use it under partial evidence, particularly when the context contains plausible competing entities, which can reduce Exact Match by up to 15 percentage points.
Figures & tables
Structure
∣F∣
Question q asks for
Bridge
2-3
Final entity e2 of (e0,r1,e1),(e1,r2,e2) with e1 unnamed in q
Comparison
2
Which of two entities has the smaller/larger v (or whether v1=v2 ) for the same relation
Cross-Property
2
Whether v1=v2 for two different relations
Contrastive
2
Which of two entities satisfies v=v∗ : A, B, both, or neither
Nested Bridge
4
Comparison of the end values of two bridge paths
Table 2: Evidence structures. v1,v2 denote the property values and v∗ a fixed reference value.
Figure 1: Evidence structure and popularity shape compensation. Left: Failure rates under role-specific evidence gaps for position-matched bridge questions. Right: Performance loss Δremoval(fi)=EMfull−EM−fi as a function of entity popularity. Error bars show 95% bootstrap confidence intervals.
Figure 2: Parametric availability and contextual behavior under partial evidence. Left: Joint outcomes of atomic recall of the missing fact and correctness of the corresponding multi-hop question under partial evidence. Right: Contrastive questions with partial evidence systematically shift response behavior toward the evidence-supported entity.
Figure 3: Candidate competition. Left: Δ EM at k=19 across candidate-consistency conditions. Right: Δ EM with increasing numbers of random candidates in reasoning-mode.
Figure 4: Competing evidence. Exact Match with increasing number of candidate documents. Left: Same-Gold candidates vs. Right: Same-Other candidates.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Example question
Example answer
Bridge
In which country is the director of the movie Titanic born?
Canada
Comparison
Do Barack Obama and George W. Bush have the same occupation?
Yes
Contrastive
Who was born in Honolulu, Barack Obama or George W. Bush?
Barack Obama
Cross-property
Is the director of Titanic also the screenwriter of Avatar ?
Yes
Nested Bridge
Who is older, the director of Titanic or the director of Jaws ?
Steven Spielberg; Director of Jaws
Appendix
Table 3: Question families in FondueQA. Representative examples illustrating the reasoning structure and typical answer form of each question type.
Figure 5: FondueQA construction filtering. Intermediate bars show the number of candidate questions removed by each validation or filtering step, while the first and final bars report the initial candidate pool and resulting validated dataset size.
Statistic
Value
Generated candidates
198,375
Candidates with complete gold context
163,499
Validated construction pool
31,077
Published paper release
13,063
Validated release backlog
18,014
Documents per question
20
Appendix
Table 4: FondueQA construction pool and rebalanced paper release. Percentages and distributional statistics refer to the final 13,063-question release.
Family
Model
Parameters
Architecture
Standard
Reasoning
Qwen
Qwen3.8-27B
27B
Dense
✓
✓
Qwen3.6-35B-A3B
35B / 3B active
MoE
✓
✓
Qwen3.6-27B
27B
Dense
✓
✓
Qwen3-8B
8B
Dense
✓
✓
Qwen3-32B
32B
Dense
✓
✓
Google
Gemma-4-E2B-IT
2B
Dense
✓
✓
Appendix
Table 5: Language models evaluated in our experiments. We evaluate standard and reasoning inference whenever supported by the model and our serving configuration. For mixture-of-experts models, parameter counts are reported as total / active parameters.
Outcome
HotpotQA -Ext
2WikiMultiHopQA -Ext
MINTQA -Ext
MuSiQue -Ext
Initial candidates
7,404
12,576
17,887
2,417
Pre-validation: single-hop
–
–
5,894
–
Pre-validation:
document resolution
–
–
2,592
–
Entered validation
7,404
12,576
9,401
2,417
Evidence mapping
414
224
0
192
Appendix
Table 6: Filtering outcomes for the extended benchmarks. MINTQA additionally requires exclusion of single-hop instances and document resolution before entering the common validation pipeline. Retention rates in parentheses are computed relative to the initial candidate pool.
Evidence availability
Evidence placement
Setting
Gold evidence present
Setting
Placement
Question only
None
Front
Starts at position 1
No gold
None; distractors only
Middle
Centered at position 10
All gold
All gold documents
End
Ends at position 19
Role only
One selected role
Leave-one-out
All except one
Appendix
Table 7: Factorized context construction. Each evaluation context combines an evidence-availability setting with a compatible evidence-placement setting. Availability determines which gold documents are present; placement determines their positions in the 20-document context.
k
Front
Middle
End
1
1
10
19
2
1–2
10–11
19–20
3
1–3
9–11
18–20
4
1–4
9–12
17–20
Appendix
Table 8: Gold-document positions. Positions depend on the number of retained gold documents k .
Figure 6: Position-matched role comparison. Difference in Exact Match between Bridge-only and Target-only when the retained gold document is placed at the same context position in both conditions. Positive values indicate higher performance with Bridge-only evidence.
Figure 7: Candidate competition across competition levels. Change in Exact Match relative to the k=0 condition as the number of random type-compatible competitors increases.
Figure 8: Candidate-consistency effects across competition levels. Exact Match with increasing numbers of competing candidates. Left: Same-Gold candidates support the correct terminal value. Right: Same-Other candidates consistently support an alternative value.
Figure 9: Entity popularity and parametric compensation on the extended benchmarks. Left: 2WikiMultiHopQA-Ext , Middle: HotpotQA-Ext , Right: MINTQA-Ext
Figure 10: Role-specific Evidence Gaps on the external benchmarks. Failure after removing bridge or target evidence, conditioned on full-context success. 2WikiMultiHopQA-Ext (left) and HotpotQA-Ext (middle) predominantly favor Target-only evidence, whereas MINTQA-Ext (right) strongly favors Bridge-only evidence.
Figure 11: Decomposition of the cross-benchmark role effect. Partial-context EM conditioned on full-context success for Bridge-only and Target-only evidence. Differences across benchmarks are particularly pronounced in the Bridge-only condition.
Figure 12: Role effects after controlling for entity popularity. Bridge-only minus Target-only EM after one-to-one popularity matching without replacement (left) and reweighting FondueQA toward the full 2WikiMultiHopQA-Ext popularity profile (right). The cross-benchmark reversal persists under both controls.
Figure 13: Role effects on questions reliably solved by other models. Bridge-only minus Target-only EM for questions on which the remaining evaluated models achieve at least 90% mean full-context EM.
Dataset
Documents
Tokens
FondueQA
20
7,243
HotpotQA-Ext
10
1,268
HotpotQA
10
1,383
2WikiMultiHopQA-Ext
10
732
MINTQA-Ext
10
952
Appendix
Table 9: Context characteristics for the role analysis. Token counts denote median inference-time context lengths.
Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at bequ.scads.ai, the benchmark website.
Although Large Language Models (LLMs) encode rich factual knowledge in their parameters, reliably recalling and verifying such knowledge remains a key bottleneck in factual question answering. Existing end-to-end methods entangle knowledge elicitation with reasoning, making it difficult to determine whether correct answers arise from parametric knowledge or the input context. To address this challenge, we propose VAKE (Verifiable Activation of Parametric KnowledgE), a two-stage reinforcement-learning framework that externalizes latent parametric knowledge through explicit Priming and transfers the acquired elicitation capability to implicit Reasoning. Given a query and an insufficient retrieved subgraph, the Priming policy explicitly inserts bridging triples as verifiable evidence, with supervision provided by rewards derived from answers generated by a separate frozen model over the augmented subgraph. Building on the policy learned during Priming, the Reasoning stage trains the model to answer from the original input, testing whether the capability acquired through explicit knowledge elicitation transfers to implicit reasoning. Experiments across seven benchmarks and models from 3B to 14B show that VAKE consistently outperforms standard baselines, including when transferring directly from HotpotQA to OOD datasets. LLM-based evaluation further shows that over 80% of the inserted triples provide factual bridging knowledge not derivable from the retrieved context, while more than half elicit knowledge inaccessible through direct prompting. These results suggest that VAKE activates latent parametric knowledge rather than copying the input context or memorizing dataset-specific associations.
Zuocheng Ying, Yang Yang, Yumou Wu +6
ByteDance · Huazhong University of Science and Technology
Long-term memory enables language models to use past interactions in future conversations. However, evidence needed to answer a question may be scattered across distant turns, while the question itself omits clues needed to locate it. Retrieved facts can reveal these clues, motivating retrieval decisions conditioned on evidence already found. We introduce MERA (Missing-Evidence Retrieval Augmentation), which separates globally searchable memory from a question-specific evidence state. Verified evidence guides subsequent retrieval without restricting access to the global memory. We train a lightweight planner through reinforcement learning, rewarding queries that recover previously missing evidence. MERA achieves strong answer accuracy across Qwen3-30B and GPT-4o-mini backbones. With Qwen3-30B for evidence processing and answer generation, the trained 0.6B planner achieves 77.40% accuracy on LoCoMo and 71.29% on LongMemEval-S, exceeding a 30B planner without retrieval-grounded training by 4.10% and 3.96%, respectively. On LoCoMo, later retrieval rounds increase cumulative evidence recall from 55.5% to 80.5%.