Improving Parametric Knowledge Access in Reasoning Language Models

Melody Ma; John Hewitt

cs.CLFeb 25, 2026

Improving Parametric Knowledge Access in Reasoning Language Models

Authors: Melody Ma, John Hewitt

Organizations: Columbia University

Abstract

We study reasoning for accessing world knowledge stored in a language model's parameters. For example, recalling that Canberra is Australia's capital may benefit from thinking through major cities and the concept of purpose-built capitals. While reasoning language models are trained via reinforcement learning to produce reasoning traces on tasks such as mathematics, they may not reason well for accessing their own world knowledge. We first find that models do not generate their best world knowledge reasoning by default: adding a simple "think step-by-step" cue demonstrates statistically significant improvement in knowledge recall but not math. Motivated by this, we propose training models to reason over their parametric knowledge using world-knowledge question answering as a verifiable reward. After reinforcement learning on TriviaQA (+9.9%), performance also improves on Natural Questions, HotpotQA, SimpleQA, and StrategyQA by 4.2%, 2.1%, 0.6%, and 3.0%, respectively. Reasoning models are under-optimized for parametric knowledge access, but can be easily trained to reason better.

Figures & tables

Figure 1: Left: GPT-OSS-20B performance: base vs. with think step-by-step cue vs. post-RLVR. Right: Sample reasoning trace from the post-RLVR model on a Natural Questions example.

Cue?	Reasoning?	TriviaQA	NQ	MATH
GPT-OSS-20B
$-$ Cue	$-$ Reasoning	45.2%	24.6%	25.9%
$-$ Cue	$+$ Reasoning	60.1%	30.7%	80.9%
$+$ Cue	$+$ Reasoning	61.2% ∗	32.0%	80.4%
Olmo-3-7B-Think
$-$ Cue	$-$ Reasoning	41.4%	23.1%	71.6%

Table 1: Performance across datasets and models with and without the think step-by-step cue and reasoning tokens. NQ = Natural Questions.

+

denotes presence and

-

denotes absence. Ex-Recall for TriviaQA and NQ; accuracy for MATH.

*

indicates the difference between

+

Cue

+

Reasoning and

-

Cue

+

Reasoning is statistically significant at the 95% level by McNemar’s test ( McNemar, 1947 ) .

		TriviaQA	NQ	HotpotQA	SimpleQA	StrategyQA
Model	Metric
Base	EM	36.5	6.0	7.5	1.8	71.5
Base	Ex-Recall	60.1	30.7	25.5	3.5	–
Reasoning-SFT	EM	38.8	6.1	7.3	1.9	72.5
Reasoning-SFT	Ex-Recall	63.8	31.5	26.0	4.0	–
RL-trained (ours)	EM	63.6 ∗	18.2 ∗	17.0 ∗	3.3 ∗	74.5 ∗

Table 2: GPT-OSS-20B performance before and after RL training on knowledge recall tasks. For our RL-trained model, ∗ indicates the improvement over the base model is statistically significant at the 95% level by McNemar’s test.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Example
TriviaQA	Q: In which country was the first permanent bungee jumping site situated?
	A: New Zealand
Natural Questions	Q: When was Harry Potter and the Philosopher’s Stone published?
	A: 1997
MATH	Q: If $(2x+5)(x-3)=14$ , find the sum of the possible values of $x$ .
	A: $\frac{1}{2}$

Appendix

Table 3: Example questions and answers from each dataset.

Model	Temperature	Top- $p$
DeepSeek-R1-Distill-Qwen-1.5B	0.6	0.95
Olmo-3-7B-Think	0.6	0.95
GPT-OSS-20B	1.0	1.0

Appendix

Table 4: Decoding hyperparameters used for each model.

Model	Metric	TriviaQA	NQ	HotpotQA	SimpleQA	StrategyQA
RL-trained (no-cues)	EM	63.6	18.2	17.0	3.3	74.5
RL-trained (no-cues)	Ex-Recall	70.0	34.9	27.6	4.1	–
RL-trained (with-cues)	EM	62.1	15.1	14.6	3.1	74.3
RL-trained (with-cues)	Ex-Recall	70.6	36.5	28.2	4.2	–
Delta	EM	-1.5 ∗	-3.1 ∗	-2.4 ∗	-0.2	-0.2
Delta	Ex-Recall	+0.6 ∗	+1.6 ∗	+0.6	+0.1	–

Appendix

Table 5: GPT-OSS-20B performance after RL training with and without the think step-by-step cue on knowledge access datasets. ∗ indicates the delta is statistically significant at the 95% level by McNemar’s test.

Cue?	Reasoning?	Accuracy
Base
$-$ Cue	$+$ Reasoning	80.9%
$+$ Cue	$+$ Reasoning	80.4%
RL-trained
$-$ Cue	$+$ Reasoning	83.9%
$+$ Cue	$+$ Reasoning	80.4%

Appendix

Table 6: GPT-OSS-20B accuracy on MATH before and after RL training. A statistically significant improvement is observed in the no-cue setting (p < 0.05) by McNemar’s test.

Dataset	Model	Avg.	Recalled	Non-Recalled
TriviaQA	Base	94.11	41.04	174.13
	SFT	103.66	46.45	204.60
	RL	106.71	55.41	226.66
	RL+cues	118.65	62.02	254.88
NQ	Base	90.56	44.70	110.93
	SFT	94.60	46.38	116.77

Appendix

Table 7: Average thinking token length across datasets for GPT-OSS-20B variants. SFT here refers to Reasoning-SFT.

Factual Knowledge in Language Models

Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.

See all 195 papers

Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
Nov 8, 2025Renfei Zhang, Manasa Kaniselvan, Rylan Schaeffer abd Niloofar MireshghallahRL for Language Model Reasoning Factual Knowledge in Language Models
Carnegie Mellon University · MIT · Stanford University
Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
May 8, 2026Wanli Yang, Hongyu Zang, Junwei Zhang +5RL for Language Models Factual Knowledge in Language Models
University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS
Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs
Mar 10, 2026Zorik Gekhman, Roee Aharoni, Eran Ofek +3Factual Knowledge in Language Models LLM Reasoning
Google Research · Technion - Israel Institute of Technology · Tel Aviv University

Nov 8, 2025J/K move · Enter open · S save

Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Renfei Zhang, Manasa Kaniselvan, Rylan Schaeffer abd Niloofar Mireshghallah

Carnegie Mellon University · MIT · Stanford University

RL for Language Model Reasoning Factual Knowledge in Language Models Large Reasoning Models

Open paper AskPDF arXiv

Reinforcement learning (RL) is often credited with improving language model reasoning at the expense of knowledge. We challenge this narrative by showing that reasoning models consistently outperform their instruction-tuned versions on pure knowledge recall tasks. These gains do not reflect newly acquired information, but rather an improved procedural skill in navigating and searching existing knowledge hierarchies within the model parameters. Structured prompting, which explicitly guides models through hierarchical traversal -- recovers most of the instruct-reasoning gap across five model families. A controlled RL experiment on unseen, non-extractable facts improves recall of held-out frequent but previously inaccessible facts, ruling out simple data exposure. On depth-stratified retrieval tasks, reasoning models exhibit superior traversal as retrieval depth grows. Layerwise activation analysis further shows that while factual representations maintain high cosine similarity between instruct and reasoning models, query representations diverge noticeably, indicating that reasoning primarily reshapes how models traverse knowledge rather than the knowledge representation itself. Finally, we find that distilled models often fail to match reasoning models on knowledge recall because they imitate self-correction without acquiring the exploratory behavior needed for hierarchical navigation. Together, these findings suggest that improving factual recall in LLMs depends not only on expanding what models know but also on teaching them to navigate it -- motivating future post-training methods that optimize traversal.

Abstract

Factual Knowledge in Language Models

Explore similar work

Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs

Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs