MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

Minsuh Joo; Youna Kim; Choonghyun Park; Soo Kyung Kim; Hyunsoo Cho

cs.LGOct 14, 2026

MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

Authors: Minsuh Joo, Youna Kim, Choonghyun Park, Soo Kyung Kim, Hyunsoo Cho

Organizations: Department of AI, Ewha Womans University · Seoul National University · Institute for Multiscale Matter and Systems, Ewha Womans University

Abstract

LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques such as activation steering, decoding adjustments, and prompting modify inference, potentially reinforcing existing predictions or surfacing less accessible answers. Yet their evaluations centered on aggregate accuracy, which obscures diverse effects by collapsing recovered failures and disrupted successes into a single score. We introduce MAPLE, a unified framework for characterizing how KEE techniques affect parametric knowledge extraction. Its diagnostic metrics disentangle these effects at the surface, probability, and beyond-sampling levels by measuring answer recovery and disruption, gold-token rank shifts on unrecovered failures, and recovery of answers missed by fixed-budget sampling. We evaluate six KEE techniques spanning decoding, activation steering, and prompting across three model families: OLMo2, Qwen2.5, and Llama3. Our analysis reveals distinct effects and trade-offs across techniques, highlighting the value of MAPLE for understanding and comparing KEE beyond aggregate accuracy.

Figures & tables

	$\tau_{0.7}$		$\tau_{1.2}$
$K$	SGR	DGR	SGR	DGR
10	0.915	0.941	0.988	0.898
15	0.927	0.969	0.958	0.956
20	1.000	1.000	1.000	1.000
25	0.983	0.991	0.972	0.950
30	0.956	0.987	0.931	0.950

Table 1: Mean Kendall rank correlation with the ranking at

K=20

Category	Model	Acc $\uparrow$	Gain $\uparrow$	Loss $\downarrow$	Model	Acc $\uparrow$	Gain $\uparrow$	Loss $\downarrow$	Model	Acc $\uparrow$	Gain $\uparrow$	Loss $\downarrow$
	OLMo2-7B	46.89	–	–	Qwen2.5-7B	43.13	–	–	Llama3-8B	50.58	–	–
Sampling	+ $\tau_{0.7}$	47.35	3.32	2.78	+ $\tau_{0.7}$	43.21	1.35	1.60	+ $\tau_{0.7}$	50.88	2.95	2.28
	+ $\tau_{1.2}$	45.43	3.61	7.20	+ $\tau_{1.2}$	42.98	3.51	4.98	+ $\tau_{1.2}$	50.27	6.21	6.68
	+ DeCoRe	46.89	3.32	3.76	+ DeCoRe	41.29	4.32	9.96	+ DeCoRe	53.34	11.18	5.46
Decoding	+ AD	46.12	1.88	3.76	+ AD	40.45	3.78	11.21	+ AD	52.11	10.09	6.83
Steering	+ ITI	47.89	4.91	3.44	+ ITI	42.06	1.21	4.09	+ ITI	50.96	4.66	3.79

Table 2: Gain/Loss (%) decomposition across model families and KEE techniques. Bold and underline indicate the best and second-best values, respectively. Alternating gray shading indicates technique category boundaries.

Figure 2: Probability Rank Shift of the gold token on unrecovered failures for KEE techniques, averaged over six models. Range bars show the minimum and maximum across models. Per-model results are in Figure 12 .

Category	Model	SGR $\uparrow$	DGR $\uparrow$	Model	SGR $\uparrow$	DGR $\uparrow$	Model	SGR $\uparrow$	DGR $\uparrow$
	OLMo2-7B	–	–	Qwen2.5-7B	–	–	Llama3-8B	–	–
Sampling	+ $\tau_{1.2}$	16.06	0.42	+ $\tau_{1.2}$	18.67	0.17	+ $\tau_{1.2}$	15.72	0.24
	+ DeCoRe	9.63	0.42	+ DeCoRe	16.00	1.35	+ DeCoRe	24.02	4.10
Decoding	+ AD	5.96	0.00	+ AD	15.33	0.85	+ AD	20.96	4.10
Steering	+ ITI	14.68	0.42	+ ITI	6.00	0.00	+ ITI	10.04	1.69
	+ CAA	9.63	0.00	+ CAA	6.67	0.17	+ CAA	23.58	4.10

Table 3: SGR/DGR (%) across model families and KEE techniques at

\tau{=}1.2

. Bold and underline indicate the best and second-best values, respectively. Alternating gray shading indicates technique category boundaries.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Instance of ( P31 )
Culture	literary work, museum, art movement, heritage site, cultural center
Education	school, university, college, campus, faculty
Health	hospital, anatomical structure, medical treatment, disease, symptom or sign
History	battle, war, dynasty, historical period, invasion
Law	legal case, prison, court, criminal case, decision of the Supreme Court of the United States
Politics	political organization, electoral unit, public election, constituency of the European Parliament, executive branch

Appendix

Table 4: Categories with representative instance of ( P31 ) types.

Panel of Figure 3 — Figure 3: System prompt for the domain classifier.

Domain	In-domain scope ( <SCOPE> )
Culture	literature, books, authorship, visual arts, paintings, sculpture, architecture, performing arts, dance, museums, libraries, collections of cultural works, monuments, cultural heritage, cultural institutions, festivals, and directly related cultural concepts
Education	schools, universities, colleges, faculties, academic departments, school districts, educational institutions, academic programs, degrees, curricula, teachers, students, education systems, and directly related educational concepts
Health	diseases, symptoms, anatomy, organs, mental health, bacteria, viruses, drugs, vaccines, medicines, treatments, antibiotics, medical procedures, diagnosis, healthcare institutions, hospitals, healthcare organizations, doctors, nurses, medical specialties, and any entity or fact closely associated with these topics
History	historical events, wars, battles, conflicts, military, empires, dynasties, reigns, historical periods, historical rulers, archaeological periods, and directly related historical concepts
Law	legal cases, court decisions, laws, legal concepts, crimes, courts, judicial institutions, judges, legal offices, constitutions, and directly related legal concepts
Politics	political parties, elections, governments, parliaments, ministries, presidents, prime ministers, politicians, legislation, political institutions, protests, revolution, political ideologies, diplomacy, international relations, and directly related political concepts

Appendix

Table 5: Per-domain instantiation of the prompt template in Figure 3 . Each row provides the <DOMAIN> name and the <SCOPE> string substituted into the shared template.

Panel of Figure 4 — Figure 4: System prompt for the answer cardinality classifier.

Subject	Relation	Object	Label	Reason
Italy	capital	Rome	SINGLE	Italy has exactly one capital, Rome, which has not changed.
aspirin	medical condition treated	pain	MULTIPLE	Aspirin treats multiple conditions including pain, fever, inflammation, and cardiovascular disease.
Microsoft	headquarters location	Redmond	MULTIPLE	The headquarters of Microsoft has changed over time (Albuquerque, then Redmond), so the answer depends on the time period.
Winston Churchill	conflict	World War II	MULTIPLE	Churchill participated in multiple conflicts including the Second Boer War, World War I, and World War II.
amygdala	part of	limbic system	SINGLE	The amygdala is part of exactly one brain system: the limbic system.
Kaba-class destroyer	country	Japan	UNSURE	Not confident enough about this specific naval vessel class to determine whether it was operated by one or multiple countries.

Appendix

Table 6: Few-shot demonstrations provided to the answer cardinality classifier. Each demonstration pairs an input triplet with its gold label and rationale.

Figure 5: Distribution of the 1,303 triplets across domain, frequency bin, and relation, showing broad knowledge coverage along all three axes. Bins are defined by frequency percentiles rather than the targeted proportions; the tail bin is smaller (7.4%) because low-frequency entities yield fewer triplets under our relation and domain constraints. The relation chart shows the top-20 of 97 unique types.

Model	num_retrieval_head
Model	10	20	30	40	50
Llama3-8B-Instruct	0.5236	0.5265	0.5178	0.5170	0.5167
Qwen2.5-7B-Instruct	0.4017	0.4028	0.3981	0.3899	0.3748

Appendix

Table 7: DeCoRe accuracy on our benchmark as num_retrieval_head varies from 10 to 50. Unlike the main results, which use greedy decoding, accuracy here is the mean per-fact correctness over

N=10

sampled trials at temperature

T=0.7

Model	info_layer
OLMo2-7B-Instruct	26 / 32
OLMo2-32B-Instruct	52 / 64
Qwen2.5-7B-Instruct	23 / 28
Qwen2.5-32B-Instruct	52 / 64
Llama3-8B-Instruct	26 / 32
Llama3-70B-Instruct	65 / 80

Appendix

Table 8: Per-model info_layer for Activation Decoding.

Model	intervention_layer
Model	Fold 1	Fold 2
OLMo2-7B-Instruct	19	11
OLMo2-32B-Instruct	48	43
Qwen2.5-7B-Instruct	17	19
Qwen2.5-32B-Instruct	47	40
Llama3-8B-Instruct	9	18

Appendix

Table 9: Per-fold intervention_layer selected for CAA and applied to the other.

Panel of Figure 6 — Figure 6: RECITE Stage 1 prompt. Five demonstrations of this form are provided in the system message.

Panel of Figure 7 — Figure 7: RECITE Stage 2 prompt.

Panel of Figure 8 — Figure 8: GenRead Stage 1 prompt (zero-shot).

Panel of Figure 9 — Figure 9: GenRead Stage 2 prompt.

Panel of Figure 10 — Figure 10: Prompt template for generating the output.

Question	Answer
What religious order was Koenigsbruck Abbey associated with?	Cistercian
Which Olympic Games did John Garrels participate in?	1908 Summer Olympics
Which dynasty was Thutmose III part of?	eighteenth dynasty
Who founded World Book and Copyright Day?	UNESCO
Which organization is Llanelli Scarlets affiliated with?	Welsh Rugby Union

Appendix

Table 10: Five few-shot demonstrations used for the output generation.

Model	Metric	Decoding		Steering		Prompting		Temperature Sampling		Denominator
Model	Metric	DeCoRe	AD	ITI	CAA	RECITE	GenRead	$\tau=0.7$	$\tau=1.2$	Denominator
OLMo2-7B	Gain $\uparrow$	23	13	34	21	161	138	23	25	692
OLMo2-7B	Loss $\downarrow$	23	23	21	31	39	63	17	44	611
OLMo2-32B	Gain $\uparrow$	23	15	14	28	129	112	15	33	479
OLMo2-32B	Loss $\downarrow$	19	27	12	35	49	67	28	36	824
Qwen2.5-7B	Gain $\uparrow$	32	28	9	11	153	185	10	26	741

Appendix

Table 11: Raw counts for Table 2 .

Panel of Figure 11 — Figure 11: Proportion of shallow failures identified with $K$ samples relative to those identified with 30 samples, $|\mathcal{D}_{\text{shallow}}(K)|/|\mathcal{D}_{\text{shallow}}(30)|$ , for $K$ from 1 to 30. (a) The left panel shows temperature sampling at $\tau=0.7$ and (b) the right panel shows $\tau=1.2$ . The dotted line marks 90%. Coverage rises steeply for small $K$ and saturates afterward, and at $K=20$ every model reaches about 90% or more at both temperatures. We therefore use $K=20$ as a balance between coverage and sampling cost.

Category	Model	SGR $\uparrow$	DGR $\uparrow$	Model	SGR $\uparrow$	DGR $\uparrow$	Model	SGR $\uparrow$	DGR $\uparrow$
	OLMo2-7B	–	–	Qwen2.5-7B	–	–	Llama3-8B	–	–
Sampling	+ $\tau_{0.7}$	16.15	0.18	+ $\tau_{0.7}$	11.11	0.00	+ $\tau_{0.7}$	14.47	0.00
	+ DeCoRe	16.15	0.36	+ DeCoRe	24.44	1.54	+ DeCoRe	32.70	4.12
Decoding	+ AD	10.00	0.00	+ AD	23.33	1.08	+ AD	27.67	4.33
Steering	+ ITI	17.69	1.96	+ ITI	6.67	0.46	+ ITI	12.58	2.06
	+ CAA	15.38	0.18	+ CAA	11.11	0.15	+ CAA	29.56	4.95

Appendix

Table 12: SGR/DGR (%) across model families and KEE techniques at

\tau{=}0.7

. Bold and underline indicate the best and second-best values, respectively. Alternating gray shading indicates technique category boundaries.

Model	Metric	Temp.	Decoding		Steering		Prompting		Denominator
Model	Metric	0.7	DeCoRe	AD	ITI	CAA	RECITE	GenRead	Denominator
OLMo2-7B	SGR $\uparrow$	21	21	13	23	20	72	62	130
OLMo2-7B	DGR $\uparrow$	1	2	0	11	1	89	76	562
OLMo2-32B	SGR $\uparrow$	15	21	15	14	27	74	69	130
OLMo2-32B	DGR $\uparrow$	0	2	0	0	1	55	43	349
Qwen2.5-7B	SGR $\uparrow$	10	22	21	6	10	54	56	90

Appendix

Table 13: Raw counts for Table 12 .

Model	Metric	Temp.	Decoding		Steering		Prompting		Denominator
Model	Metric	1.2	DeCoRe	AD	ITI	CAA	RECITE	GenRead	Denominator
OLMo2-7B	SGR $\uparrow$	35	21	13	32	21	113	97	218
OLMo2-7B	DGR $\uparrow$	2	2	0	2	0	48	41	474
OLMo2-32B	SGR $\uparrow$	26	21	14	14	28	97	86	199
OLMo2-32B	DGR $\uparrow$	0	2	1	0	0	32	26	280
Qwen2.5-7B	SGR $\uparrow$	28	24	23	9	10	80	90	150

Appendix

Table 14: Raw counts for Table 3 .

Model	Category	Decoding		Steering		Prompting
Model	Category	DeCoRe	AD	ITI	CAA	RECITE	GenRead
OLMo2-7B	$n_{00}$	669	679	658	671	531	554
	$n_{\text{exc}}$	102	91	74	88	61	88
	$n_{\text{inc}}$	567	588	584	583	470	466
	Improved	28.7 (163)	25.7 (151)	20.0 (117)	36.4 (212)	18.1 (85)	17.8 (83)
	Tied	23.8 (135)	39.5 (232)	60.8 (355)	25.4 (148)	6.4 (30)	3.9 (18)

Appendix

Table 15: Per-model PRS on unrecovered failures. Among the

n_{00}

unrecovered failures,

n_{\text{exc}}

are excluded because the gold token is ranked first by either the baseline or the KEE technique, and for RECITE this includes ranking first in any of its 20 recitation paths. The remaining

n_{\text{inc}}=N_{\text{Improved}}+N_{\text{Tied}}+N_{\text{Worsened}}

failures are categorized by Eq. 3 , and each category reports

\text{PRS}_{c}

in percent followed by

N_{c}

in parentheses. For RECITE, each failure is assigned the most frequent category across its 20 recitation paths.

Figure 12: Per-model PRS on unrecovered failures in a bar graph.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

May 26, 2026

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.

Luca Giordano, Simon Razniewski

ScaDS.AI Dresden/Leipzig & TU Dresden, Germany

LLM Evaluation Factual Knowledge in Language Models

Ask PDF

Oct 2, 2026

LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

Current analyses of LLMs' parametric knowledge are largely output-centric, drawing conclusions about what a model knows without verifying what it was actually trained on. This leaves fundamental questions, such as whether a correct response reflects genuine generalization or rote memorization, grounded in speculation rather than evidence. To resolve these ambiguities, we introduce LUMOS, a diagnostic framework that traces knowledge along the causal chain from training-data exposure to behavioral output, leveraging OLMo 2 with its fully transparent training corpus. By grounding analysis in verified exposure, we reveal that models internally encode rare facts with high separability (84%) yet fail to express them behaviorally (54%), though this retrieval gap narrows with scale. Furthermore, when models are asked to self-reflect on their own answers, they perform reliably on trained content (83%) but drop to random-baseline levels (49%) on unseen content. This collapse persists even under chain-of-thought prompting, which inflates confidence signals rather than improving calibration. Collectively, these findings demonstrate that incorporating the training-data axis into LLM evaluation transforms speculative diagnoses into verifiable claims, and we advocate that this axis should be a standard component of knowledge assessment in LLMs.

Seoyeon Ye, Gayoung Kim, Jiyoung Hong +2

Ewha Womans University

Factual Knowledge in Language Models LLM Evaluation Training Data Attribution Language Model Self-Assessment

Ask PDF

Aug 19, 2026

From Storage to Access: Verifiable Activation of Parametric Knowledge in LLMs via Explicit Priming and Implicit Reasoning

Although Large Language Models (LLMs) encode rich factual knowledge in their parameters, reliably recalling and verifying such knowledge remains a key bottleneck in factual question answering. Existing end-to-end methods entangle knowledge elicitation with reasoning, making it difficult to determine whether correct answers arise from parametric knowledge or the input context. To address this challenge, we propose VAKE (Verifiable Activation of Parametric KnowledgE), a two-stage reinforcement-learning framework that externalizes latent parametric knowledge through explicit Priming and transfers the acquired elicitation capability to implicit Reasoning. Given a query and an insufficient retrieved subgraph, the Priming policy explicitly inserts bridging triples as verifiable evidence, with supervision provided by rewards derived from answers generated by a separate frozen model over the augmented subgraph. Building on the policy learned during Priming, the Reasoning stage trains the model to answer from the original input, testing whether the capability acquired through explicit knowledge elicitation transfers to implicit reasoning. Experiments across seven benchmarks and models from 3B to 14B show that VAKE consistently outperforms standard baselines, including when transferring directly from HotpotQA to OOD datasets. LLM-based evaluation further shows that over 80% of the inserted triples provide factual bridging knowledge not derivable from the retrieved context, while more than half elicit knowledge inaccessible through direct prompting. These results suggest that VAKE activates latent parametric knowledge rather than copying the input context or memorizing dataset-specific associations.

Zuocheng Ying, Yang Yang, Yumou Wu +6

ByteDance · Huazhong University of Science and Technology

Factual Knowledge in Language Models Implicit Reasoning in Language Models RL for Language Model Reasoning LLM Reasoning with Graphs

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Beyond Questions: Evaluating the Stated Knowledge of LLMs

LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

From Storage to Access: Verifiable Activation of Parametric Knowledge in LLMs via Explicit Priming and Implicit Reasoning