Co-Evolving Structured Knowledge and Reasoning in Language Models

Ryan Thomas Noonan; Linxi Zhao; Menghan Xu; Akanksha Sarkar; Mihir Mishra; Dongyoung Go; Kilian Q. Weinberger; Yoav Artzi

cs.CLAug 26, 2026

Co-Evolving Structured Knowledge and Reasoning in Language Models

Authors: Ryan Thomas Noonan, Linxi Zhao, Menghan Xu, Akanksha Sarkar, Mihir Mishra, Dongyoung Go, Kilian Q. Weinberger, Yoav Artzi,

Organizations: Cornell University

Abstract

Retrieval-augmented methods improve factual accuracy by grounding language models in external knowledge, but retrieving over unstructured text often introduces irrelevant context and offers limited control over the retrieved information. Structured knowledge bases offer a more controllable alternative, yet they are expensive to construct and often brittle to reason over. To address these limitations, we propose KBevo: a co-evolving framework that jointly learns to construct a structured knowledge base and reason over it for knowledge-intensive question answering. By optimizing both components end-to-end with QA outcome rewards, our method enables reasoning success to directly improve the quality of the constructed knowledge base. This leads to larger, better-connected knowledge structures with higher answer reachability, while also improving compositional factual reasoning and controllability compared to standard retrieval baselines.

Figures & tables

Figure 1: Co-evolving knowledge and reasoning. Standard retrieval-based RL improves the reasoning policy while operating over fixed text chunks. In contrast, KBevo jointly optimizes reasoning and structured knowledge base construction, allowing the model and its structured knowledge base to improve together during training.

Method	Construction	Schema	Quality Control	Downstream-Coupled	Persistence
EDC	Extract–Define–Canonicalize	Induced	Canonicalization	✗	Persistent
Wikontic	Multi-stage pipeline	Wikidata ontology	Ontology filtering + dedup	✗	Persistent
AutoSchemaKG	Autonomous pipeline	Induced	Schema-guided filtering	✗	Persistent
Ours	Joint RL training	Emergent (task-driven)	QA reward signal	✓	Persistent + Editable

Table 1: Comparison of knowledge graph construction methods.

Figure 2: Overview of our co-evolving framework. The model first constructs a structured knowledge base from context, then answers multihop questions by retrieving and reasoning over it. Both phases are jointly optimized so that knowledge construction and reasoning improve together.

\{G_{b,k}\}_{b=1,k=1}^{B,K}\sim\pi_{\theta}(\cdot\mid p^{\text{kb}}(c_{b}))\quad\forall\,b

Algorithm 1 Co-evolving KB Construction and QA via GRPO

Method	HotpotQA ∗	MuSiQue	2Wiki	PopQA	Avg
Qwen3-1.7B
Direct	12.1	0.7	21.7	16.0	12.6
RAG	25.4	6.3	21.8	57.8	27.8
IRCoT	23.7	12.2	44.0	60.7	35.2
Search-R1 †	44.1	14.3	45.9	65.3	42.4
KBevo -SFT	29.5	13.3	44.9	57.0	36.2

Table 2: Evaluation on multi-hop QA benchmarks and PopQA (EM). ∗ marks the in-domain benchmark. † Reimplemented on Qwen3 backbone. See Section 4 .

Figure 3: Reasoning with edited external knowledge on ConFiQA-MR (EM). KBevo maintains strong performance under counterfactual knowledge edits, showing that its structured KB can be directly updated and effectively used for reasoning without model retraining.

Panel of Figure 4 — Figure 4: Co-evolution improves KB coverage and grounded reasoning. (a) GRPO improves answer reachability across hop depths and outperforms static KB construction methods. (b–c) GRPO also improves answer grounding and reasoning correctness over the constructed KB. The analysis is conducted on Qwen3-4B using 1K HotpotQA examples.

Method	HotpotQA ∗	MuSiQue	2Wiki	Avg
(a) Swap KB at inference (reasoning policy fixed)
KBevo -1.7B GRPO †	40.3	19.3	47.3	35.6
– EDC DB (Mistral-7B)	30.2	8.3	40.1	26.2
– AutoSchemaKG DB (Llama-3.1-8B)	29.0	16.1	28.8	24.6
– Gemini DB	38.1	19.0	54.5	37.2
(b) Train with a fixed KB (w/o Co-evolution)

Table 3: Ablating co-evolution. Our learned KB substantially outperforms KBs from static construction methods, while training with a fixed KB generally underperforms joint co-evolution. Results are EM on 1K examples per dataset due to the cost of Gemini-based KB construction.

\dagger

KBevo results are evaluated on this same 1K subset and therefore differ slightly from the full-set results in Table 2 . ∗ marks the in-domain benchmark.

Panel of Figure 5 — Figure 5: Effect of SFT warmup before RL. Starting RL directly from the base model leads to collapse in both reward and lookup usage, while even relatively early SFT checkpoints are sufficient to enter the structured reasoning regime and achieve similar improvement during RL.

Figure 6: A representative example. The model constructs a knowledge base from source documents and then reasons over it to answer a multi-hop question.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Phase	Cost	Recurs at inference?
Training (one-time)
SFT warm-start	6k HotpotQA traces (Gemini 2.5 Flash), 3 epochs	No
GRPO	36 rollouts/question (4 KB + 4 $\times$ 8 QA), 500 steps, $\beta=0$	No
Inference (per query)
KB construction	Built offline once per document, reused across queries	Amortized
Retrieval	top- $k=4$ MiniLM lookup, no LLM call	Per lookup

Appendix

Table 4: Cost breakdown of KBevo . The additional expense is confined to a one-time SFT warm-start and the

K+K\!\times\!M

rollouts during GRPO. At inference the knowledge base is constructed offline once per document and reused across queries, so per-query cost is a single Phase-2 generation plus embedding lookups; no

K\!\times\!M

sampling and no teacher model recur at inference.

Category	SFT	GRPO
Model & Data
Base model	Qwen3-1.7B / Qwen3-4B	SFT checkpoint (step 368 for 1.7B, step 735 for 4B)
Dataset	Gemini 2-phase trajectories (HotpotQA, 6k)	HotpotQA
Train size / Eval size	6k / –	7000 / 100
Use special DB lookup tokens	Yes	Yes
Optimization

Appendix

Table 5: Training configurations for KBevo on multi-hop QA. We first perform supervised fine-tuning (SFT) on two-phase KB-construction and QA trajectories, then continue joint optimization with GRPO.

	Qwen3-1.7B		Qwen3-4B
Metric	SFT	GRPO	SFT	GRPO
KB Structure
$\|E\|$ (unique entities)	156.6	169.4	156.8	186.6
$\|R\|$ (unique relations)	111.3	135.9	109.7	175.3
#Triplets	166.2	225.9	165.7	299.1
Avg Degree	2.15	2.71	2.14	3.30

Appendix

Table 6: Knowledge base structure and lookup efficiency on HotpotQA

\dagger

, averaged per example. Redundancy Rate is the fraction of lookups repeating an entity–relation pair already queried; Unknown Rate is the fraction matching no entry. Co-evolution yields larger, better-connected knowledge bases that the reasoning policy also queries more successfully.

\dagger

These diagnostics were computed using an earlier set of model checkpoints and are retained for reference.

	Triplet Quality			Graph Structure			Answer Reachability
KB Source	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	$\|R\|$	#Comp. $\downarrow$	Giant % $\uparrow$	$\leq$ 2 hop $\uparrow$	$\leq$ 4 hop $\uparrow$
EDC (Mistral-7B)	0.864	0.477	0.600	–	–	–	63.7%	69.5%
AutoSchemaKG (Llama-3.1-8B)	0.888	0.838	0.860	58.8	18.8	51.0%	72.1%	80.8%
KBevo -1.7B GRPO	0.925	0.923	0.924	135.9	10.3	65.1%	81.3%	88.4%
KBevo -4B GRPO	0.938	0.952	0.945	175.3	7.9	73.8%	84.0%	90.4%

Appendix

Table 7: Intrinsic knowledge base quality on HotpotQA, independent of downstream QA. KBevo produces more accurate and better-connected KBs than static construction pipelines, with substantially higher multi-hop answer reachability.

Category	AutoSchemaKG	Qwen3-1.7B		Qwen3-4B
	(Llama-3.1-8B)	SFT	GRPO	SFT	GRPO
Valid Triplets $\uparrow$	35.6%	92.3%	85.0%	96.0%	84.3%
Hallucinated $\downarrow$	11.0%	3.3%	7.5%	1.5%	6.2%
Soundness Issues $\downarrow$	53.4%	4.4%	7.4%	2.6%	9.5%

Appendix

Table 8: Triplet-level correctness on HotpotQA (100 samples), judged with the same LLM-as-judge protocol across all knowledge bases (Appendix B.4 ). Valid Triplets are free of hallucination and soundness issues. GRPO increases both hallucination and soundness-issue rates relative to SFT; we examine whether this explains the downstream gains in Appendix C.2 .

	Post-hoc Filtering (EM)		Faithfulness by Reward Exposure
Model	No Filter	Filter Unfaithful	Used Faithful%	Used Cov.%	Unused Faithful%	Unused Cov.%
KBevo -1.7B SFT	–	–	97.6	5.3	96.8	94.7
KBevo -1.7B GRPO	36.2	38.4	94.3	5.3	92.7	94.7
KBevo -4B SFT	–	–	98.6	5.4	98.2	94.6
KBevo -4B GRPO	41.2	43.6	94.1	5.1	93.2	94.9

Appendix

Table 9: Faithfulness analysis of the constructed knowledge base on HotpotQA (1K examples). Left: removing triplets judged unfaithful slightly improves EM, indicating that hallucinated triplets act primarily as retrieval noise rather than an exploitable shortcut. Right: triplets retrieved during reasoning ( Used ) and never retrieved ( Unused ) show similar faithfulness under the same evaluation protocol (Appendix B.4 ), suggesting that sparse reward exposure does not substantially degrade the unretrieved portion of the knowledge base.

Model	Retrieval Faithfulness	Share%	EM	Fully Grounded	Partially Grounded	Ungrounded
KBevo -1.7B SFT	All faithful	84.0	33.0	49.6	10.6	30.2
KBevo -1.7B SFT	Has hallucinated	16.0	17.5	28.1	9.4	39.4
KBevo -1.7B GRPO	All faithful	60.7	40.3	59.7	11.1	27.1
KBevo -1.7B GRPO	Has hallucinated	39.3	24.2	41.6	10.2	39.5
KBevo -4B SFT	All faithful	90.7	35.0	50.1	8.8	27.2
KBevo -4B SFT	Has hallucinated	9.3	24.7	35.5	7.5	35.5

Appendix

Table 10: Faithfulness-conditioned grounding and QA performance on HotpotQA (1K examples). Reasoning chains are partitioned by whether all retrieved triplets are judged faithful or at least one is hallucinated. GRPO substantially improves both EM and grounding even when reasoning relies entirely on faithful triplets, indicating that the gains are not driven by hallucinated content. The fraction of chains retrieving at least one hallucinated triplet nevertheless increases under GRPO.

Knowledge-Intensive QA

Knowledge-intensive question answering combines language models with external knowledge to answer factual, multi-step, and domain-specific questions.

See all 74 papers

KARLA: Knowledge-base Augmented Retrieval for Language Models
Jun 25, 2026Francois Crespin, Fabian M. Suchanek, Nils HolzenbergerRetrieval-Augmented Generation LLM Grounding
IP Paris, LTCI · Télécom Paris · Institut Polytechnique de Paris, France
Foresight-over-Graph: Reasoning Beyond Local Horizons for Knowledge Base Question Answering
Oct 6, 2026Yang Hong, Yajun Yang, Xin Wang +2Multi-Hop QA LLM Reasoning with Graphs
Tianjin University, Tianjin, China · Beijing Jiaotong University, Beijing, China
KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering
Dec 10, 2025Xin Sun, Zhongqi Chen, Xing Zheng +6LLM Grounding RL for Language Model Reasoning
University of Science and Technology of China, Hefei, China · Independent · Institute of Automation, Chinese Academy of Sciences, Beijing, China.

Jun 25, 2026J/K move · Enter open · S save

KARLA: Knowledge-base Augmented Retrieval for Language Models

Francois Crespin, Fabian M. Suchanek, Nils Holzenberger

IP Paris, LTCI · Télécom Paris · Institut Polytechnique de Paris, France

Retrieval-Augmented Generation LLM Grounding Knowledge Augmentation for Language Models Factual Knowledge in Language Models

Open paper AskPDF arXiv

We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation. This means that (1)~factual knowledge in the LLM output can be updated without retraining the LLM, (2)~facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)~smaller models can achieve the same factual accuracy as larger models. Our core idea is to train the model to produce special tokens that trigger a query to the knowledge base. Our experiments show that our method improves factual grounding in both short and long-form generation, and allows factual revisions to take effect through KB edits rather than parameter updates.

Abstract

Knowledge-Intensive QA

Explore similar work

KARLA: Knowledge-base Augmented Retrieval for Language Models

Foresight-over-Graph: Reasoning Beyond Local Horizons for Knowledge Base Question Answering

KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering