Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers

Zongyuan Sun; Shuai Zhao; Yingjun Du; Ke Shu; Ziyue Zhu; Haoran Luo

cs.AIOct 27, 2026

Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers

Authors: Zongyuan Sun, Shuai Zhao, Yingjun Du, Ke Shu, Ziyue Zhu, Haoran Luo

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · University of Amsterdam, Amsterdam, Netherlands · University of Helsinki, Helsinki, Finland · Nanyang Technological University, Singapore

Abstract

Large language models (LLMs) can answer many factual questions correctly, but a correct answer is not necessarily a supported one. For an LLM agent, a knowledge graph (KG) plays two roles: as a source of answers, it supplies facts that the model lacks; as a source of evidence, it supplies the supporting triples that make an answer traceable and auditable. However, existing KG agents treat the KG only as a source of answers: their training data carries answers but not evidence, and their training rewards check the answer but not the retrieved evidence, so agents often answer correctly without retrieving the supporting triples. We propose FoG, a framework that trains KG agents to retrieve the evidence behind their answers. FoG consists of evidence-grounded synthesis, a KG interaction agent, and an evidence-coverage process reward (ECPR): the synthesis recursively hides explicit entities behind verifiable sub-questions, so that every question carries its supporting triples by construction; the agent records the triples returned by the KG; and ECPR credits each supporting triple to the turn that first retrieves it. Experiments on three KGQA benchmarks demonstrate that FoG enables 3B--4B open-source agents to outperform substantially larger baselines in answer accuracy and evidence coverage, with zero-shot cross-KG transfer.

Figures & tables

Figure 1: Motivation of FoG. Answer-only training yields correct but unsupported answers, whereas FoG embeds supporting triples at synthesis and rewards their retrieval during RL training.

Figure 2: Comparison of KG-augmented reasoning paradigms. All three methods return the correct answer, but retrieval-based methods miss key evidence and agent-based methods answer after the first hop, whereas FoG retrieves every supporting triple of the answer.

Figure 3: The FoG pipeline. Evidence-grounded synthesis produces complex questions with gold supporting triples, the KG interaction agent records the triples that it retrieves, and ECPR rewards the retrieval of the gold supporting triples.

Category	Methods	Backbone	WikiWebQuestions		QALD10-en		GTSQA
Category	Methods	Backbone	F1	Hit@1	F1	Hit@1	F1	Hit@1
LLM Reasoning	IO Prompt Team (2025b)	Qwen2.5-7B-Instruct	28.24	44.43	11.05	12.30	9.62	10.23
	IO Prompt OpenAI (2023)	GPT-4o-mini	40.31	68.10	15.34	18.78	15.37	16.46
	CoT Wei et al. (2022)	GPT-4o-mini	41.66	58.42	22.68	26.81	18.87	20.12
	Self-Consistency Wang et al. (2023)	GPT-4o-mini	46.34	65.20	24.27	28.57	19.07	20.21
	IO Prompt DeepSeek-AI (2024)	DeepSeek-V3	42.36	59.89	18.97	21.77	24.77	26.20

Table 1: Main results on three KGQA benchmarks. Best results are shown in bold , and second-best results are underlined . Our methods are highlighted in blue.

Variants	WikiWebQuestions				GTSQA
Variants	F1	Hit@1	RetQ	Cov	F1	Hit@1	RetQ	Cov
SFT only	59.95 ${}_{\pm\text{0.18}}$	70.10 ${}_{\pm\text{0.24}}$	79.42 ${}_{\pm\text{0.31}}$	70.63 ${}_{\pm\text{0.29}}$	44.73 ${}_{\pm\text{0.22}}$	46.19 ${}_{\pm\text{0.27}}$	58.47 ${}_{\pm\text{0.33}}$	49.85 ${}_{\pm\text{0.30}}$
SFT + RL w/o $R_{\text{ecpr}}$	63.87 ${}_{\pm\text{0.44}}$	74.35 ${}_{\pm\text{0.53}}$	83.17 ${}_{\pm\text{0.49}}$	72.91 ${}_{\pm\text{0.56}}$	46.02 ${}_{\pm\text{0.49}}$	46.97 ${}_{\pm\text{0.56}}$	61.02 ${}_{\pm\text{0.52}}$	51.43 ${}_{\pm\text{0.58}}$
SFT + RL w/o $R_{\text{step}}$	65.02 ${}_{\pm\text{0.33}}$	76.48 ${}_{\pm\text{0.40}}$	85.62 ${}_{\pm\text{0.38}}$	77.25 ${}_{\pm\text{0.42}}$	47.96 ${}_{\pm\text{0.39}}$	48.17 ${}_{\pm\text{0.47}}$	63.18 ${}_{\pm\text{0.44}}$	57.21 ${}_{\pm\text{0.46}}$
SFT + RL, $R_{\text{ecpr}}\!\rightarrow$ LLM-as-judge	64.95 ${}_{\pm\text{0.39}}$	75.41 ${}_{\pm\text{0.51}}$	84.03 ${}_{\pm\text{0.45}}$	74.12 ${}_{\pm\text{0.50}}$	46.88 ${}_{\pm\text{0.43}}$	48.25 ${}_{\pm\text{0.52}}$	62.24 ${}_{\pm\text{0.48}}$	53.36 ${}_{\pm\text{0.51}}$
SFT + RL (full FoG)	66.41 ${}_{\pm\text{0.27}}$	76.93 ${}_{\pm\text{0.38}}$	86.35 ${}_{\pm\text{0.35}}$	78.34 ${}_{\pm\text{0.37}}$	48.66 ${}_{\pm\text{0.34}}$	50.52 ${}_{\pm\text{0.41}}$	65.34 ${}_{\pm\text{0.39}}$	58.64 ${}_{\pm\text{0.40}}$

Table 2: Ablation of training stages and reward terms. The LLM-as-judge variant replaces

R_{\text{ecpr}}

with a model-based process score. Mean and std. over three seeds.

Figure 4: Effect of model scale. F1 and Hit@1 of backbones of different sizes after SFT on the same data.

Data Source	WikiWebQuestions		QALD10-en		GTSQA
Data Source	F1	Hit@1	F1	Hit@1	F1	Hit@1
Train-Only	53.58	64.51	36.94	37.86	41.33	41.52
Synth-Only	57.84	68.17	39.52	40.36	43.21	44.58
Mixed	59.95	70.10	41.38	42.31	44.73	46.19

Table 3: Impact of synthetic training data.

Methods	WikiWebQuestions			GTSQA
Methods	Hit@1	RetQ	Cov	Hit@1	RetQ	Cov
PoG	71.89	75.66	69.85	39.33	35.70	25.38
ToG	72.75	73.53	68.79	42.36	50.99	37.34
KG-R1	73.01	82.28	74.50	47.73	61.41	54.05
FoG-4B	76.93	86.35	78.34	50.52	65.34	58.64

Table 4: Answer accuracy and evidence coverage of agent-based methods.

Figure 5: Performance under structural complexity on GTSQA by reasoning hop and isomorphism type.

Figure 6: Training dynamics of FoG. Rewards rise while interaction turns and response length decline.

Credit assignment	Hit@1	Cov	Dup. rate
Trajectory-level	46.69	54.29	11.8%
Every-match	47.01	52.21	29.4%
First-match (ours)	50.52	58.64	8.6%

Table 5: Credit-assignment strategies for ECPR on GTSQA. Dup. rate is the fraction of KG queries that re-retrieve an already retrieved gold triple.

Method	Train KG	Backbone	WebQSP	WebQSP
Method	Train KG	Backbone	train	Hit@1
ToG Sun et al. (2024)	None	GPT-4	No	82.6
RoG Luo et al. (2024)	Freebase	LLaMA2-Chat-7B	Yes	85.7
KG-R1 Lin et al. (2025)	Freebase	Qwen2.5-3B-Instruct	Yes	82.8
FoG SFT-only	Wikidata	Qwen3-4B-Instruct	No	71.6
FoG full	Wikidata	Qwen3-4B-Instruct	No	77.2

Table 6: Zero-shot cross-KG transfer to Freebase (WebQSP). Methods developed on Freebase are reference points, not matched baselines.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Topic-domain distribution of the synthetic questions. The questions are drawn from major Wikidata domains.

Hops	Isomorphism Type	Questions	Proportion
1 hop	(1)	500	7.59%
	(1)(1)	600	9.11%
	(1)(1)(1)	1,211	18.38%
2 hop	(2)	550	8.35%
	(2)(1)	853	12.95%
	((1)(1))	700	10.63%

Appendix

Table 7: Topology distribution of the synthetic questions. Query-graph isomorphism types are grouped by hop count, the longest path from a topic entity to the answer.

Stage	Input	Retained	Rejected	Teacher calls
Seed-question generation	20,000	14,400	28.0%	20,000
Backward expansion	14,400	8,784	39.0%	86,400
Complex-question composition	8,784	6,588	25.0%	8,784
End-to-end	20,000	6,588	67.1%	115,184

Appendix

Table 8: Synthesis cost and retention.

Category	Function	Description	Input Parameters	Output
Relation search	find_outgoing_relations	Given an entity, return all outgoing relations.	entity_name	List of outgoing relations
	find_incoming_relations	Given an entity, return all incoming relations.	entity_name	List of incoming relations
Neighborhood expansion	get_tail_entities	Follow the relation forward from head to tail.	head_entity_name ; relation_name	Set of tail entities
	get_head_entities	Follow the relation backward from tail to head.	tail_entity_name ; relation_name	Set of head entities
Constraint filtering	get_entity_by_constraint	Filter the provided entity_set and return the entities satisfying the specified constraint.	entity_set ; relation ; operator ; value	Entity subset
Set combination	intersect	Return the intersection of the given list of entity sets.	entity_set_list	Intersection set

Appendix

Table 9: The seven KG operations of the kg-query action. Each operation is listed with its description, input parameters, and output.

Question	Candidate path	Reason for rejection
Who was the prime minister of Pakistan in 1999?	Links candidates only to the office “Prime Minister of Pakistan”.	Does not establish that the candidate held the office in 1999.
What town did Justin Bieber grow up in?	Justin Bieber $\rightarrow$ Canada; Waterloo $\rightarrow$ Canada.	Sharing the same country does not establish childhood residence.

Appendix

Table 10: Representative rejected paths. The verifier checks whether a path covers the semantic constraints of the question, not only whether it reaches a plausible answer entity.

Parameter	Value
Fine-tuning method	LoRA
LoRA rank / alpha / dropout	16 / 32 / 0.05
Max sequence length	3,000
Training epochs	3
Per-device batch size	16
Gradient accumulation steps	8

Appendix

Table 11: Hyperparameters of supervised fine-tuning.

Parameter	Value
Actor learning rate	$1\times 10^{-6}$
KL loss coefficient	0.03 ( $k_{3}$ type)
GRPO rollout $N$	4
Max interaction turns per question	10
Train batch size	32
PPO mini-batch size	16

Appendix

Table 12: Hyperparameters of reinforcement learning.

Reward term	Coefficient	Value
Answer F1 ( $R_{\text{out}}$ )	$\lambda_{\text{f1}}$	0.7
Answer retrieval ( $R_{\text{out}}$ )	$\lambda_{\text{ret}}$	0.3
Step validity ( $R_{\text{step}}$ )	$\lambda_{\text{step}}$	0.5
Evidence coverage ( $R_{\text{ecpr}}$ )	$\lambda_{\text{ecpr}}$	1.0

Appendix

Table 13: Coefficients of the composite reward. Each coefficient weights one term of the reward in Section 3.3 .

Question: Which airline flies the Embraer 195 and is based at the airport named after Margaret Kenyatta ’s father?

<think> I need to find an airline that satisfies two conditions: 1) flies the Embraer 195, and 2) is based at the airport named after Margaret Kenyatta’s father. Let me start by exploring the relationships around the first given entity to understand what information is stored about Margaret Kenyatta in the knowledge graph.</think>

<kg-query> find_outgoing_relations(entity_name=Margaret Kenyatta) </kg-query>

<information> Outgoing relations for entity Margaret Kenyatta: country of citizenship, father, given name, instance of, mother, occupation, sex or gender……… </information>

<think> The outgoing triple shows that Margaret Kenyatta has a "father" in the knowledge graph. Let me find out who her father is by querying the triple that directly connects Margaret Kenyatta to her father. </think>

<kg-query> get_tail_entities(head_entity_name=Margaret Kenyatta, relation_name=father) </kg-query>

<information> Entities outgoing via relation father from entity Margaret Kenyatta: Jomo Kenyatta………. </information>

Appendix

Table 14: Case study of FoG. The trajectory recovers the complete gold evidence before answering.

Question: Which airline flies the Embraer 195 and is based at the airport named after Margaret Kenyatta ’s father?

Reasoning chains: Turn 1: ‘Margaret Kenyatta -> father -> Jomo Kenyatta‘; ‘Margaret Kenyatta -> country of citizenship -> Kenya‘; ‘Embraer 195 -> subclass of -> E-Jet‘; ‘Margaret Kenyatta -> occupation -> diplomat‘; ‘Margaret Kenyatta -> occupation -> politician‘; Turn 2: ‘Kenya -> named after -> Mount Kenya‘; ‘diplomat -> named after -> Diplomatstaden‘; ‘E-Jet -> manufacturer -> Embraer‘; ‘politician -> category combines topics -> Category:Dutch politicians‘; ‘Jomo Kenyatta -> country of citizenship -> Kenya‘; Turn 3: ‘Mount Kenya -> country -> Kenya‘; ‘Embraer -> headquarters location -> São José dos Campos‘; ‘Category:Dutch politicians -> is a list of -> human‘; ‘Kenya -> named after -> Mount Kenya‘; ‘Diplomatstaden -> located in the administrative territorial entity -> Stockholm Municipality‘; Turn 4: ‘Kenya -> named after -> Mount Kenya‘; ‘Mount Kenya -> country -> Kenya‘; ‘Kenya -> capital -> Nairobi‘; ‘São José dos Campos -> country -> Brazil‘; ‘São José dos Campos -> located in the administrative territorial entity -> São Paulo‘;

<answer> First, the airline that flies the Embraer 195 is Kenya Airways. Second, the airport named after Margaret Kenyatta’s father is Jomo Kenyatta International Airport. The answer is {Kenya Airways} </answer>

Appendix

Table 15: Case study of ToG. The trajectory reaches the correct answer while covering only the first part of the gold evidence.

Question: Which airline flies the Embraer 195 and is based at the airport named after Margaret Kenyatta ’s father?

Reasoning chains: Turn 1: Embraer 195 -> item operated -> Kenya Airways;

<answer> Kenya Airways operates the Embraer 195 and is based at Jomo Kenyatta International Airport, which is named after Margaret Kenyatta’s father. The answer is {Kenya Airways}</answer>

Appendix

Table 16: Case study of PoG. The trajectory reaches the correct answer through a shortcut retrieval and misses the bridge evidence.

Knowledge Graph Question Answering

Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.

See all 169 papers

AgentKGV: Agentic LLM-RAG Framework with Two-Stage Training for the Fact Verification of Knowledge Graphs
Jul 10, 2026Yumin Heo, Hyeon-gu Lee, Sumin Seo +1Agentic RAG LLM Agent Training
SungKyunKwan University · NAVER
Foresight-over-Graph: Reasoning Beyond Local Horizons for Knowledge Base Question Answering
Oct 6, 2026Yang Hong, Yajun Yang, Xin Wang +2Multi-Hop QA LLM Reasoning with Graphs
Tianjin University, Tianjin, China · Beijing Jiaotong University, Beijing, China
UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG
Jan 28, 2026Dobrik Georgiev, Kheeran K. Naidu, Alberto Cattaneo +3Multi-Hop QA LLM Reasoning with Graphs
Graphcore Research

Jul 10, 2026J/K move · Enter open · S save

AgentKGV: Agentic LLM-RAG Framework with Two-Stage Training for the Fact Verification of Knowledge Graphs

Yumin Heo, Hyeon-gu Lee, Sumin Seo, Youngjoong Ko

SungKyunKwan University · NAVER

Agentic RAG LLM Agent Training Knowledge Graphs

Open paper AskPDF arXiv

Knowledge graphs (KGs) are often automatically constructed from large-scale corpora, but they inevitably contain factual errors due to noisy sources and extraction failures, and verifying them reliably at industrial scale remains a critical challenge. To address this, we propose AgentKGV, the Agentic LLM-RAG framework for KG fact Verification, that integrates dynamic routing and iterative query rewriting, which handles surface-form mismatch in document-level retrieval. To make this framework more accurate and cost-efficient for industrial deployment, we further introduce a two-stage training strategy: turn-level distillation-based SFT that transfers reasoning ability from a large teacher model into a small model for stable query rewriting and reasoning, and trajectory-level GRPO that optimizes the search policy to reduce unnecessary retrieval at scale. On the long-tail-predicate split of the open-domain T-REx benchmark, our framework improves macro-F1 over single-turn RAG by 5.5 %p, and two-stage training does it further by 9.4 %p. GRPO also cuts the average number of search calls from 3.24 to 1.63 without lowering accuracy.

Abstract

Knowledge Graph Question Answering

Explore similar work

AgentKGV: Agentic LLM-RAG Framework with Two-Stage Training for the Fact Verification of Knowledge Graphs

Foresight-over-Graph: Reasoning Beyond Local Horizons for Knowledge Base Question Answering

UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG