SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents

Shengze Huang; Zihao Tang; Jinluan Yang; Xin Yu; Zhenghao Lin; Xiao Liu; Zengxuan Wen; Yeyun Gong

cs.CLOct 9, 2026

SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents

Authors: Shengze Huang, Zihao Tang, Jinluan Yang, Xin Yu, Zhenghao Lin, Xiao Liu, Zengxuan Wen, Yeyun Gong,

Organizations: Zhejiang University · Microsoft

Abstract

Reinforcement learning (RL) for search agents often suffers from evidence fabrication, a failure mode where agents bypass external information ( $E_{ext}$ ) and fabricate an incorrect answer. We argue that this fabrication arises from an over-reliance on unreliable internal knowledge ( $E_{int}$ ) as shortcuts due to the inherent opacity of reasoning trajectories. To address this, we propose SOAR (SOurce-Aware Reinforcement learning), a paradigm that transforms monolithic reasoning into source-explicit trajectories. SOAR (i) disentangles $E_{ext}$ and $E_{int}$ into parallel, observable streams via a Source-Aware Workflow, and (ii) incorporates Source-Consistent Reward to guide the policy evolution. By making information-source attribution explicit, SOAR enables precise credit assignment between different knowledge streams. Evaluations show that SOAR outperforms baselines on 3B and 7B backbones by 7.3% and 5.4%, while reducing evidence fabrication by 74.8%. These confirm that SOAR significantly improves performance on knowledge-intensive benchmarks while effectively mitigating evidence fabrication, fostering more faithful and grounded reasoning with enhanced robustness.

Figures & tables

Figure 1: Comparison of reasoning trajectories and evidence fabrication rates. (a) Previous methods often fall victim to evidence fabrication when initial retrieval results are incomplete, relying on hallucinated internal priors to fill gaps. (b) Our method (SOAR) implements a source-aware adjudication mechanism that monitors the alignment between internal knowledge (

\mathcal{E}_{int}

) and external evidence (

\mathcal{E}_{ext}

). By realizing the unreliability of internal information, the model initiates targeted retrieval to ensure a faithful reasoning trajectory. (c) Quantitative analysis shows SOAR reduces the evidence fabrication rate to 4.2%, outperforming existing baselines.

Figure 2: The training framework of SOAR. (a) Overall workflow: for a given question

q

, the Actor Model interacts with a search engine and its internal knowledge base to generate a group of diverse reasoning trajectories

\{o_{1},o_{2},\dots,o_{g}\}

. These trajectories are evaluated by a reference model and rule-based rewards to compute the Group Advantage for reinforcement learning. (b) Fine-grained information selection: each trajectory

o_{g}

represents a unique reasoning path derived from different selections and integration of

\mathcal{E}_{int}

and

\mathcal{E}_{ext}

. By contrasting these trajectories, the Group Advantage facilitates precise credit assignment to specific information selection behaviors, guiding the agent to learn optimal strategies for grounding its final answer.

Task Instruction:

Determine whether a model’s reasoning is grounded in the provided information. Only judge whether the reasoning steps or the final answer actually depend on facts stated in the <information> block. Do NOT evaluate whether the final answer is correct.

Output Format:

{

"reason": "...",

"grounded": true/false

}

Table 2: System prompt for Evidence Fabrication Judge.

Figure 3: Analysis of model behaviors during training and inference. (a) Mean reward convergence of SOAR and Search-R1 over training steps. (b) Search frequency dynamics during training: SOAR adaptively reduces evidence fabrication by shifting from internal shortcuts to proactive external search, as clearly reflected in the rising and stabilizing external retriever call rate. (c) Search frequency across benchmarks: the search-proactive policy learned during training generalizes consistently and robustly across diverse tasks.

Method	HotpotQA	2Wiki	Musique	Bamboogle	Avg
	Multi-Hop QA
SOAR	0.378	0.419	0.187	0.346	0.333
w/o Source-Consistent Reward	0.364	0.388	0.192	0.333	0.319
w/o Workflow & Reward	0.359	0.369	0.160	0.289	0.294

Table 3: Ablation study on SOAR components. The “w/o Workflow & Reward” row ablates both components. Removing the Source-Consistent Reward alone degrades performance; removing the Source-Aware Workflow additionally (i.e., removing both) leads to a further degradation. This confirms that both components are necessary.

	Answer-Correct		Answer-Wrong		Total
Method	Rate	Reduct.	Rate	Reduct.	Rate	Reduct.
Search-R1	7.57%	—	24.14%	—	16.08%	—
w/o Source-Consistent Reward	2.75%	$\downarrow$ 63.67%	10.79%	$\downarrow$ 55.30%	6.75%	$\downarrow$ 58.02%
SOAR	1.34%	$\downarrow$ 82.30%	6.92%	$\downarrow$ 71.33%	4.05%	$\downarrow$ 74.81%

Table 4: Ablation of reward learning on evidence fabrication rate across different answer groups.

Method	Single-Hop	Multi-Hop	Average
SOAR (Ours)	0.545	0.375	0.448
w/ disturb	0.543 ( $\downarrow$ 0.002)	0.345 ( $\downarrow$ 0.030)	0.429 ( $\downarrow$ 0.019)
Search-R1	0.515	0.342	0.415
w/ disturb	0.490 ( $\downarrow$ 0.025)	0.290 ( $\downarrow$ 0.052)	0.376 ( $\downarrow$ 0.039)
AutoRefine	0.522	0.352	0.425
w/ disturb	0.510 ( $\downarrow$ 0.012)	0.305 ( $\downarrow$ 0.047)	0.392 ( $\downarrow$ 0.033)

Table 5: QA Stability in Corrupted Training.

Method	Default	w/ Disturb	Increase
Search-R1	16.08%	18.23%	+2.15%
IKEA	13.19%	14.98%	+1.79%
AutoRefine	11.45%	12.96%	+1.51%
SOAR (Ours)	4.05%	4.38%	+0.33%

Table 6: Evidence Fabrication in Corrupted Training.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg
w/ EM check	$0.4642{\scriptstyle\pm 0.0056}$	$\mathbf{0.6370{\scriptstyle\pm 0.0035}}$	$0.4740{\scriptstyle\pm 0.0043}$	$\mathbf{0.3778{\scriptstyle\pm 0.0039}}$	$\mathbf{0.4168{\scriptstyle\pm 0.0053}}$	$0.1845{\scriptstyle\pm 0.0048}$	$0.3562{\scriptstyle\pm 0.0073}$	$0.4160{\scriptstyle\pm 0.0016}$
w/ Claude Opus 4.8 judge	$\mathbf{0.4698{\scriptstyle\pm 0.0021}}$	$0.6348{\scriptstyle\pm 0.0046}$	$\mathbf{0.4798{\scriptstyle\pm 0.0046}}$	$0.3692{\scriptstyle\pm 0.0036}$	$0.4142{\scriptstyle\pm 0.0029}$	$\mathbf{0.1917{\scriptstyle\pm 0.0042}}$	$\mathbf{0.3705{\scriptstyle\pm 0.0091}}$	$\mathbf{0.4187{\scriptstyle\pm 0.0019}}$

Appendix

Table 7: Ablation of the Source-Consistency Match Reward: exact-match (EM) check vs. LLM judge (Claude Opus 4.8). Results are reported as mean

\pm

std over 4 runs.

Phase	Dataset	Number of Examples
Train	HotpotQA	90,447
	Natural Questions (NQ)	79,168
	Total Train	169,615
Test	PopQA	14,267
	2WikiMultihopQA	12,576
	TriviaQA	11,313

Appendix

Table 8: Detailed sample statistics and configurations of the datasets used for training and evaluation phases.

Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg
	Single-Hop QA			Multi-Hop QA
Qwen2.5-3B-Instruct
Search-R1-v0.3	0.350	0.572	0.395	0.301	0.318	0.112	0.228	0.325
AutoRefine	0.424	0.587	0.400	0.348	0.364	0.144	0.320	0.369
AdaSearch	0.379	0.568	0.428	0.334	0.385	0.144	0.280	0.360
IKEA	0.444	0.632	0.465	0.329	0.347	0.125	0.276	0.375

Appendix

Table 9: Performance comparison on the Qwen2.5-3B-Instruct backbone. Bold denotes the best results.

	ASQA	WebWalkerQA
Method	F1	Easy	Mid	Hard	Avg
Search-R1	66.5	27.5	20.0	17.9	21.1
SOAR	68.5	29.4	26.1	21.3	25.2

Appendix

Table 10: Zero-shot performance comparison on challenging long-form QA benchmarks using the Qwen2.5-7B backbone. Evaluation metrics include F1 score for ASQA and Accuracy judged by GPT-4o-mini for WebWalkerQA. Bold text indicates the best performance.

Method	Acc	Token Cost
Search-R1	0.372	408.4
AutoRefine	0.386	415.9
CoDA	0.365	745.2
IKEA	0.382	396.7
SOAR	0.414	431.0

Appendix

Table 11: Performance and efficiency trade-off on 3B backbone. Acc denotes the average score across the 7 main benchmarks. Token Cost refers to the average output tokens per reasoning turn, encompassing both information and thinking content.

Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg
Search-R1	$0.3902{\scriptstyle\pm 0.0031}$	$0.5940{\scriptstyle\pm 0.0029}$	$0.4412{\scriptstyle\pm 0.0041}$	$0.3582{\scriptstyle\pm 0.0017}$	$0.3715{\scriptstyle\pm 0.0031}$	$0.1590{\scriptstyle\pm 0.0014}$	$0.2850{\scriptstyle\pm 0.0086}$	$0.3713{\scriptstyle\pm 0.0021}$
SOAR	$\mathbf{0.4698{\scriptstyle\pm 0.0021}}$	$\mathbf{0.6348{\scriptstyle\pm 0.0046}}$	$\mathbf{0.4798{\scriptstyle\pm 0.0046}}$	$\mathbf{0.3692{\scriptstyle\pm 0.0036}}$	$\mathbf{0.4142{\scriptstyle\pm 0.0029}}$	$\mathbf{0.1917{\scriptstyle\pm 0.0042}}$	$\mathbf{0.3705{\scriptstyle\pm 0.0091}}$	$\mathbf{0.4187{\scriptstyle\pm 0.0019}}$

Appendix

Table 12: Performance of Search-R1 and SOAR under four random seeds (temperature

=1.0

) on the Qwen2.5-3B backbone. Results are reported as mean

\pm

std over 4 runs. Bold denotes the best results.

You are an expert assistant capable of solving knowledge-intensive tasks efficiently. You will be given a question to answer as accurately as possible by carefully balancing INTERNAL information and EXTERNAL information.

If you find you lack some knowledge, you can call an information engine by <search> query </search> . It will return the top internal and external information.

To arrive at the answer, you will proceed step-by-step in a structured cycle of <think> , <search> , <information> , and <information_source> sequences. At end, you should use <answer> to provide answer. You can only generate content within these special tags.

Thinking Phase ( <think> ): For question, it may be decomposed into sub-questions for you to think about. Some sub-questions may be answered by searching, while others may not. You can also use the <think> tag to express your uncertainty about the sub-question, weigh the accuracy and usefulness of both types of information, and decide whether further search is needed.

Searching Phase ( <search> ): Formulate a search query only if required to fill a knowledge gap or verify uncertainty.

Information Phase ( <information> ): Use internal and external results as context for further steps.

Information Source ( <information_source> ): Before providing your final answer, indicate clearly whether you are using your own internal knowledge or external information. Only output either “internal” or “external” inside this tag, nothing else.

Appendix

Table 13: System prompt used for Agent RL training.

You are a helpful assistant.

Your task is to list knowledge relevant to the question using ONLY your internal knowledge. Do NOT invent, guess, speculate, or hallucinate any facts. If your internal knowledge is limited, state only what is confidently known.

You must STRICTLY follow all rules below:

1) Your output MUST contain exactly {top-

k

} items.

2) Each item MUST be a single declarative sentence.

3) Each item MUST be formatted as a numbered bullet.

4) Output ONLY the {top-

k

} numbered bullets and nothing else.

Appendix

Table 14: System prompt used for Internal Knowledge Extraction.

Question: Where was the Louisiana Purchase signed in 1803?
Search-r1 ( Success )
<think> To answer this question, I first need to find the location where the Louisiana Purchase was signed in 1803. </think>
<search> where was the Louisiana Purchase signed in 1803 </search>
<information>
Doc1: Background of the purchase ( the U.S. acquired the Louisiana territory from France in 1803 for about $15 million. )
Doc2: States the treaty was signed on April 30, 1803, and that documents arrived in Washington D.C. on July 14. ( no mention of the signing location. )

Appendix

Table 15: Case study illustrating spurious correctness caused by evidence fabrication. Search-r1 reaches the correct answer despite failing to retrieve the key supporting evidence. Instead of grounding the prediction in retrieved information, the model fills the missing evidence using internal knowledge, producing a correct answer by chance. This hidden retrieval failure may introduce misleading reward signals during reinforcement learning optimization.

Question: Which film was released earlier, Ek Ladka Ek Ladki or The Original Latin Kings Of Comedy ?
Search-r1 ( Failure )
<think> First, I need to determine the release dates of both films. </think>
<search> which film was released earlier, Ek Ladka Ek Ladki or The Original Latin Kings Of Comedy </search>
<information>
… Ek Ladka Ek Ladki is an Indian Hindi-language romantic comedy film released on June 19, 1992…
(No information about The Original Latin Kings Of Comedy)

Appendix

Table 16: Case study comparing Search-r1 and SOAR on a multi-hop question. Search-r1 prematurely relies on hallucinated internal knowledge, leading to insufficient evidence retrieval and an incorrect answer. In contrast, SOAR explicitly separates internal and external knowledge sources, detects source inconsistency, and triggers deeper evidence retrieval, enabling accurate reasoning and answer generation.

Claim Attribution in Language Models

See all 13 papers

Contextual Information Policy Optimization for Search Agents
Aug 6, 2026Xingyu Guo, Wei Chen, Linlin Yang +1Evidence-Grounded Reasoning RL for Language Model Reasoning
National College for Excellent Engineers, Beihang University, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China
SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
Mar 30, 2026Yifan Wang, Bolian Li, David Cho +3RL for Language Model Reasoning Reinforcement Learning
Purdue University · Texas Instruments
Dense Process Supervision for Search Agents via Fact Utility Estimation
Sep 1, 2026Rongzhi Zhu, Xiangyu Liu, Yi Liu +7Reinforcement Learning Credit Assignment in RL
State Key Laboratory for Novel Software Technology, Nanjing University, China · Ant Group, China · National Institute of Healthcare Data Science, Nanjing University, China

Aug 6, 2026J/K move · Enter open · S save

Contextual Information Policy Optimization for Search Agents

Xingyu Guo, Wei Chen, Linlin Yang, Baochang Zhang

National College for Excellent Engineers, Beihang University, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China

Evidence-Grounded Reasoning RL for Language Model Reasoning Agentic RAG

Open paper AskPDF arXiv

Search agents extend large language models beyond static parametric memory by enabling them to acquire and use external evidence during multi-step reasoning. For knowledge-intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant evidence but also on using it to guide subsequent reasoning. However, existing methods primarily reward final-answer correctness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence. This misalignment encourages prior-driven reasoning: agents form conclusions based on internal knowledge and use retrieval mainly to confirm them, resulting in confirmation bias and inefficient evidence use. To address this issue, we propose Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework that explicitly aligns policy optimization with external evidence use. CIPO assigns dense, turn-level credit to reasoning actions influenced by retrieved information, while combining this evidence-use signal with a global outcome reward to preserve answer correctness. With this manner, CIPO discourages evidence-detached guesses and promotes reasoning trajectories in which retrieved facts can guide or revise subsequent reasoning. Importantly, CIPO requires neither human process annotations nor an additional reward model. Extensive experiments on seven in-domain and out-of-domain benchmarks show that CIPO reduces the prevalence of prior-driven reasoning and achieves excellent performance on most tasks.

Abstract

Claim Attribution in Language Models

Explore similar work

Contextual Information Policy Optimization for Search Agents

SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology

Dense Process Supervision for Search Agents via Fact Utility Estimation