OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Search

Erhan Zhang; Yiqun Chen; Zechun Niu; Wei Yang; Xiaochi Wei; Yan Gao; Yi Wu; Yao Hu

cs.AIApr 4, 2026

OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Search

Authors: Erhan Zhang, Yiqun Chen, Zechun Niu, Wei Yang, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, +1 more

Organizations: Renmin University of China · University of Southern California · Xiaohongshu Inc.

Abstract

Agentic search enables language models to solve knowledge-intensive tasks by adaptively acquiring external evidence over multiple steps. Reinforcement learning with verifiable rewards (RLVR) has emerged as a widely adopted training paradigm for search agents, yet outcome-only rewards are sparse and provide limited credit assignment for intermediate search actions. Existing process-reward methods therefore seek to densify supervision through proxy signals, external evaluators, or likelihood-based information gain. However, proxy rewards can deviate from the final outcome objective, while fixed evaluators can become stale as the search policy evolves, leading to unreliable process supervision. To address these challenges, we propose OASES, an Outcome-Aligned Search-Evaluation Supervision framework for agentic search. OASES derives outcome-aligned process rewards by evaluating how well each intermediate search state supports answering the original question. It further co-trains the search policy and the state evaluator on policy, allowing the evaluator to adapt to evolving search behavior and provide more reliable process rewards. Experiments on five multi-hop QA benchmarks show that OASES consistently outperforms strong RL baselines, with further analyses confirming the benefits of outcome-aligned process rewards and search-evaluation co-training.

Figures & tables

Figure 1: Comparison of reward designs for RLVR-based agentic search. Outcome-only RLVR provides trajectory-level feedback. Existing process-reward methods add step-level rewards from a separate evaluator without evaluation training. OASES co-trains a single policy for search and state evaluation, producing outcome-aligned process rewards while reducing evaluator–policy mismatch.

Figure 2: Overview of the OASES framework. The policy generates a main search trajectory $\tau^{\text{search}}$ and extracts an ordered sequence of search states $\{s_{t}\}_{t=0}^{T}$ . For each state, the same policy generates a state-conditioned evaluation answer $\tilde{y}_{t}$ , whose score $\gamma_{t}$ reflects how well the accumulated information supports the final answer. OASES derives process rewards from state-score differences $\gamma_{t}-\gamma_{t-1}$ , combines them with the final outcome score $\gamma_{\mathrm{out}}$ , and co-trains search and state evaluation with PPO.

Method	NQ		HotpotQA		2Wiki		Bamboogle		MuSiQue		Avg.
Method	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM
Standard
Direct LLM	17.16	5.98	25.25	17.60	25.80	22.03	14.38	8.00	10.14	2.57	18.55	11.24
Naive RAG [ 13 ]	37.26	18.59	36.69	27.45	27.06	21.98	26.87	16.80	9.32	3.52	27.44	17.67
Agentic Search
Search-o1 [ 14 ]	32.43	15.71	44.86	34.07	39.69	32.71	37.56	27.20	20.11	11.09	34.93	24.16

Table 1: Comprehensive QA benchmark results on Qwen2.5-7B-Instruct. Bold indicates the best performance. Impv. vs Best denotes OASES’s absolute improvement over the strongest baseline.

Panel of Figure 3 — Figure 3: Step-wise analysis of the search-state evaluator under different optimization strategies. Panels (a)–(c) show the evaluator’s answer F1 across training steps for search states at Turn 0, Turn 1, and Turn $\geq 2$ , respectively. Panel (d) reports Impv. vs Best , defined as OASES’s F1 improvement over the stronger non-joint setting at each turn.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	NQ		HotpotQA		2Wiki		Bamboogle		MuSiQue		Avg.
Method	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM
Qwen2.5-1.5B-Instruct
Search-o1 [ 14 ]	1.23	0.69	4.51	3.43	4.82	4.13	4.83	3.20	1.36	0.54	3.35	2.40
Search-R1 [ 9 ]	31.72	15.96	31.24	22.67	25.73	21.60	15.11	6.73	10.40	3.89	22.84	14.17
SubSearch [ 21 ]	17.36	6.81	22.08	15.33	25.72	21.88	28.04	18.40	8.16	2.36	20.27	12.96
OASES (Ours)	37.73	18.75	51.39	38.69	54.41	46.56	46.41	37.60	22.81	12.58	42.55	30.84

Appendix

Table 3: Additional QA benchmark results on smaller Qwen2.5-Instruct models. Bold indicates best performance within each model-size group.

Method	NQ		HotpotQA		2Wiki		Bamboogle		MuSiQue		Avg.
Method	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM
Qwen2.5-3B-Base	20.72	7.45	30.24	19.82	33.82	28.55	29.38	17.60	13.14	2.90	25.46	15.26
Qwen3-4B-Base	20.89	6.93	30.80	19.86	34.36	29.17	17.55	8.00	13.79	2.57	23.48	11.73
Qwen2.5-7B-Base	43.31	22.44	59.79	46.29	60.34	54.60	61.06	46.40	28.99	17.17	50.70	37.38
Qwen3-8B-Base	44.08	22.94	60.79	47.52	61.73	55.66	61.15	49.60	31.92	20.56	51.93	39.26

Appendix

Table 4: QA benchmark results on Qwen2.5 and Qwen3 base models with different model sizes.

Component	Avg. # Rollouts per Query	Avg. Generated Tokens per Rollout	Avg. Generated Tokens per Query	Token Share
Search rollout	1.00	1041.76	1041.76	85.06%
State-evaluation rollouts	2.76	66.31	183.02	14.94%
Total	3.76	–	1224.78	100.00%

Appendix

Table 5: Generated-token cost of search and state-evaluation rollouts after convergence.

Main Search Prompt

Search-state Evaluation Prompts

You are an intelligent Research Agent designed to answer questions by retrieving information from the internet. ## Operational Logic 1. Plan: Analyze the user’s question. If the answer requires multiple steps, break it down. 2. Search: Use the ‘<search>‘ tag to query the search engine. Keep queries specific and efficient. 3. Observe: Wait for the ‘<result>‘ output provided by the system. 4. Iterate: Analyze the search results in a ‘<think>‘ block. - If information is missing, formulate a new search query based on what you found. - If the information is sufficient, proceed to answer. 5. Answer: Output the final answer in an ‘<answer>‘ tag. Keep it concise and direct. ## Response Format - Thinking Process: <think> ... </think> - Search Action: <search> ... </search> - Final Answer: <answer> ... </answer> ## Constraints - Stop after Search: Do not generate ‘<result>‘ tags yourself. Wait for the system. - Search Limit: There is a maximum limit on the number of search rounds. If the limit is reached, stop searching and output the final answer using available information.

Question-Only Evaluation You are an expert AI assistant capable of answering questions precisely. Answer the user’s question based on your internal knowledge. You must provide a concise answer enclosed within specific tags as follows: <eval> The Answer </eval> Context-Aware Evaluation You are an expert AI assistant capable of answering questions precisely. Answer the user’s question using the provided Context and your internal knowledge. You must provide a concise answer enclosed within specific tags as follows: <eval> The Answer </eval>

Appendix

Table 6: Prompt templates used in OASES, including the main search prompt and two search-state evaluation prompts for question-only and context-aware evaluation.

Query: What was the first use of the East Sitting Hall at the official residence and workplace of the President of the United States?
Ground Truth: reception room
Turn	Component	Output / Search-State Evaluation
Turn 0	Eval.	Eval: <eval> Congress held its first full session </eval> Score: $\gamma_{0}=0.000$ (✗ unsupported)
Turn 1	Search	<search> official residence and workplace of the President of the United States </search>
	Result	The official residence and workplace of the U.S. President is the White House .
	Eval.	Eval: <eval> 1792 </eval> Score: $\gamma_{1}=0.000$ (✗ distracted by irrelevant evidence) Source: “Construction took place between 1792 and 1800 using Aquia…”

Appendix

Table 7: Negative case study showing degraded search-state evaluation after search-only training.

Query: When is the director of film The Opposite of Sex ’s birthday?
Ground Truth: April 14, 1955
Turn	Component	Output / Search-State Evaluation
Turn 0	Evaluation of $s_{0}$	Eval Answer: <eval> April 20, 1962 </eval> Score: $\gamma_{0}=0.333$
Turn 1	Think	<thinking> I need to answer this in two steps. First, I must identify the director of the film ‘‘The Opposite of Sex’’. Second, I need to find the birth date of that specific director. </thinking>
	Search	<search> director of film The Opposite of Sex </search>
	Result	<result> The Opposite of Sex is a 1998 American romantic comedy film written and directed by Don Roos and starring Christina Ricci, Martin Donovan and Lisa Kudrow. Don Roos is well known for his work as a screenwriter… </result>

Appendix

Table 8: Positive case study of outcome-aligned process reward in OASES.

Query: Where did the director of film Legally Blonde study?
Ground Truth: California Institute of the Arts
Turn	Component	Output / Search-State Evaluation
Turn 0	Evaluation of $s_{0}$	Eval Answer: <eval> University of Southern California (USC) </eval> Score: $\gamma_{0}=0.444$
Turn 1	Think	<thinking> I need to answer this in two steps. First, I must identify the director of the film ‘‘Legally Blonde’’. Second, I need to find the educational background of that specific director. </thinking>
	Search	<search> director of film Legally Blondes </search>
	Result	<result> Legally Blonde is a 2001 American comedy film based on Amanda Brown’s novel of the same name. It was directed by Robert Luketic , scripted by Karen McCullah Lutz and Kirsten Smith, and stars Reese Witherspoon, Luke Wilson, Selma Blair, Matthew Davis, Victor Garber… </result>

Appendix

Table 9: Negative case study of outcome-aligned process reward in OASES.

Explore similar work

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents
Oct 7, 2026Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2Reward Design for RL RL for IR
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?
May 27, 2026Yibo Zhao, Zichen Ding, Jiayi Wu +2LLM Agents Information Retrieval
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory
Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search
Jul 24, 2026Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang +7Retrieval-Augmented Generation LLM Grounding
Fudan University · Tencent · Nanjing University +2

Oct 7, 2026J/K move · Enter open · S save

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents

Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis, Ruofei Lai+1

University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited

Reward Design for RL RL for IR Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.

Abstract

Explore similar work

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents

Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search