Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks.
Figures & tables
Figure 1: CW-GRPO assesses the quality of each search round within successful trajectories and to concentrate the learning advantage on high-quality search rounds, thereby enabling round-level credit assignment.
Figure 2: CW-GRPO employs an LLM judge to assess both the retrieval utility and the reasoning correctness at each individual search round. Based on these two signals, it computes a contribution factor for every search round, which is subsequently applied to the original advantage obtained from group-relative comparisons, thereby producing the final round advantage. As a result, training signals from highly contributive rounds are amplified, while those from low-contribution rounds are attenuated.
Parameter
GRPO / CW-GRPO
PPO-based (Actor)
PPO-based (Critic)
Learning Rate
1×10−6
1×10−6
1×10−5
Warmup Ratio
0.285
0.285
0.015
Training Steps
200
200
200
Batch Size
32
128
128
Group Size
4
1
1
GAE γ/λ
-
1.0/1.0
1.0/1.0
Table 1: Detailed training hyperparameters of CW-GRPO and baselines.
Method
General QA
Multi-Hop QA
Overall
NQ †
TriviaQA *
PopQA *
HotpotQA †
2wiki *
Musique *
Bamboogle *
Proprietary Models
GPT-5.1
31.0
60.0
38.5
36.0
43.5
13.8
39.0
34.44
Gemini-3-Pro-Preview
31.0
66.0
32.5
42.0
38.5
14.0
42.0
35.00
Qwen3-Max
31.5
62.5
30.5
39.5
49.0
18.5
42.5
36.56
Open-sourced Models
Table 2: Avg@4 Exact Match on seven subsets of AgentGym-SearchQA-test. † / * indicate in-domain and out-of-domain datasets. CW-GRPO achieves the best performance among methods built on the same backbone.
α
General QA
Multi-Hop QA
Overall
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
0
33.5
57.0
26.0
33.5
32.5
12.5
31.5
29.88
1
27.0
58.0
28.0
32.5
35.0
12.8
31.5
29.69
3
33.0
58.5
31.0
29.5
36.0
13.0
31.0
30.63
5
27.0
58.5
27.0
35.0
39.0
12.0
33.5
30.50
∞
35.5
57.5
28.5
33.5
33.0
13.0
37.0
31.38
Table 3: Effect of the sharpness parameter α on search agent performance. Larger α value generally yields better overall accuracy, indicating that concentrating learning signals on high-contribution search rounds is beneficial.
Method
General QA
Multi-Hop QA
Overall
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
CW-GRPO
35.5
57.5
28.5
33.5
33.0
13.0
37.0
31.38
wo-retrieval
27.0
53.5
26.0
31.0
37.5
12.0
34.5
29.19
wo-reasoning
23.5
55.5
31.0
34.0
28.5
12.3
31.5
28.56
Table 4: Ablation on contribution supervision in CW-GRPO. Removing either retrieval utility or reasoning correctness supervision degrades performance, showing both are necessary for effective credit assignment.
Figure 3: Training curves of CW-GRPO on Qwen3-8B. We report the exponential moving averages of Exact Match, retrieval utility, and reasoning correctness over training steps.
Figure 4: Case study of CW-GRPO on Qwen3-8B.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9
Figure 5: The prompt for LLM Judge to specify the rubric for assessing each search round in terms of retrieval utility and reasoning correctness.
Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.
Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Existing search-augmented LLM agents are trained using Reinforcement Learning to boost its reasoning capabilities. However, these approaches primarily rely on outcome-level rewards, which provide little supervision over search behavior and overlook agent's ability to decompose complex queries properly. To mitigate this issue, we propose PROGRESS which utilizes teacher-guided coverage reward to explicitly shape decomposed query generation of the policy model. During training, frozen teacher models are used to decompose complex queries into essential search queries. These essential search queries are utilized to guide the search behavior of the policy model. Integrated into an R1-style training framework, our approach provides lightweight guidance over query decomposition decisions without dense process-level supervision. Experiments show that coverage-guided RL improves overall task performance, highlighting the importance of explicitly supervising search strategies in agentic LLMs.
Reinforcement learning (RL) improves large language model (LLM) agents on long-horizon search tasks that require multiple intermediate decisions before a final outcome. However, rollout budgets are often allocated without assessing intermediate-state utility, which can waste computation on unpromising branches. We propose Information Gain-based Rollout Policy Optimization (IGRPO), a framework that organizes rollout collection around intermediate-state informativeness. Specifically, IGRPO performs budget-aware tree-structured rollouts in which expansion probabilities depend on node-level informativeness, allowing informative branches to receive more computation while less informative branches are expanded less frequently within a fixed rollout budget. By directly shaping how training trajectories are generated, IGRPO induces a limiting teacher distribution over search trajectories that favors higher cumulative informativeness. The resulting distribution provides an explicit policy optimization target, connecting adaptive rollout collection with principled policy learning. Experiments on seven search-augmented question answering benchmarks show that IGRPO achieves higher average accuracy than strong baselines on both 3B and 7B backbones under comparable rollout budgets, supporting informativeness-guided trajectory generation for training search agents. Code is available at https://github.com/e3trange/IGRPO.