Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks.
Figures & tables
Figure 1: CW-GRPO assesses the quality of each search round within successful trajectories and to concentrate the learning advantage on high-quality search rounds, thereby enabling round-level credit assignment.
Figure 2: CW-GRPO employs an LLM judge to assess both the retrieval utility and the reasoning correctness at each individual search round. Based on these two signals, it computes a contribution factor for every search round, which is subsequently applied to the original advantage obtained from group-relative comparisons, thereby producing the final round advantage. As a result, training signals from highly contributive rounds are amplified, while those from low-contribution rounds are attenuated.
Parameter
GRPO / CW-GRPO
PPO-based (Actor)
PPO-based (Critic)
Learning Rate
1×10−6
1×10−6
1×10−5
Warmup Ratio
0.285
0.285
0.015
Training Steps
200
200
200
Batch Size
32
128
128
Group Size
4
1
1
GAE γ/λ
-
1.0/1.0
1.0/1.0
Table 1: Detailed training hyperparameters of CW-GRPO and baselines.
Method
General QA
Multi-Hop QA
Overall
NQ †
TriviaQA *
PopQA *
HotpotQA †
2wiki *
Musique *
Bamboogle *
Proprietary Models
GPT-5.1
31.0
60.0
38.5
36.0
43.5
13.8
39.0
34.44
Gemini-3-Pro-Preview
31.0
66.0
32.5
42.0
38.5
14.0
42.0
35.00
Qwen3-Max
31.5
62.5
30.5
39.5
49.0
18.5
42.5
36.56
Open-sourced Models
Table 2: Avg@4 Exact Match on seven subsets of AgentGym-SearchQA-test. † / * indicate in-domain and out-of-domain datasets. CW-GRPO achieves the best performance among methods built on the same backbone.
α
General QA
Multi-Hop QA
Overall
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
0
33.5
57.0
26.0
33.5
32.5
12.5
31.5
29.88
1
27.0
58.0
28.0
32.5
35.0
12.8
31.5
29.69
3
33.0
58.5
31.0
29.5
36.0
13.0
31.0
30.63
5
27.0
58.5
27.0
35.0
39.0
12.0
33.5
30.50
∞
35.5
57.5
28.5
33.5
33.0
13.0
37.0
31.38
Table 3: Effect of the sharpness parameter α on search agent performance. Larger α value generally yields better overall accuracy, indicating that concentrating learning signals on high-contribution search rounds is beneficial.
Method
General QA
Multi-Hop QA
Overall
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
CW-GRPO
35.5
57.5
28.5
33.5
33.0
13.0
37.0
31.38
wo-retrieval
27.0
53.5
26.0
31.0
37.5
12.0
34.5
29.19
wo-reasoning
23.5
55.5
31.0
34.0
28.5
12.3
31.5
28.56
Table 4: Ablation on contribution supervision in CW-GRPO. Removing either retrieval utility or reasoning correctness supervision degrades performance, showing both are necessary for effective credit assignment.
Figure 3: Training curves of CW-GRPO on Qwen3-8B. We report the exponential moving averages of Exact Match, retrieval utility, and reasoning correctness over training steps.
Figure 4: Case study of CW-GRPO on Qwen3-8B.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9
Figure 5: The prompt for LLM Judge to specify the rubric for assessing each search round in terms of retrieval utility and reasoning correctness.