Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

Qirui Liu; Hao Chen; Yi Wen; Weijie Shi; Yao Zhao; Xianhe Chen; Hao Zhao; Xing Tang

cs.LGOct 16, 2026

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

Authors: Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang,

Organizations: South China University of Technology · FiT, Tencent · City University of Hong Kong · The Hong Kong University of Science and Technology ·

Abstract

Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it. We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the evidence-supported share of correct answers falls from 84.0% to 42.9%. We attribute this to GRPO's credit assignment: rollouts that retrieve answer-bearing evidence are favoured in aggregate only in proportion to their accuracy gain over the rest, a grounding gap that answering from memory or misusing evidence shrinks or reverses. To address this, we propose Grounded Credit Assignment (GCA), which uses verifiable sub-outcomes (evidence retrieval and answer correctness) to correct search and answer credit separately: a wrong answer is blamed more than the search that found the evidence, and a correct answer that bypasses retrievable evidence is credited less. Never negative on evidence-finding search, the correction does not shrink with the grounding gap; capped at 10% of each token's advantage, it never overrides the outcome signal. Without extra models, rollouts, or training stages, GCA improves average EM over Search-R1 by 5.7 (Qwen2.5-3B) and 6.3 points (Qwen3-30B-A3B) and raises this share from 76.9% to 93.4%.

Figures & tables

Figure 1: Outcome-only RL improves accuracy without improving grounding. (a) Share of correct answers that are supported by retrieved evidence ( SupCor /Acc) on Qwen2.5-3B. Search-R1 raises EM on every benchmark (Table 2 ) but lowers this share below the initial checkpoint (before task RL) on MuSiQue and Bamboogle; GCA restores it. (b) Token-level view of the Search-R1 run (Appendix D.1 ): share of positively credited tokens (type

A

D

rollouts) that are post-hit redundant searches or belong to ungrounded type-

D

rollouts, and share of answer-finding search tokens that lie in hit-but-wrong (type

B

) rollouts. Shares are computed per rollout type and ignore homogeneous groups, whose advantage is zero.

Figure 2: Overview of GCA. (a) Uniform advantage broadcasting conflates search and answer credit within a rollout. (b) GCA diagnoses each rollout by retrieval hit $H$ and answer correctness $C$ , routes signed residuals to functional segments ( vSearch , rSearch , Ans ), and concentrates them on decision-critical tokens via margin–entropy salience. (c) Resulting residuals; signs show residual polarity, not the final advantage’s sign.

Type	Condition	vSearch	rSearch	Ans
$A$	$H{=}1,C{=}1$	$+0.5$	$-0.1$	$+1.0$
$B_{1}$	$H{=}1,C{=}0$ , wrong	$+0.5$	$-0.2$	$-1.0$
$B_{2}$	$H{=}1$ , timeout	$+0.5$	$-0.1$	$0$
$C$	$H{=}0,C{=}0$	–	Base	Base
$D$	$H{=}0,C{=}1,\mathrm{Retr}{=}1$	–	$-0.2$	$-0.1$
$D$	$H{=}0,C{=}1,\mathrm{Retr}{=}0$	Mask

Table 1: Routing coefficients. Entries are signed residual coefficients

P_{T,S}\lambda_{T,S}

; signs follow each segment’s proxy sub-outcome (§ 4.1 ). Shaded: conflict cells, where outcome advantage and sub-outcome disagree in sign. Base : original advantage only; Mask : excluded from the policy loss; –: segment absent.

Qwen2.5-3B-Instruct
Methods	General QA			Multi-Hop QA				Avg.
	NQ †	TriviaQA ⋆	PopQA ⋆	HotpotQA †	2Wiki ⋆	MuSiQue ⋆	Bamboogle ⋆
Initial checkpoint + search	16.15	25.81	16.54	12.37	10.66	4.18	12.00	13.96
IRCoT	11.1	31.2	20.0	16.4	17.1	6.7	24.0	18.1
Rejection Sampling	29.4	48.8	33.2	24.0	23.3	5.9	21.0	26.5
ZeroSearch	41.4	57.4	44.8	27.4	30.0	9.8	11.1	31.7

Table 2: Main results (EM, %) on seven QA benchmarks. † / ⋆ : in-domain/out-of-domain. Best results in bold . Initial checkpoint + search (before task RL) and Search-R1 are run in the same setting as GCA; GCA rows report the mean and

1\sigma

over 3 training seeds.

Backbone	Method	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
Qwen2.5-3B	Initial checkpoint	13.96	43.00	55.27	23.77	75.0%
	Search-R1	37.36	45.79	65.98	30.21	76.9%
	+GCA	43.06	64.68	66.17	42.80	93.4%
Qwen3-30B-A3B	Initial checkpoint	35.75	57.20	69.37	39.68	89.3%
	Search-R1	46.29	65.09	68.05	44.29	92.3%
	+GCA	52.59	71.87	72.60	52.18	94.1%

Table 3: Aggregate grounding metrics (%, averaged over the seven benchmarks). ToolHit

=P(H{=}1)

; HitAcc

=P(\tilde{C}{=}1\mid H{=}1)

with the lenient correctness

\tilde{C}

(§ 5.1 ); SupCor

=

ToolHit

\times

HitAcc estimates the supported-correct rate; SupCor /Acc is its share of the lenient accuracy Acc.

	EM								Grounding
Variant	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	ToolHit	HitAcc	SupCor	SupCor /Acc
Search-R1	47.17	62.72	40.25	36.37	34.94	12.08	28.00	37.36	45.79	65.98	30.21	76.9%
w/o salience	45.93	62.46	43.14	41.98	40.89	16.52	43.13	42.01	57.85	66.89	38.70	87.6%
w/o segment split	44.86	61.38	42.54	40.21	41.52	14.14	37.51	40.31	53.81	64.50	34.71	82.0%
GCA (full)	48.06	63.68	42.94	44.65	44.12	18.74	39.20	43.06	64.68	66.17	42.80	93.4%

Table 4: Budget-matched leave-one-out ablation on Qwen2.5-3B (%, mean over 3 training seeds). W/o salience replaces token salience by its per-cell mean; w/o segment split merges vSearch and rSearch into one search segment. Best results in bold .

Figure 3: Training dynamics on Qwen2.5-3B (§ 5.6 ). Faint lines show per-step values and solid lines an exponential moving average. Top: grounded behaviour ( SupCor , ToolHit , and the hit-but-wrong share of all rollouts, type

B_{1}

); bottom: optimisation diagnostics (mean outcome reward, zero-variance group rate, and actor entropy).

Routing	In-domain	Out-of-domain	Overall
Search-R1	41.77	35.60	37.36
Learned magnitudes	46.41	41.53	42.92
Fixed (Table 1 )	46.36	41.74	43.06

Table 5: Fixed vs. learned routing magnitudes on Qwen2.5-3B (average EM, %, 3 seeds). The learned router keeps the fixed polarities and the residual budget (Appendix H ).

$\lambda$ scale $s$	0.50	0.75	1.00	1.25	1.50
$\kappa$	0.05	0.075	0.10	0.125	0.15
EM (3B)	41.23	42.76	43.06	41.57	36.92
EM (30B)	–	48.01	52.59	46.77	–
Residual scale $\eta$	0.05	0.10	0.20	0.50	1.00
$\kappa$	0.025	0.05	0.10	0.25	0.50
EM (3B)	41.98	43.72	43.06	40.80	37.89

Table 6: Sensitivity of average EM (%) to the

\lambda

scale

s

, the residual scale

\eta

, and the salience cap

c_{\max}

\kappa=\eta\lambda_{\max}c_{\max}

is the worst-case relative residual (Proposition 2 ); shaded: default; –: not run. Search-R1 scores 37.36 (3B) and 46.29 (30B); ToolHit , HitAcc , and SupCor are in Appendix D.4 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$r_{i}$ , $\mathcal{G}(x)$	$i$ -th rollout; group of $n$ rollouts for question $x$
$R_{i}$ , $\hat{A}_{i}$	Outcome reward; GRPO/GSPO group-relative advantage
$\mathcal{A}(x)$ , $m(a,o)$	Gold alias set; case-insensitive containment test
$H_{i}$ , $C_{i}$	Retrieval hit ( ToolHit ); EM answer correctness (Eq. 1 )
$\tilde{C}_{i}$	Lenient evaluation correctness (alias-aware substring match) used by HitAcc , SupCor , and Acc (§ 5.1 )
$k^{\star}_{i}$	First search turn whose tool response contains a gold alias

Appendix

Table 7: Summary of notation.

Cell $(T,S)$	Base	Step 300	Step 600	Step 900
$A\!\cdot\!\textsc{vSearch}$	114,426 (30.68%)	74,815 (36.44%)	100,449 (25.62%)	102,219 (30.07%)
$A\!\cdot\!\textsc{rSearch}$	100,010 (26.81%)	54,214 (26.41%)	116,053 (29.60%)	111,889 (32.91%)
$A\!\cdot\!\textsc{Ans}$	158,538 (42.51%)	76,254 (37.15%)	175,504 (44.77%)	125,836 (37.02%)
$B_{1}\!\cdot\!\textsc{vSearch}$	113,462 (27.97%)	49,766 (29.26%)	69,547 (23.23%)	61,024 (24.59%)
$B_{1}\!\cdot\!\textsc{rSearch}$	102,110 (25.18%)	57,275 (33.67%)	91,709 (30.63%)	97,712 (39.37%)
$B_{1}\!\cdot\!\textsc{Ans}$	190,023 (46.85%)	63,066 (37.07%)	138,163 (46.14%)	89,439 (36.04%)

Appendix

Table 8: Per-cell trainable token counts

n_{T,S}

and within-type share at four checkpoints of the Search-R1 Qwen2.5-3B run. “Trainable” = response tokens minus the eight structural-tag literals. Percentages are within-row ratios

n_{T,S}/\sum_{S^{\prime}}n_{T,S^{\prime}}

for that rollout type at that checkpoint; type

C

has no segment partition.

Figure 4: Within-row segment ratios for type

A

(left) and type

B_{1}

(right) across the four checkpoints. Percentages denote

n_{T,S}/\sum_{S^{\prime}}n_{T,S^{\prime}}

Figure 5: Absolute trainable-token counts for type

B_{2}

(left) and type

D

(right) across the four checkpoints, with stacked segment composition.

Type	Condition	vSearch	rSearch	Ans
$A$	$H{=}1,C{=}1$	$P{=}{+}1,\;\lambda{=}0.5,\;c^{+}$	$P{=}{-}1,\;\lambda{=}0.1,\;c^{-}$	$P{=}{+}1,\;\lambda{=}1.0,\;c^{+}$
$B_{1}$	$H{=}1,C{=}0$ , valid wrong	$P{=}{+}1,\;\lambda{=}0.5,\;c^{+}$	$P{=}{-}1,\;\lambda{=}0.2,\;c^{-}$	$P{=}{-}1,\;\lambda{=}1.0,\;c^{-}$
$B_{2}$	$H{=}1,C{=}0$ , timeout	$P{=}{+}1,\;\lambda{=}0.5,\;c^{+}$	$P{=}{-}1,\;\lambda{=}0.1,\;c^{-}$	$P{=}0,\;\lambda{=}0.0$
$C$	$H{=}0,C{=}0$	N/A	base	base
$D$	$H{=}0,C{=}1$ , Retr $=1$	N/A	$P{=}{-}1,\;\lambda{=}0.2,\;c^{-}$	$P{=}{-}1,\;\lambda{=}0.1,\;c^{-}$
$D$	$H{=}0,C{=}1$ , Retr $=0$	masked	masked	masked

Appendix

Table 9: Routing coefficients

(P_{T,S},\lambda_{T,S})

used in all reported experiments.

c^{\pm}

indicates the salience variant applied (§ E ). Base = original GRPO/GSPO advantage only; Mask = removed from the policy update; N/A = segment absent for this type. We use

\eta\!=\!0.2

and

c_{\max}\!=\!0.5

, giving worst-case residual

\eta\,\lambda_{\max}\,c_{\max}\,|\hat{A}_{i}|=0.1\,|\hat{A}_{i}|

Figure 6: Actor entropy traces of the FM1 and FM2 diagnostic runs on Qwen2.5-3B. FM1 (

\lambda_{B_{1},\textsc{Ans}}\!=\!2.0,\,\eta\!=\!0.5

, no

c_{t}

cap) reaches its peak around step

145

and collapses thereafter; FM2 keeps the same

\lambda

but halves

\eta

0.2

, which only delays the explosion.

Qwen2.5-3B
$s$	EM	ToolHit	HitAcc	SupCor
Search-R1 (no residual)	37.36	45.79	65.98	30.21
$0.50$	41.23	61.80	65.44	40.44
$0.75$	42.76	62.02	66.75	41.40
$\mathbf{1.00}$	43.06	64.68	66.17	42.80
$1.25$	41.57	59.57	67.84	40.41

Appendix

Table 10: Sensitivity to a global

\lambda

scale

s

on Qwen-family backbones.

s\!=\!1

corresponds to Table 9 .

Qwen2.5-3B
$\eta$	EM	ToolHit	HitAcc	SupCor
$0.05$	41.98	63.01	65.76	41.44
$0.10$	43.72	63.61	65.95	41.95
$\mathbf{0.20}$	43.06	64.68	66.17	42.80
$0.50$	40.80	62.61	64.67	40.48
$1.00$	37.89	59.45	63.14	37.53

Appendix

Table 11: Sensitivity to the global residual scale

\eta

on Qwen-family backbones.

Qwen2.5-3B
$c_{\max}$	EM	ToolHit	HitAcc	SupCor
$0.20$	40.83	64.42	62.77	40.43
$0.30$	43.07	64.58	65.12	42.05
$\mathbf{0.50}$	43.06	64.68	66.17	42.80
$1.00$	42.42	63.75	65.69	41.88
Qwen3-30B-A3B-Instruct

Appendix

Table 12: Sensitivity to the per-token cap

c_{\max}

on Qwen-family backbones.

Bucket $(T,S)$	Base		Step 300		Step 900
Bucket $(T,S)$	$\bar{\delta}$ @ $3$ / $\hat{H}$ @ $32$	low- $\delta$ / high- $\hat{H}$	$\bar{\delta}$ @ $3$ / $\hat{H}$ @ $32$	low- $\delta$ / high- $\hat{H}$	$\bar{\delta}$ @ $3$ / $\hat{H}$ @ $32$	low- $\delta$ / high- $\hat{H}$
$A\!\cdot\!\textsc{vSearch}$	6.61 / 0.167	16.1% / 12.0%	10.65 / 0.052	0 5.1% / 0 1.4%	10.95 / 0.059	0 5.9% / 0 2.1%
$A\!\cdot\!\textsc{rSearch}$	6.22 / 0.178	16.7% / 13.7%	0 7.55 / 0.119	11.9% / 0 5.1%	0 6.70 / 0.152	15.0% / 0 8.5%
$A\!\cdot\!\textsc{Ans}$	5.35 / 0.204	19.7% / 15.5%	0 8.73 / 0.091	0 9.1% / 0 3.7%	0 7.44 / 0.139	13.7% / 0 8.8%
$B_{1}\!\cdot\!\textsc{vSearch}$	6.72 / 0.161	15.5% / 11.6%	10.04 / 0.068	0 6.6% / 0 2.6%	0 9.74 / 0.089	0 8.9% / 0 4.4%
$B_{1}\!\cdot\!\textsc{rSearch}$	6.14 / 0.185	17.3% / 14.7%	0 6.58 / 0.154	15.3% / 0 8.8%	0 5.76 / 0.189	18.4% / 12.8%

Appendix

Table 13: Per-(checkpoint, type, segment) margin and entropy on trainable non-structural tokens. “

\bar{\delta}

3

” is the bucket mean of the sampled-token margin against the top-

3

competitors; “

\hat{H}

32

” is the bucket mean of the top-

32

normalised entropy. “low-

\delta

” / “high-

\hat{H}

” denote tail fractions

\Pr[\delta_{t}\!<\!0]

and

\Pr[\hat{H}_{t}\!>\!0.5]

, respectively. Buckets with

P\!=\!0

(Type-

C

, all segments) are listed for reference but contribute no residual.

Figure 7: Density of margin@

3

over trainable non-structural tokens, per (T,S) bucket and checkpoint. Line colour encodes the segment (green: vSearch , red: rSearch , blue: Ans ); line style encodes the rollout type (

A\!:\!{-}

B_{1}\!:\!\text{--}\,\text{--}

B_{2}\!:\!\cdots

D\!:\!{-}\!\cdot\!{-}

). The within-bucket distribution is heavy-tailed and overlaps substantially with

\delta_{t}\!<\!0

at every checkpoint, leaving a non-empty low-margin sub-population for

d_{t}^{+}

to amplify on positive segments and a heavy high-margin sub-population for

d_{t}^{-}

to amplify on negative segments.

Figure 8: Density of entropy@

32

over trainable non-structural tokens, per (T,S) bucket and checkpoint. Encoding matches Fig. 7 . The mass concentrates near

\hat{H}_{t}\!\to\!0

as training progresses (more tokens become high-confidence picks), but a non-empty right tail above

\hat{H}_{t}\!=\!0.5

persists across all active buckets even at step 900 , especially on the failure-mode buckets

B_{1}\!\cdot\!\textsc{Ans}

B_{1}\!\cdot\!\textsc{rSearch}

, and

B_{2}

Margin (vs. $K\!=\!3$ anchor)			Entropy (vs. $M\!=\!32$ anchor)
$K$	$\|\Delta\bar{\delta}\|$ (nats)	min overlap@ $1\%$	$M$	$\|\Delta\hat{H}\|$
$2$	$0.343$	$0.83$	$2$	$0.202$
$\mathbf{3}$	$\mathbf{0.000}$	$\mathbf{1.00}$	$4$	$0.131$
$5$	$0.477$	$0.75$	$8$	$0.077$
$10$	$1.157$	$0.75$	$16$	$0.034$
$32$	$2.316$	$0.75$	$\mathbf{32}$	$\mathbf{0.000}$

Appendix

Table 14: Worst-case cell-level shifts of margin@

K

and entropy@

M

across the

36

EDA cells (3 checkpoints

\times

12 active

(T,S)

buckets). “

|\Delta\bar{\delta}|

” is the maximum absolute shift of the bucket-mean margin (in nats) relative to the

K\!=\!3

anchor; “min overlap@

1\%

” is the minimum overlap of the bottom-

1\%

margin tail against the

K\!=\!3

tail. “

|\Delta\hat{H}|

” is the analogous bucket-mean shift of normalised entropy relative to the

M\!=\!32

anchor. Bucket-mean margin shifts smoothly and monotonically in

K

(a known property of

\log\mathrm{mean}\exp

as more competitors are added), but the active region – the bottom-

1\%

tail – is preserved at

\geq\!0.75

overlap across the entire

\sim\!30\!\times\!

sweep, so the same high-salience tokens absorb the residual regardless of

K

. Bucket-mean entropy shifts are below

0.034

for

M\!\geq\!16

and below

0.077

for

M\!\geq\!8

, putting

M\!=\!32

on a stable plateau with

M\!=\!64

Figure 9: Bucket-conditional margin@

K

curves with p25/p75 bands at base (left), step 300 (middle), step 900 (right). Each panel overlays all active

P\!\neq\!0

(T,S)

buckets; line colour encodes the segment (green: vSearch , red: rSearch , blue: Ans ) and line style encodes the rollout type (

A\!:\!{-}

B_{1}\!:\!\text{--}\,\text{--}

B_{2}\!:\!\cdots

D\!:\!{-}\!\cdot\!{-}

). The

K

-axis sweeps

\{2,3,5,10,32,63\}

on a log scale. Curves shift smoothly and monotonically with

K

\log\mathrm{mean}\exp

property), but the relative ordering of buckets and the within-bucket dispersion (p25–p75 band) are preserved across the full sweep, matching the bottom-

1\%

overlap statistic in Table 14 .

Figure 10: Bucket-conditional entropy@

M

curves with p25/p75 bands at base (left), step 300 (middle), step 900 (right). The

M

-axis sweeps

\{2,4,8,16,32,64\}

on a log scale; legend encoding matches Fig. 9 . Once

M\!\geq\!16

the bucket means are within

0.034

of their

M\!=\!32

anchor and within

0.026

M\!=\!64

across all

36

cells, justifying

M\!=\!32

as a stable plateau choice.

$K$	EM	ToolHit	HitAcc	SupCor
$2$	42.72	64.26	66.34	42.63
$\mathbf{3}$	43.06	64.68	66.17	42.80
$5$	42.78	64.06	66.13	42.36
$32$	43.18	64.96	65.29	42.41

Appendix

Table 15: Sensitivity of GCA to the margin bandwidth

K

on Qwen2.5-3B (fixed

M\!=\!32

). The bold

K\!=\!3

row is the reported configuration. EM deviates by at most

0.34

points from the default

K\!=\!3

setting; the full range across tested settings is

0.46

points. All four metrics remain within

1.0

point of the

K\!=\!3

anchor.

$M$	EM	ToolHit	HitAcc	SupCor
$8$	42.96	64.08	65.79	42.16
$16$	43.10	64.50	66.03	42.58
$\mathbf{32}$	43.06	64.68	66.17	42.80
$64$	42.31	64.10	65.93	42.26

Appendix

Table 16: Sensitivity of GCA to the entropy bandwidth

M

on Qwen2.5-3B (fixed

K\!=\!3

). The bold

M\!=\!32

row is the reported configuration. EM deviates by at most

0.75

points from the default

M\!=\!32

setting (at

M\!=\!64

); the full range across tested settings is

0.79

points. All four metrics remain within

1.0

point of the

M\!=\!32

anchor.

Case	Count	Share
Yes/no gold answer (5,481 in HotpotQA)	5,483	3.23%
More than one alias (all NQ)	7,750	4.57%
Alias of $\leq 4$ characters	26,595	15.68%
of which $\leq 2$ characters	–	3.30%
Single-token alias	56,004	33.02%
Numeric-only alias	16,198	9.55%

Appendix

Table 17: Audit of gold aliases in the training data.

KL coef.	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
$1\!\times\!10^{-3}$	37.36	45.79	65.98	30.21	76.9%
$5\!\times\!10^{-3}$	36.61	45.45	63.31	28.77	74.4%

Appendix

Table 18: Search-R1 KL control on Qwen2.5-3B-Instruct. Aggregate percentages over the seven QA benchmarks.

Benchmark	Method	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
NQ	Initial checkpoint	16.15	59.92	60.80	36.43	89.3%
	Search-R1	47.17	70.33	67.15	47.23	97.4%
	+GCA	48.06	77.76	63.09	49.06	99.4%
TriviaQA	Initial checkpoint	25.81	60.20	75.26	45.31	86.1%
	Search-R1	62.72	73.58	87.63	64.48	95.5%
	+GCA	63.68	80.24	84.72	67.98	99.2%

Appendix

Table 19: Per-benchmark grounding results on Qwen2.5-3B-Instruct. Values are percentages. Initial checkpoint : the instruction-tuned model before task RL, evaluated with the same search tool and protocol; its EM values form the initial-checkpoint row of Table 2 . HitAcc , SupCor , and SupCor /Acc use the lenient evaluation correctness

\tilde{C}

(§ 5.1 ), so SupCor can exceed EM. From Search-R1 to GCA, HitAcc falls on NQ, TriviaQA, PopQA, and HotpotQA and rises on the other three benchmarks; since the additional hits may involve questions of different difficulty, these changes alone do not indicate better or worse evidence use.

Benchmark	Method	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
NQ	Initial checkpoint	30.44	66.54	65.40	43.52	96.5%
	Search-R1	49.61	79.75	63.42	50.58	99.2%
	+GCA	54.71	83.32	66.86	55.71	99.5%
TriviaQA	Initial checkpoint	55.57	71.19	88.09	62.71	96.0%
	Search-R1	69.04	80.58	88.66	71.44	98.3%
	+GCA	71.83	83.87	90.35	75.78	99.2%

Appendix

Table 20: Per-benchmark grounding results on Qwen3-30B-A3B-Instruct-2507. Rows and metrics as in Table 19 .

Method	Non-Live	Live	Multi-Turn	Irrelevance	v3 Overall
Base	87.02	77.21	37.62	85.42	67.29
Search-R1	87.71	77.03	41.12	85.00	68.62
+GCA	86.85	76.45	44.12	83.33	69.14

Appendix

Table 21: BFCL v3 function-calling results on Qwen3-30B-A3B-Instruct-2507. Values are percentages.

Method	Credit granularity	Decouple S/A	W/o extra model	W/o extra rollout	One-pass
Search-R1	trajectory	×	✓	✓	✓
DeSA	trajectory	✓ (staged)	✓	✓	×
IGPO	turn	×	✓	✓	✓
GiGPO	step/action	×	✓	✓	✓
HGPO	step/action hier.	×	✓	✓	✓
MT-PPO/GRPO	token (GAE)	×	× (critic)	× $O(G^{K-1})$	×

Appendix

Table 22: Design-space comparison with representative credit-assignment methods for search-agent RL.

Credit Assignment in RL

Credit assignment in reinforcement learning studies how to trace rewards and failures back to the actions, states, or agents that caused them.

See all 212 papers

Jul 24, 2026

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Yet terminal correctness assigns equal credit to direct answers from parametric knowledge, evidence-grounded answers, and answers recalled after redundant search. Process supervision and LLM judges provide richer feedback but can require high-cost annotation or inference; low-cost, non-contrastive policy confidence signals do not directly identify whether retrieved evidence changes an answer. We propose Search-G1, a reward framework that couples retrieval necessity with evidence reliance through two intervention-calibrated hidden-state readouts for selective grounded search. A prompt readout predicts closed-book sufficiency, whose complement proxies policy-relative retrieval necessity; an answer-commit readout predicts evidence reliance as sensitivity to evidence deletion conditional on the realized reasoning prefix. A correctness-first reward gates evidence-reliance credit by retrieval necessity, rewards sufficient direct answers, and penalizes repeated search. After calibration, scoring needs neither process annotations nor LLM-judge inference. As policy-relative targets, trajectory distributions, and representations may drift, Search-G1 periodically regenerates targets and refits its readouts. Across four QA benchmarks and two scales, Search-G1 achieves competitive task accuracy, with stronger counterfactual evidence dependence and fewer searches in the main comparisons, complemented by independent human audits of answer support.

Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang +7

Fudan University · Tencent · Nanjing University +2

Retrieval-Augmented Generation LLM Grounding Intrinsic Reward Methods Agentic Retrieval Reward Design for RL RL for LLM Agents

Ask PDF

Oct 18, 2026

EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents

LLM-based search agents demonstrate strong capabilities in long-horizon information seeking and reasoning, typically trained via reinforcement learning. However, existing approaches predominantly rely on trajectory-level outcome rewards, providing limited supervision for complex search and reasoning processes. Although step-level credit assignment offers a promising solution, obtaining reliable and task-grounded supervision for open-ended search remains challenging. In this paper, we propose Evidence-Guided Group Relative Policy Optimization (EG-GRPO), a framework that derives step-level supervision from the evidence annotations available in synthetic deep search data. EG-GRPO identifies the first supporting citation for each required evidence claim and uses the newly acquired evidence to construct step-level process rewards. By representing the accumulated evidence as an evolving knowledge state, EG-GRPO further performs state-conditioned relative advantage estimation and combines the resulting step-level signal with the standard trajectory-level outcome advantage. Experiments on five challenging deep search benchmarks show that EG-GRPO generally outperforms standard GRPO and existing reward-enhanced and step-level RL baselines across different model scales. These results demonstrate that task-grounded evidence annotations can provide effective process supervision for credit assignment in deep search RL.

Sijia Luo, Mengjie Gao, Yuxiao He +4

Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China · Ant Group, Beijing, China +2

RL for LLM Agents Step-Level Credit Assignment in RL RL for IR Group Relative Policy Optimization

Ask PDF

Jul 13, 2026

STAMP: Provenance-Guided Credit Assignment for Deep Search Agents

Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring -- outcome correctness, citation-aware rewards, and evidence coverage. Yet the actions that expose supporting documents receive no targeted credit, a gap we call the reward-credit mismatch. We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation in a training-time evidence graph, and first-exposure attribution traces each supported citation back to the action that first surfaced it. This step credit is injected through sign-preserving advantage modulation, which redistributes advantage across steps without changing the trajectory-level reward or the relative ranking of trajectories within each group. On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points under matched SFT initialization, training data, and search tools, and composes with both outcome-only and citation-rubric base rewards. Component ablations confirm that the provenance-based credit signal and the sign-preserving advantage modulation each contribute to the gains.

Ke Xu, Han Xu, Xinran Chen +6

Baidu Inc. · Peking University

Inference-Time Search Step-Level Credit Assignment in RL Step-Level Credit Assignment Credit Assignment Reward Design for RL RL for LLM Agents

Ask PDF

Abstract

Credit Assignment in RL

Explore similar work

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents

STAMP: Provenance-Guided Credit Assignment for Deep Search Agents