Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

Qirui Liu; Hao Chen; Yi Wen; Weijie Shi; Yao Zhao; Xianhe Chen; Hao Zhao; Xing Tang

cs.LGOct 16, 2026

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

Authors: Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang,

Organizations: South China University of Technology · FiT, Tencent · City University of Hong Kong · The Hong Kong University of Science and Technology ·

Abstract

Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it. We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the evidence-supported share of correct answers falls from 84.0% to 42.9%. We attribute this to GRPO's credit assignment: rollouts that retrieve answer-bearing evidence are favoured in aggregate only in proportion to their accuracy gain over the rest, a grounding gap that answering from memory or misusing evidence shrinks or reverses. To address this, we propose Grounded Credit Assignment (GCA), which uses verifiable sub-outcomes (evidence retrieval and answer correctness) to correct search and answer credit separately: a wrong answer is blamed more than the search that found the evidence, and a correct answer that bypasses retrievable evidence is credited less. Never negative on evidence-finding search, the correction does not shrink with the grounding gap; capped at 10% of each token's advantage, it never overrides the outcome signal. Without extra models, rollouts, or training stages, GCA improves average EM over Search-R1 by 5.7 (Qwen2.5-3B) and 6.3 points (Qwen3-30B-A3B) and raises this share from 76.9% to 93.4%.

Figures & tables

Figure 1: Outcome-only RL improves accuracy without improving grounding. (a) Share of correct answers that are supported by retrieved evidence ( SupCor /Acc) on Qwen2.5-3B. Search-R1 raises EM on every benchmark (Table 2 ) but lowers this share below the initial checkpoint (before task RL) on MuSiQue and Bamboogle; GCA restores it. (b) Token-level view of the Search-R1 run (Appendix D.1 ): share of positively credited tokens (type

A

D

rollouts) that are post-hit redundant searches or belong to ungrounded type-

D

rollouts, and share of answer-finding search tokens that lie in hit-but-wrong (type

B

) rollouts. Shares are computed per rollout type and ignore homogeneous groups, whose advantage is zero.

Figure 2: Overview of GCA. (a) Uniform advantage broadcasting conflates search and answer credit within a rollout. (b) GCA diagnoses each rollout by retrieval hit $H$ and answer correctness $C$ , routes signed residuals to functional segments ( vSearch , rSearch , Ans ), and concentrates them on decision-critical tokens via margin–entropy salience. (c) Resulting residuals; signs show residual polarity, not the final advantage’s sign.

Type	Condition	vSearch	rSearch	Ans
$A$	$H{=}1,C{=}1$	$+0.5$	$-0.1$	$+1.0$
$B_{1}$	$H{=}1,C{=}0$ , wrong	$+0.5$	$-0.2$	$-1.0$
$B_{2}$	$H{=}1$ , timeout	$+0.5$	$-0.1$	$0$
$C$	$H{=}0,C{=}0$	–	Base	Base
$D$	$H{=}0,C{=}1,\mathrm{Retr}{=}1$	–	$-0.2$	$-0.1$
$D$	$H{=}0,C{=}1,\mathrm{Retr}{=}0$	Mask

Table 1: Routing coefficients. Entries are signed residual coefficients

P_{T,S}\lambda_{T,S}

; signs follow each segment’s proxy sub-outcome (§ 4.1 ). Shaded: conflict cells, where outcome advantage and sub-outcome disagree in sign. Base : original advantage only; Mask : excluded from the policy loss; –: segment absent.

Qwen2.5-3B-Instruct
Methods	General QA			Multi-Hop QA				Avg.
	NQ †	TriviaQA ⋆	PopQA ⋆	HotpotQA †	2Wiki ⋆	MuSiQue ⋆	Bamboogle ⋆
Initial checkpoint + search	16.15	25.81	16.54	12.37	10.66	4.18	12.00	13.96
IRCoT	11.1	31.2	20.0	16.4	17.1	6.7	24.0	18.1
Rejection Sampling	29.4	48.8	33.2	24.0	23.3	5.9	21.0	26.5
ZeroSearch	41.4	57.4	44.8	27.4	30.0	9.8	11.1	31.7

Table 2: Main results (EM, %) on seven QA benchmarks. † / ⋆ : in-domain/out-of-domain. Best results in bold . Initial checkpoint + search (before task RL) and Search-R1 are run in the same setting as GCA; GCA rows report the mean and

1\sigma

over 3 training seeds.

Backbone	Method	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
Qwen2.5-3B	Initial checkpoint	13.96	43.00	55.27	23.77	75.0%
	Search-R1	37.36	45.79	65.98	30.21	76.9%
	+GCA	43.06	64.68	66.17	42.80	93.4%
Qwen3-30B-A3B	Initial checkpoint	35.75	57.20	69.37	39.68	89.3%
	Search-R1	46.29	65.09	68.05	44.29	92.3%
	+GCA	52.59	71.87	72.60	52.18	94.1%

Table 3: Aggregate grounding metrics (%, averaged over the seven benchmarks). ToolHit

=P(H{=}1)

; HitAcc

=P(\tilde{C}{=}1\mid H{=}1)

with the lenient correctness

\tilde{C}

(§ 5.1 ); SupCor

=

ToolHit

\times

HitAcc estimates the supported-correct rate; SupCor /Acc is its share of the lenient accuracy Acc.

	EM								Grounding
Variant	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	ToolHit	HitAcc	SupCor	SupCor /Acc
Search-R1	47.17	62.72	40.25	36.37	34.94	12.08	28.00	37.36	45.79	65.98	30.21	76.9%
w/o salience	45.93	62.46	43.14	41.98	40.89	16.52	43.13	42.01	57.85	66.89	38.70	87.6%
w/o segment split	44.86	61.38	42.54	40.21	41.52	14.14	37.51	40.31	53.81	64.50	34.71	82.0%
GCA (full)	48.06	63.68	42.94	44.65	44.12	18.74	39.20	43.06	64.68	66.17	42.80	93.4%

Table 4: Budget-matched leave-one-out ablation on Qwen2.5-3B (%, mean over 3 training seeds). W/o salience replaces token salience by its per-cell mean; w/o segment split merges vSearch and rSearch into one search segment. Best results in bold .

Figure 3: Training dynamics on Qwen2.5-3B (§ 5.6 ). Faint lines show per-step values and solid lines an exponential moving average. Top: grounded behaviour ( SupCor , ToolHit , and the hit-but-wrong share of all rollouts, type

B_{1}

); bottom: optimisation diagnostics (mean outcome reward, zero-variance group rate, and actor entropy).

Routing	In-domain	Out-of-domain	Overall
Search-R1	41.77	35.60	37.36
Learned magnitudes	46.41	41.53	42.92
Fixed (Table 1 )	46.36	41.74	43.06

Table 5: Fixed vs. learned routing magnitudes on Qwen2.5-3B (average EM, %, 3 seeds). The learned router keeps the fixed polarities and the residual budget (Appendix H ).

$\lambda$ scale $s$	0.50	0.75	1.00	1.25	1.50
$\kappa$	0.05	0.075	0.10	0.125	0.15
EM (3B)	41.23	42.76	43.06	41.57	36.92
EM (30B)	–	48.01	52.59	46.77	–
Residual scale $\eta$	0.05	0.10	0.20	0.50	1.00
$\kappa$	0.025	0.05	0.10	0.25	0.50
EM (3B)	41.98	43.72	43.06	40.80	37.89

Table 6: Sensitivity of average EM (%) to the

\lambda

scale

s

, the residual scale

\eta

, and the salience cap

c_{\max}

\kappa=\eta\lambda_{\max}c_{\max}

is the worst-case relative residual (Proposition 2 ); shaded: default; –: not run. Search-R1 scores 37.36 (3B) and 46.29 (30B); ToolHit , HitAcc , and SupCor are in Appendix D.4 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$r_{i}$ , $\mathcal{G}(x)$	$i$ -th rollout; group of $n$ rollouts for question $x$
$R_{i}$ , $\hat{A}_{i}$	Outcome reward; GRPO/GSPO group-relative advantage
$\mathcal{A}(x)$ , $m(a,o)$	Gold alias set; case-insensitive containment test
$H_{i}$ , $C_{i}$	Retrieval hit ( ToolHit ); EM answer correctness (Eq. 1 )
$\tilde{C}_{i}$	Lenient evaluation correctness (alias-aware substring match) used by HitAcc , SupCor , and Acc (§ 5.1 )
$k^{\star}_{i}$	First search turn whose tool response contains a gold alias

Appendix

Table 7: Summary of notation.

Cell $(T,S)$	Base	Step 300	Step 600	Step 900
$A\!\cdot\!\textsc{vSearch}$	114,426 (30.68%)	74,815 (36.44%)	100,449 (25.62%)	102,219 (30.07%)
$A\!\cdot\!\textsc{rSearch}$	100,010 (26.81%)	54,214 (26.41%)	116,053 (29.60%)	111,889 (32.91%)
$A\!\cdot\!\textsc{Ans}$	158,538 (42.51%)	76,254 (37.15%)	175,504 (44.77%)	125,836 (37.02%)
$B_{1}\!\cdot\!\textsc{vSearch}$	113,462 (27.97%)	49,766 (29.26%)	69,547 (23.23%)	61,024 (24.59%)
$B_{1}\!\cdot\!\textsc{rSearch}$	102,110 (25.18%)	57,275 (33.67%)	91,709 (30.63%)	97,712 (39.37%)
$B_{1}\!\cdot\!\textsc{Ans}$	190,023 (46.85%)	63,066 (37.07%)	138,163 (46.14%)	89,439 (36.04%)

Appendix

Table 8: Per-cell trainable token counts

n_{T,S}

and within-type share at four checkpoints of the Search-R1 Qwen2.5-3B run. “Trainable” = response tokens minus the eight structural-tag literals. Percentages are within-row ratios

n_{T,S}/\sum_{S^{\prime}}n_{T,S^{\prime}}

for that rollout type at that checkpoint; type

C

has no segment partition.

Figure 4: Within-row segment ratios for type

A

(left) and type

B_{1}

(right) across the four checkpoints. Percentages denote

n_{T,S}/\sum_{S^{\prime}}n_{T,S^{\prime}}

Figure 5: Absolute trainable-token counts for type

B_{2}

(left) and type

D

(right) across the four checkpoints, with stacked segment composition.

Type	Condition	vSearch	rSearch	Ans
$A$	$H{=}1,C{=}1$	$P{=}{+}1,\;\lambda{=}0.5,\;c^{+}$	$P{=}{-}1,\;\lambda{=}0.1,\;c^{-}$	$P{=}{+}1,\;\lambda{=}1.0,\;c^{+}$
$B_{1}$	$H{=}1,C{=}0$ , valid wrong	$P{=}{+}1,\;\lambda{=}0.5,\;c^{+}$	$P{=}{-}1,\;\lambda{=}0.2,\;c^{-}$	$P{=}{-}1,\;\lambda{=}1.0,\;c^{-}$
$B_{2}$	$H{=}1,C{=}0$ , timeout	$P{=}{+}1,\;\lambda{=}0.5,\;c^{+}$	$P{=}{-}1,\;\lambda{=}0.1,\;c^{-}$	$P{=}0,\;\lambda{=}0.0$
$C$	$H{=}0,C{=}0$	N/A	base	base
$D$	$H{=}0,C{=}1$ , Retr $=1$	N/A	$P{=}{-}1,\;\lambda{=}0.2,\;c^{-}$	$P{=}{-}1,\;\lambda{=}0.1,\;c^{-}$
$D$	$H{=}0,C{=}1$ , Retr $=0$	masked	masked	masked

Appendix

Table 9: Routing coefficients

(P_{T,S},\lambda_{T,S})

used in all reported experiments.

c^{\pm}

indicates the salience variant applied (§ E ). Base = original GRPO/GSPO advantage only; Mask = removed from the policy update; N/A = segment absent for this type. We use

\eta\!=\!0.2

and

c_{\max}\!=\!0.5

, giving worst-case residual

\eta\,\lambda_{\max}\,c_{\max}\,|\hat{A}_{i}|=0.1\,|\hat{A}_{i}|

Figure 6: Actor entropy traces of the FM1 and FM2 diagnostic runs on Qwen2.5-3B. FM1 (

\lambda_{B_{1},\textsc{Ans}}\!=\!2.0,\,\eta\!=\!0.5

, no

c_{t}

cap) reaches its peak around step

145

and collapses thereafter; FM2 keeps the same

\lambda

but halves

\eta

0.2

, which only delays the explosion.

Qwen2.5-3B
$s$	EM	ToolHit	HitAcc	SupCor
Search-R1 (no residual)	37.36	45.79	65.98	30.21
$0.50$	41.23	61.80	65.44	40.44
$0.75$	42.76	62.02	66.75	41.40
$\mathbf{1.00}$	43.06	64.68	66.17	42.80
$1.25$	41.57	59.57	67.84	40.41

Appendix

Table 10: Sensitivity to a global

\lambda

scale

s

on Qwen-family backbones.

s\!=\!1

corresponds to Table 9 .

Qwen2.5-3B
$\eta$	EM	ToolHit	HitAcc	SupCor
$0.05$	41.98	63.01	65.76	41.44
$0.10$	43.72	63.61	65.95	41.95
$\mathbf{0.20}$	43.06	64.68	66.17	42.80
$0.50$	40.80	62.61	64.67	40.48
$1.00$	37.89	59.45	63.14	37.53

Appendix

Table 11: Sensitivity to the global residual scale

\eta

on Qwen-family backbones.

Qwen2.5-3B
$c_{\max}$	EM	ToolHit	HitAcc	SupCor
$0.20$	40.83	64.42	62.77	40.43
$0.30$	43.07	64.58	65.12	42.05
$\mathbf{0.50}$	43.06	64.68	66.17	42.80
$1.00$	42.42	63.75	65.69	41.88
Qwen3-30B-A3B-Instruct

Appendix

Table 12: Sensitivity to the per-token cap

c_{\max}

on Qwen-family backbones.

Bucket $(T,S)$	Base		Step 300		Step 900
Bucket $(T,S)$	$\bar{\delta}$ @ $3$ / $\hat{H}$ @ $32$	low- $\delta$ / high- $\hat{H}$	$\bar{\delta}$ @ $3$ / $\hat{H}$ @ $32$	low- $\delta$ / high- $\hat{H}$	$\bar{\delta}$ @ $3$ / $\hat{H}$ @ $32$	low- $\delta$ / high- $\hat{H}$
$A\!\cdot\!\textsc{vSearch}$	6.61 / 0.167	16.1% / 12.0%	10.65 / 0.052	0 5.1% / 0 1.4%	10.95 / 0.059	0 5.9% / 0 2.1%
$A\!\cdot\!\textsc{rSearch}$	6.22 / 0.178	16.7% / 13.7%	0 7.55 / 0.119	11.9% / 0 5.1%	0 6.70 / 0.152	15.0% / 0 8.5%
$A\!\cdot\!\textsc{Ans}$	5.35 / 0.204	19.7% / 15.5%	0 8.73 / 0.091	0 9.1% / 0 3.7%	0 7.44 / 0.139	13.7% / 0 8.8%
$B_{1}\!\cdot\!\textsc{vSearch}$	6.72 / 0.161	15.5% / 11.6%	10.04 / 0.068	0 6.6% / 0 2.6%	0 9.74 / 0.089	0 8.9% / 0 4.4%
$B_{1}\!\cdot\!\textsc{rSearch}$	6.14 / 0.185	17.3% / 14.7%	0 6.58 / 0.154	15.3% / 0 8.8%	0 5.76 / 0.189	18.4% / 12.8%

Appendix

Table 13: Per-(checkpoint, type, segment) margin and entropy on trainable non-structural tokens. “

\bar{\delta}

3

” is the bucket mean of the sampled-token margin against the top-

3

competitors; “

\hat{H}

32

” is the bucket mean of the top-

32

normalised entropy. “low-

\delta

” / “high-

\hat{H}

” denote tail fractions

\Pr[\delta_{t}\!<\!0]

and

\Pr[\hat{H}_{t}\!>\!0.5]

, respectively. Buckets with

P\!=\!0

(Type-

C

, all segments) are listed for reference but contribute no residual.

Figure 7: Density of margin@

3

over trainable non-structural tokens, per (T,S) bucket and checkpoint. Line colour encodes the segment (green: vSearch , red: rSearch , blue: Ans ); line style encodes the rollout type (

A\!:\!{-}

B_{1}\!:\!\text{--}\,\text{--}

B_{2}\!:\!\cdots

D\!:\!{-}\!\cdot\!{-}

). The within-bucket distribution is heavy-tailed and overlaps substantially with

\delta_{t}\!<\!0

at every checkpoint, leaving a non-empty low-margin sub-population for

d_{t}^{+}

to amplify on positive segments and a heavy high-margin sub-population for

d_{t}^{-}

to amplify on negative segments.

Figure 8: Density of entropy@

32

over trainable non-structural tokens, per (T,S) bucket and checkpoint. Encoding matches Fig. 7 . The mass concentrates near

\hat{H}_{t}\!\to\!0

as training progresses (more tokens become high-confidence picks), but a non-empty right tail above

\hat{H}_{t}\!=\!0.5

persists across all active buckets even at step 900 , especially on the failure-mode buckets

B_{1}\!\cdot\!\textsc{Ans}

B_{1}\!\cdot\!\textsc{rSearch}

, and

B_{2}

Margin (vs. $K\!=\!3$ anchor)			Entropy (vs. $M\!=\!32$ anchor)
$K$	$\|\Delta\bar{\delta}\|$ (nats)	min overlap@ $1\%$	$M$	$\|\Delta\hat{H}\|$
$2$	$0.343$	$0.83$	$2$	$0.202$
$\mathbf{3}$	$\mathbf{0.000}$	$\mathbf{1.00}$	$4$	$0.131$
$5$	$0.477$	$0.75$	$8$	$0.077$
$10$	$1.157$	$0.75$	$16$	$0.034$
$32$	$2.316$	$0.75$	$\mathbf{32}$	$\mathbf{0.000}$

Appendix

Table 14: Worst-case cell-level shifts of margin@

K

and entropy@

M

across the

36

EDA cells (3 checkpoints

\times

12 active

(T,S)

buckets). “

|\Delta\bar{\delta}|

” is the maximum absolute shift of the bucket-mean margin (in nats) relative to the

K\!=\!3

anchor; “min overlap@

1\%

” is the minimum overlap of the bottom-

1\%

margin tail against the

K\!=\!3

tail. “

|\Delta\hat{H}|

” is the analogous bucket-mean shift of normalised entropy relative to the

M\!=\!32

anchor. Bucket-mean margin shifts smoothly and monotonically in

K

(a known property of

\log\mathrm{mean}\exp

as more competitors are added), but the active region – the bottom-

1\%

tail – is preserved at

\geq\!0.75

overlap across the entire

\sim\!30\!\times\!

sweep, so the same high-salience tokens absorb the residual regardless of

K

. Bucket-mean entropy shifts are below

0.034

for

M\!\geq\!16

and below

0.077

for

M\!\geq\!8

, putting

M\!=\!32

on a stable plateau with

M\!=\!64

Figure 9: Bucket-conditional margin@

K

curves with p25/p75 bands at base (left), step 300 (middle), step 900 (right). Each panel overlays all active

P\!\neq\!0

(T,S)

buckets; line colour encodes the segment (green: vSearch , red: rSearch , blue: Ans ) and line style encodes the rollout type (

A\!:\!{-}

B_{1}\!:\!\text{--}\,\text{--}

B_{2}\!:\!\cdots

D\!:\!{-}\!\cdot\!{-}

). The

K

-axis sweeps

\{2,3,5,10,32,63\}

on a log scale. Curves shift smoothly and monotonically with

K

\log\mathrm{mean}\exp

property), but the relative ordering of buckets and the within-bucket dispersion (p25–p75 band) are preserved across the full sweep, matching the bottom-

1\%

overlap statistic in Table 14 .

Figure 10: Bucket-conditional entropy@

M

curves with p25/p75 bands at base (left), step 300 (middle), step 900 (right). The

M

-axis sweeps

\{2,4,8,16,32,64\}

on a log scale; legend encoding matches Fig. 9 . Once

M\!\geq\!16

the bucket means are within

0.034

of their

M\!=\!32

anchor and within

0.026

M\!=\!64

across all

36

cells, justifying

M\!=\!32

as a stable plateau choice.

$K$	EM	ToolHit	HitAcc	SupCor
$2$	42.72	64.26	66.34	42.63
$\mathbf{3}$	43.06	64.68	66.17	42.80
$5$	42.78	64.06	66.13	42.36
$32$	43.18	64.96	65.29	42.41

Appendix

Table 15: Sensitivity of GCA to the margin bandwidth

K

on Qwen2.5-3B (fixed

M\!=\!32

). The bold

K\!=\!3

row is the reported configuration. EM deviates by at most

0.34

points from the default

K\!=\!3

setting; the full range across tested settings is

0.46

points. All four metrics remain within

1.0

point of the

K\!=\!3

anchor.

$M$	EM	ToolHit	HitAcc	SupCor
$8$	42.96	64.08	65.79	42.16
$16$	43.10	64.50	66.03	42.58
$\mathbf{32}$	43.06	64.68	66.17	42.80
$64$	42.31	64.10	65.93	42.26

Appendix

Table 16: Sensitivity of GCA to the entropy bandwidth

M

on Qwen2.5-3B (fixed

K\!=\!3

). The bold

M\!=\!32

row is the reported configuration. EM deviates by at most

0.75

points from the default

M\!=\!32

setting (at

M\!=\!64

); the full range across tested settings is

0.79

points. All four metrics remain within

1.0

point of the

M\!=\!32

anchor.

Case	Count	Share
Yes/no gold answer (5,481 in HotpotQA)	5,483	3.23%
More than one alias (all NQ)	7,750	4.57%
Alias of $\leq 4$ characters	26,595	15.68%
of which $\leq 2$ characters	–	3.30%
Single-token alias	56,004	33.02%
Numeric-only alias	16,198	9.55%

Appendix

Table 17: Audit of gold aliases in the training data.

KL coef.	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
$1\!\times\!10^{-3}$	37.36	45.79	65.98	30.21	76.9%
$5\!\times\!10^{-3}$	36.61	45.45	63.31	28.77	74.4%

Appendix

Table 18: Search-R1 KL control on Qwen2.5-3B-Instruct. Aggregate percentages over the seven QA benchmarks.

Benchmark	Method	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
NQ	Initial checkpoint	16.15	59.92	60.80	36.43	89.3%
	Search-R1	47.17	70.33	67.15	47.23	97.4%
	+GCA	48.06	77.76	63.09	49.06	99.4%
TriviaQA	Initial checkpoint	25.81	60.20	75.26	45.31	86.1%
	Search-R1	62.72	73.58	87.63	64.48	95.5%
	+GCA	63.68	80.24	84.72	67.98	99.2%

Appendix

Table 19: Per-benchmark grounding results on Qwen2.5-3B-Instruct. Values are percentages. Initial checkpoint : the instruction-tuned model before task RL, evaluated with the same search tool and protocol; its EM values form the initial-checkpoint row of Table 2 . HitAcc , SupCor , and SupCor /Acc use the lenient evaluation correctness

\tilde{C}

(§ 5.1 ), so SupCor can exceed EM. From Search-R1 to GCA, HitAcc falls on NQ, TriviaQA, PopQA, and HotpotQA and rises on the other three benchmarks; since the additional hits may involve questions of different difficulty, these changes alone do not indicate better or worse evidence use.

Benchmark	Method	EM	ToolHit	HitAcc	SupCor	SupCor /Acc
NQ	Initial checkpoint	30.44	66.54	65.40	43.52	96.5%
	Search-R1	49.61	79.75	63.42	50.58	99.2%
	+GCA	54.71	83.32	66.86	55.71	99.5%
TriviaQA	Initial checkpoint	55.57	71.19	88.09	62.71	96.0%
	Search-R1	69.04	80.58	88.66	71.44	98.3%
	+GCA	71.83	83.87	90.35	75.78	99.2%

Appendix

Table 20: Per-benchmark grounding results on Qwen3-30B-A3B-Instruct-2507. Rows and metrics as in Table 19 .

Method	Non-Live	Live	Multi-Turn	Irrelevance	v3 Overall
Base	87.02	77.21	37.62	85.42	67.29
Search-R1	87.71	77.03	41.12	85.00	68.62
+GCA	86.85	76.45	44.12	83.33	69.14

Appendix

Table 21: BFCL v3 function-calling results on Qwen3-30B-A3B-Instruct-2507. Values are percentages.

Method	Credit granularity	Decouple S/A	W/o extra model	W/o extra rollout	One-pass
Search-R1	trajectory	×	✓	✓	✓
DeSA	trajectory	✓ (staged)	✓	✓	×
IGPO	turn	×	✓	✓	✓
GiGPO	step/action	×	✓	✓	✓
HGPO	step/action hier.	×	✓	✓	✓
MT-PPO/GRPO	token (GAE)	×	× (critic)	× $O(G^{K-1})$	×

Appendix

Table 22: Design-space comparison with representative credit-assignment methods for search-agent RL.

Credit Assignment in RL

Credit assignment in reinforcement learning studies how to trace rewards and failures back to the actions, states, or agents that caused them.

See all 212 papers

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search
Jul 24, 2026Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang +7Retrieval-Augmented Generation LLM Grounding
Fudan University · Tencent · Nanjing University +2
EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents
Oct 18, 2026Sijia Luo, Mengjie Gao, Yuxiao He +4RL for LLM Agents Step-Level Credit Assignment in RL
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China · Ant Group, Beijing, China +2
STAMP: Provenance-Guided Credit Assignment for Deep Search Agents
Jul 13, 2026Ke Xu, Han Xu, Xinran Chen +6Inference-Time Search Step-Level Credit Assignment in RL
Baidu Inc. · Peking University

Jul 24, 2026J/K move · Enter open · S save

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang, Junming Zhang+6

Fudan University · Tencent · Nanjing University+2

Retrieval-Augmented Generation LLM Grounding Intrinsic Reward Methods Agentic Retrieval Reward Design for RL RL for LLM Agents

Open paper AskPDF arXiv

Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Yet terminal correctness assigns equal credit to direct answers from parametric knowledge, evidence-grounded answers, and answers recalled after redundant search. Process supervision and LLM judges provide richer feedback but can require high-cost annotation or inference; low-cost, non-contrastive policy confidence signals do not directly identify whether retrieved evidence changes an answer. We propose Search-G1, a reward framework that couples retrieval necessity with evidence reliance through two intervention-calibrated hidden-state readouts for selective grounded search. A prompt readout predicts closed-book sufficiency, whose complement proxies policy-relative retrieval necessity; an answer-commit readout predicts evidence reliance as sensitivity to evidence deletion conditional on the realized reasoning prefix. A correctness-first reward gates evidence-reliance credit by retrieval necessity, rewards sufficient direct answers, and penalizes repeated search. After calibration, scoring needs neither process annotations nor LLM-judge inference. As policy-relative targets, trajectory distributions, and representations may drift, Search-G1 periodically regenerates targets and refits its readouts. Across four QA benchmarks and two scales, Search-G1 achieves competitive task accuracy, with stronger counterfactual evidence dependence and fewer searches in the main comparisons, complemented by independent human audits of answer support.

Abstract

Credit Assignment in RL

Explore similar work

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents

STAMP: Provenance-Guided Credit Assignment for Deep Search Agents