Benchmarking and Training Reward Models for Cross-Source Evidence Grounding in Search Agents

Haochen Zhao; Minsheng Hao; Zheng Li; Yifan Li; Zikang Liu; Bowen Qin; Yuhao Cheng; Chuang Fan

cs.AIOct 13, 2026

Benchmarking and Training Reward Models for Cross-Source Evidence Grounding in Search Agents

Authors: Haochen Zhao, Minsheng Hao, Zheng Li, Yifan Li, Zikang Liu, Bowen Qin, Yuhao Cheng, Chuang Fan,

Organizations: Peking University · Tencent · Renmin University of China

Abstract

Existing approaches to training and evaluating search agents focus heavily on final-answer quality, largely neglecting how generated answers map back to retrieved evidence. Recent methods track citations to enforce grounding, but citation support alone does not establish source independence or whether an answer appropriately handles conflicting evidence. We argue that credibility in open-web retrieval depends on principled cross-source evidence grounding that accounts for source independence and systematically attributes and adjudicates conflicting evidence, while hierarchical judgments of these grounding decisions provide a unified signal for training and evaluation. To this end, we propose an evaluation framework, Cross-Source Evidence Grounding (CSEG), that employs a fine-grained, six-layer decision tree over the complete search trajectory. We introduce CSEG-Bench and trainable reward models to provide targeted supervision following this framework. Our approach helps eliminating the severe data skew of natural trajectories by providing targeted coverage over complex, deep-layer decision paths, such as cross-source conflict resolution and nuanced attribution, that standard rollouts fail to capture. Specifically, (1) we define the hierarchy at label, node, and path levels; (2) we construct 1,160 verified trajectories over nine canonical paths by rewriting answers on real retrieved document pools, with independent frontier judges reproducing the entire gold path and human experts deciding inclusion; (3) we train CSEG-RM, which reaches 83.9% exact accuracy on our benchmark while lying on the performance-token efficiency Pareto frontier. Experiments that use this reward alone to supervise a frozen-retrieval answer policy raise the independently judged mean label from 2.83 to 4.19, and show that learning the hierarchy generalizes across layers at a fraction of frontier-judge cost.

Figures & tables

Figure 1: Same answer, different evidence quality. All four trajectories answer 1945 and receive outcome reward 1, yet their evidence differs: (A) two citations from independent evidence roots; (B) three citations from the same evidence root; (C) conflicting evidence is retrieved but ignored; and (D) conflicting evidence is incorrectly cited as support. An outcome-only reward cannot distinguish these trajectories.

Representative works	Claim support	Retrieved pool	Root indep.	Conflict	Authority	Traj.	RM resource
GAIA, BrowseComp	✗	✗	✗	✗	✗	✗	✗
Search-R1, R1-Searcher, ReSearch	✗	$\blacktriangle$ !	✗	✗	✗	$\blacktriangle$ !	✗
ALCE, AttrScore, SelfCite	✓	✗	✗	✗	✗	$\blacktriangle$ !	$\blacktriangle$ !
RAGAS, ARES, RAGChecker	✓	$\blacktriangle$ !	✗	$\blacktriangle$ !	✗	$\blacktriangle$ !	$\blacktriangle$ !
ClashEval, WikiContradict, RAMDocs, et al.	$\blacktriangle$ !	✓	✗	✓	$\blacktriangle$ !	$\blacktriangle$ !	✗
DRAGged into Conflicts	✓	✓	✗	✓	$\blacktriangle$ !	✓	✗

Table 1: Construct-level comparison of grounding evaluation and reward resources. ✓ : explicit;

\blacktriangle

! : partial; ✗ : not targeted. Symbols indicate task coverage, not latent capability. Appendix A defines the columns and discusses each family.

Figure 2: CSEG hierarchy and benchmark construction. (a) Rectangles denote layers and red capsules denote terminal outcomes. Red arrows indicate failures (labels 1–6); either all-pass outcome yields label 7. (b) Evidence scenario construction, answer rewriting on frozen pools, and full-path validation yield 1,160 trajectories. Roots and paths are schematic.

Layer	Aspect	Criterion
L1	Citation presence	A core claim cites an existing, topically relevant document.
L2	Multiple sources	At least two distinct cited sources support the core claim.
L3	Source independence	Cited support traces to at least two independent evidence roots.
L4	Conflict routing	Conflict classification: consistent / conflict identified / conflict missed.
L5-C	Consensus aggregation	Aggregation has appropriate scope and avoids over- or under-claiming.
L5-X	Conflict attribution	The acknowledged disagreement is attributed at source level.

Table 2: The CSEG hierarchy. Labels encode the depth of the first failed decision.

Label	L1_ fail	L2_ fail	L3_ fail	L4_conflict _missed	L5_consensus _fail	L5_conflict _fail	L6_fail_ conflict	all_pass_ conflict	all_pass_ consensus
$n$	78	79	184	140	105	151	181	128	114

Table 3: Composition of CSEG-Bench (

n=1{,}160

n

counts trajectories for each terminal path.

Model	Total	Think	Exact	Macro R.	MAE $\downarrow$	QWK $\uparrow$	Path ED $\downarrow$	Tokens
Qwen3-4B	4.0B	high	0.3828	0.3934	1.5560	0.3590	1.7612	3,725
Qwen3-8B	8.2B	high	0.4233	0.4543	1.3793	0.4809	1.5767	4,578
GPT-OSS-20B	20.9B	low	0.4940	0.5454	1.1845	0.5035	1.5078	1,156
GPT-OSS-20B	20.9B	medium	0.5293	0.5919	1.2517	0.4656	1.5690	3,763
GPT-OSS-20B	20.9B	high	0.4164	0.4998	1.9655	0.2376	2.1121	23,860
Muse-Glimmer-30B	30B	high	0.4595	0.5187	1.8129	0.3035	2.2440	5,175

Table 4: Benchmark results on CSEG-Bench (

n=1{,}160

). “Total” is the overall parameter count; “Tokens” is mean completion tokens per judgment. Bold marks the best value per block.

\dagger

is also the rejection-sampling SFT teacher.

Figure 3: Thinking effort, training, and accuracy–cost trade-offs on CSEG-Bench (

n=1{,}160

). (a) Selected zero-shot models improve (top) or regress (bottom) at higher thinking effort; the two plots use different vertical scales. (b) Qwen3-4B/8B accuracy across training routes. (c) Accuracy versus mean completion tokens; the dashed staircase is the Pareto frontier and gray points are other zero-shot configurations. Dark/light blue mark 4B/8B CSEG-RM; squares, triangles, and stars denote SFT, GRPO, and SFT+GRPO.

Quantity	Step 0 (Qwen3-8B base)	Step 200 (SFT+GRPO-400 RM)
Mean label	2.834	4.193
Median label	3	4
Label histogram, 1–7	186/42/123/63/53/20/25	20/4/172/110/123/33/50
$P(\text{label}\geq 4)$	0.314	0.617
$P(\text{label}\geq 5)$	0.191	0.402
$P(\text{label}=7)$	0.049	0.098

Table 5: Answer policy before and after CSEG-supervised optimization, on 512 frozen-retrieval questions. Scores come from an independent frontier judge running the same six-layer pipeline. The reward model is Qwen3-4B SFT+GRPO-400 (grader exact 0.8388).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Terminal type	$n$	Share	Depth
L1 fail (no valid citation)	23	9.0%	shallow
L2 fail (single source)	4	1.6%	shallow
L3 fail (multiple URLs, one evidence root)	85	33.2%	shallow
L4 conflict missed	32	12.5%	middle
L5-X fail (conflict attribution)	16	6.3%	deep
L5-C fail (consensus aggregation)	9	3.5%	deep

Appendix

Table 6: Gold distribution over 256 production trajectories, every one of which passes the service’s closed-QA outcome verifier. These are the original audit counts, before consensus paths bypassed L6.

Policy	Mean label	Fail by L3
Production agent (Table 6 )	4.48	43.8%
DeepSeek-V4-Flash (low)	4.11	52.4%
Qwen3-8B (thinking)	2.89	70.1%
GPT-OSS-120B (low)	2.37	84.2%

Appendix

Table 7: Open-weight policies rolled out on the same 256 frozen

(q,D)

prefixes as Table 6 , rewriting only the final answer. The production row is the outcome-passing gold audit (mean label computed from that distribution); the other rows are not outcome-filtered and do not replace the 43.8% / 33.2% claims above.

Figure 4: A concrete CSEG judgment: inputs (left), the six layer decisions with case-specific rationales (center), and the resulting path, node verdicts, terminal label, and path metrics (right).

Concept	Related expressions	Usage and distinction
Reasoning configuration	Thinking effort; reasoning effort; deliberation	Effort denotes a model-specific setting. Thinking mode indicates whether thinking is enabled; deliberation describes the generated reasoning process. Matching setting names do not establish equal reasoning budgets across models.
Label prediction	Exact accuracy; terminal accuracy; accuracy	In benchmark label results, these refer to exact label accuracy: agreement between predicted and gold terminal labels. Exact whole-path match instead requires the entire decision path to agree.
Evaluation role	Judge; scorer; grader; reward model (RM)	A judge makes evaluation decisions; the scorer composes layer decisions through routing. An RM supplies a training reward. Grader is contextual shorthand for the scoring model, not a separate model family; an independent evaluation judge is distinct from the RM used for policy training.
Efficiency measure	Completion tokens; token efficiency; cost	Completion-token count is the reported cost proxy. Accuracy–token comparisons do not by themselves measure latency, monetary cost, or compute across model sizes.
Hierarchy structure	Layer; node	There are six layers and seven node classifiers because layer L5 has alternative L5-C and L5-X nodes. The node prompts use the same layer terminology. Terminal label 7 means that every visited layer passed: five on the consensus branch or six on the conflict branch. No seventh layer is evaluated.
Evidence provenance	Document; source; evidence root	A document is a retrieved item; cited source refers to a document cited as evidence. Its evidence root is the origin of its support for the core claim. Different documents or websites need not provide independent evidence roots.

Appendix

Table 8: Terminology guide. Related expressions are not interchangeable unless explicitly identified as shorthand; model-specific settings retain their original names.

Type	Label	Required terminal behavior	Retrieved-pool requirement
L1_fail	1	L1 fail	—
L2_fail	2	L1 pass; L2 fail	independent
L3_fail	3	L1–L2 pass; L3 fail	redundant
L4_conflict_missed	4	conflict exists but is missed	independent, conflict
L5_consensus_fail	5	consistent pool; aggregation fails	independent, no conflict
L5_conflict_fail	5	conflict identified; attribution fails	independent, conflict

Appendix

Table 9: The nine canonical paths. “Independent,” “redundant,” and “conflict” are defined relative to the core claim.

Resource	Size	Role
Reward-model SFT train / dev	14,081 / 795	rejection-sampled Qwen3-32B traces; 486 / 26 seed split
Reward-model GRPO train / dev	7,320 / 911	train balanced to 488 per (layer, verdict) cell; dev natural
Data-source ablation arm	7,320 / 911	Qwen3-8B rollouts on the same seeds, truncated to the same rows
CSEG-Bench	1,160 trajectories	nine paths; 692 two-judge plus 468 single-judge; 5,166 layer decisions; final reporting only
Production audit (App. B )	256 trajectories	natural prevalence; motivates synthesis (RQ1)
Frozen-prefix policy rollouts	256 prefixes	same $(q,D)$ as App. B ; answer only; Table 7

Appendix

Table 10: Resources and split discipline. Benchmark, reward-model training, and answer-policy training rest on pairwise disjoint base pools. Layer examples are per-layer flattenings of a trajectory; a trajectory stopping at layer

N

contributes

N

of them.

Model	L1	L2	L3	L4	L5-C	L5-X	L6
	(1,160)	(1,082)	(1,003)	(819)	(219)	(460)	(423)
Qwen3-8B SFT	0.520	0.935	0.823	0.753	0.466	0.410	0.751
Qwen3-4B SFT+GRPO	0.863	0.965	0.888	0.847	0.709	0.982	0.924
Qwen3-8B SFT+GRPO	0.583	0.973	0.897	0.856	0.766	0.937	0.890

Appendix

Table 11: Conditional macro F1 per layer on CSEG-Bench, with gold reach

n

in the header. A node is scored only on examples whose gold path reaches it, and a prediction that stopped earlier counts as “not reached.”

Training data	Model	Exact	Macro R.	MAE $\downarrow$	QWK $\uparrow$	Path ED $\downarrow$
Constructed, balanced	Qwen3-4B	0.7353	0.7615	0.6672	0.7052	0.8466
Qwen3-8B natural rollouts	Qwen3-4B	0.6095	0.6410	0.8284	0.6641	1.0578
Constructed, balanced	Qwen3-8B	0.7672	0.7957	0.5776	0.7358	0.7026
Qwen3-8B natural rollouts	Qwen3-8B	0.5655	0.6260	0.9181	0.6386	1.1543

Appendix

Table 12: Training-data source, GRPO from the base model for 400 steps with every other setting byte-identical. Both arms hold 7,320 training and 911 development layer examples built on the same seed questions.

Figure 5: Step-200 high band (labels 5–7) on the same 512 prefixes. A weaker GRPO-only grader also moves the policy, but yields a proportion of trajectories with labels 5–7 about 10 percentage relatively lower than the official SFT+GRPO-400 reward.

Figure 6: Training dynamics of the official Qwen3-4B SFT+GRPO-400 reward model. (a) Supervised distillation, one epoch. (b–f) GRPO against the verifiable layer verdict. The validation markers are the in-loop split, not CSEG-Bench; reported RM quality remains Table 4 .

Figure 7: Training reward assigned by the frozen Qwen3-4B SFT+GRPO-400 CSEG-RM while optimizing the Qwen3-8B answer policy. This curve shows the policy fitting the reward model. Predictive validity is the independent-judge comparison in Table 5 , not this figure.

Node	Layer	Verdict	Next
L1	Citation presence	pass	L2
L1	Citation presence	fail	STOP (label 1)
L2	Multiple sources	pass	L3
L2	Multiple sources	fail	STOP (label 2)
L3	Source independence	pass	L4
L3	Source independence	fail	STOP (label 3)

Appendix

Table 13: Routing of each node verdict.

Slot	Meaning	Use
{user_question}	The user original question/query	Filled from the trajectory at inference.
{retrieved_sources}	Formatted list of retrieved web documents, including source URL, site, authority level, publish time, and content excerpt	Filled from the trajectory at inference.
{final_answer}	The agent final answer text (excluding tool call sections)	Filled from the trajectory at inference.
{agent_search_queries}	The search queries issued by the agent	Declared in the spec; unused by the current node templates.

Appendix

Table 14: Input slots interpolated into every node prompt.

Label	Meaning
1	No citation support: core claims lack any relevant source backing
2	Single-source: claim supported by only one source, no multi-source verification
3	Redundant multi-source: multiple sources cited but all trace to the same origin
4	Conflict blindness: sources contain real conflicts but agent completely missed them (source blending)
5	Aggregation/attribution defect: consistency/conflict detected, but aggregation introduced framing errors or conflict attribution was insufficient
6	Deficient adjudication: reached L6 but adjudication was unsupported, overconfident, or incorrectly weighted

Appendix

Table 15: Label assigned when evaluation stops.

Evidence-Grounded Generation

Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.

See all 148 papers

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search
Jul 24, 2026Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang +7Retrieval-Augmented Generation LLM Grounding
Fudan University · Tencent · Nanjing University +2
Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents
Oct 16, 2026Qirui Liu, Hao Chen, Yi Wen +7Credit Assignment in RL Reinforcement Learning with Verifiable Rewards
South China University of Technology · FiT, Tencent · City University of Hong Kong +5
When Connectivity Misleads: Guiding Agentic Search with Contrastive Evidence Paths
Oct 26, 2026Junchen Li, Liang Xu, Rongzheng Wang +3Web Search Agents Graph-Based Retrieval
University of Electronic Science and Technology of China Chengdu, China

Jul 24, 2026J/K move · Enter open · S save

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang, Junming Zhang+6

Fudan University · Tencent · Nanjing University+2

Retrieval-Augmented Generation LLM Grounding Intrinsic Reward Methods Agentic Retrieval Reward Design for RL RL for LLM Agents

Open paper AskPDF arXiv

Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Yet terminal correctness assigns equal credit to direct answers from parametric knowledge, evidence-grounded answers, and answers recalled after redundant search. Process supervision and LLM judges provide richer feedback but can require high-cost annotation or inference; low-cost, non-contrastive policy confidence signals do not directly identify whether retrieved evidence changes an answer. We propose Search-G1, a reward framework that couples retrieval necessity with evidence reliance through two intervention-calibrated hidden-state readouts for selective grounded search. A prompt readout predicts closed-book sufficiency, whose complement proxies policy-relative retrieval necessity; an answer-commit readout predicts evidence reliance as sensitivity to evidence deletion conditional on the realized reasoning prefix. A correctness-first reward gates evidence-reliance credit by retrieval necessity, rewards sufficient direct answers, and penalizes repeated search. After calibration, scoring needs neither process annotations nor LLM-judge inference. As policy-relative targets, trajectory distributions, and representations may drift, Search-G1 periodically regenerates targets and refits its readouts. Across four QA benchmarks and two scales, Search-G1 achieves competitive task accuracy, with stronger counterfactual evidence dependence and fewer searches in the main comparisons, complemented by independent human audits of answer support.

Abstract

Evidence-Grounded Generation

Explore similar work

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

When Connectivity Misleads: Guiding Agentic Search with Contrastive Evidence Paths