Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Tianle Xia; Ming Xu; Lingxiang Hu; Yiding Sun; Wenwei Li; Linfang Shang; Liqun Liu; Peng Shu

cs.CLFeb 26, 2026

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Authors: Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun, Wenwei Li, Linfang Shang, Liqun Liu, Peng Shu, +2 more

Organizations: Tencent

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge, yet traditional single-round retrieval struggles with complex multi-step reasoning. Agentic RAG addresses this by enabling LLMs to dynamically decide when and what to retrieve, but current RL-based training methods suffer from sparse outcome rewards that discard intermediate signals and low sample efficiency where failed samples contribute nothing. We propose Search-P1, a framework that introduces path-centric reward shaping for agentic RAG training, comprising two key components: (1) Path-Centric Reward, which evaluates the structural quality of reasoning trajectories through order-agnostic step coverage and soft scoring that extracts learning signals even from failed samples, and (2) Dual-Track Path Scoring with offline-generated reference planners that assesses paths from both self-consistency and reference-alignment perspectives. Experiments on multiple QA benchmarks demonstrate that Search-P1 achieves significant improvements over Search-R1 and other strong baselines, with an average accuracy gain of 7.7 points.

Figures & tables

Figure 1: Performance comparison of Search-P1 against baselines on QA benchmarks. Our method achieves the highest average accuracy across all datasets on both (a) Qwen2.5-7B and (b) Qwen2.5-3B models.

Figure 2: Overview of Search-P1 framework. Our approach introduces path-centric reward shaping for agentic RAG training, comprising: (1) Dual-Track Path Scoring that evaluates trajectories from both self-consistency and reference-alignment perspectives, and (2) Soft Outcome Scoring that extracts training signals even from incorrect answers.

Method	General QA			Multi-Hop QA				Avg.	Internal
Method	NQ †	TriviaQA	PopQA	HotpotQA †	2Wiki	Musique	Bamboogle	Avg.	AD-QA
Qwen2.5-7B
Direct	13.4	40.8	14.0	18.3	25.0	3.1	12.0	18.1	10.3
CoT	4.8	18.5	5.4	9.2	11.1	2.2	23.2	10.6	8.7
RAG	34.9	58.5	39.2	29.9	23.5	5.8	20.8	30.4	60.4
IRCoT	22.4	47.8	30.1	13.3	14.9	7.2	22.4	23.9	52.3

Table 1: Main results (ACC %) on seven public QA benchmarks and one internal dataset. Best results are in bold , second best are underlined . † denotes in-domain datasets used for training; others are out-of-domain. AD-QA is a proprietary advertising QA dataset. HiPRAG results are from our reproduction using the same retrieval setup.

Method	Avg. ACC
Search-P1 (Full)	47.3
w/o Reference-Alignment	42.0
w/o Self-Consistency	44.2
Search-R1 (Baseline)	39.6

Table 2: Ablation study on path-centric reward components (Qwen2.5-7B). Per-dataset results are in Appendix F.4 .

Figure 3: Training dynamics comparison of different format reward strategies. Soft Format (our buffered design) achieves faster ACC improvement and higher stable rewards compared to Strict Format (zero reward for invalid format) and Without Format baseline.

Figure 4: Effect of soft outcome scoring across datasets. Gray bars show accuracy without soft scoring (binary outcome), blue bars show accuracy with soft scoring. Per-dataset results are in Appendix F.6 .

Figure 5: Hyperparameter sensitivity analysis. All rewards are averaged over steps 195–205. (a) Effect of path reward weight

\lambda_{p}

. (b) Effect of accuracy weight

\lambda_{a}

. Per-dataset results are in Appendix F.7 .

Figure 6: Efficiency analysis. (a) Training efficiency: accuracy and interaction turns comparison between Search-P1 and Search-R1 during training. (b) Inference efficiency: turns by outcome across dataset types.

Model	RL	Single	Multi	AD
Qwen2.5-3B	GRPO	58.5	28.7	79.5
Qwen2.5-3B	PPO	57.2	27.5	77.8
Llama-3.2-3B	GRPO	56.8	27.1	76.3
Llama-3.2-3B	PPO	55.6	26.2	74.6

Table 3: ACC (%) across base models and RL algorithms. All models use Instruct versions. Per-dataset results are in Appendix F.5 .

Evaluator	ACC	Human Agree. (%)
Evaluator	ACC	Plan	Step	Outc.
HY 2.0-Inst.	47.3	91.2	94.5	88.7
Qwen3-32B	46.5	89.0	92.5	85.0
Qwen3-8B	44.1	83.5	88.0	78.5

Table 4: Effect of LLM evaluator choice on Search-P1 Avg. ACC and human agreement. Per-dataset results are in Appendix F.8 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Prompt template for agentic RAG inference. The model is instructed to plan, search iteratively, and provide structured outputs.

Panel of Figure 8 — Figure 8: Prompt template for reference planner generation. HY 2.0-Instruct generates optimal search strategies for each training sample.

Panel of Figure 9 — Figure 9: Prompt template for dual-track evaluation. The evaluator LLM assesses both self-consistency and reference-alignment of model trajectories.

Panel of Figure 10 — Figure 10: Comparison of reasoning trajectories. Search-R1’s imprecise query retrieves valid but irrelevant results; Search-P1 ’s planning-driven query retrieves the correct information. Highlighted text shows search queries.

Model	# Docs	General QA			Multi-Hop QA				Avg.	Internal
Model	# Docs	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
7B	1	45.8	67.2	38.5	35.2	32.0	15.5	36.0	38.6	72.8
	2	52.0	74.2	44.0	39.5	36.8	18.8	41.0	43.8	80.5
	3	56.6	78.6	47.5	42.9	39.8	21.8	44.0	47.3	86.2
	5	55.8	77.8	46.8	42.5	39.2	22.4	45.2	47.1	85.5
	10	53.5	74.5	44.5	40.5	37.2	20.1	42.8	44.7	82.5

Appendix

Table 5: Performance (ACC %) with different numbers of retrieved documents per search. Retrieving 3 documents achieves the best average performance. While 5 documents shows advantages on specific datasets (MuSiQue, Bamboogle for 7B; HotpotQA, Bamboogle for 3B), the overall best configuration is 3 documents.

Model	Method	General QA			Multi-Hop QA				Internal
Model	Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	AD-QA
7B	w/o Format Reward	82.8	84.2	81.5	79.1	76.8	75.2	83.6	88.2
7B	w/ Format Reward	95.1	95.6	94.2	91.5	88.8	87.6	96.4	97.5
3B	w/o Format Reward	72.8	75.5	72.2	67.1	61.8	62.4	65.2	78.8
3B	w/ Format Reward	87.2	88.1	85.4	80.2	74.8	75.1	78.0	91.5

Appendix

Table 6: Format compliance rate (%) with and without format reward. Adding format reward significantly improves the model’s ability to produce properly structured responses with parseable answers.

Model	Dataset	Successful Cases			Failed Cases
Model	Dataset	1 iter	2 iter	3+ iter	1 iter	2 iter	3+ iter
7B	NQ	68.5%	22.3%	9.2%	45.2%	28.6%	26.2%
	TriviaQA	72.1%	19.8%	8.1%	48.3%	26.4%	25.3%
	PopQA	65.8%	24.5%	9.7%	42.1%	29.8%	28.1%
	HotpotQA	28.5%	48.2%	23.3%	35.4%	30.1%	34.5%
	2Wiki	25.1%	50.6%	24.3%	33.8%	29.5%	36.7%

Appendix

Table 7: Distribution of search iterations for successful and failed cases. General QA datasets (NQ, TriviaQA, PopQA) show high success rates with single-iteration searches, while Multi-Hop QA datasets require more iterations. Failed cases consistently show higher proportions of 3+ iterations, suggesting that excessive searching indicates difficulty in finding relevant information.

Method	General QA			Multi-Hop QA				Avg.	Internal
Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
Qwen2.5-7B
Search-P1 (Full)	56.6	78.6	47.5	42.9	39.8	21.8	44.0	47.3	86.2
w/o Reference-Alignment	50.8	73.0	43.2	37.5	34.0	16.8	39.5	42.1	78.8
w/o Self-Consistency	53.0	75.5	44.8	39.8	37.2	19.2	40.2	44.2	82.5
Search-R1 (Baseline)	42.9	62.3	42.7	38.6	34.6	16.2	40.0	39.6	65.6

Appendix

Table 8: Detailed ablation study on path reward components (ACC %). Removing reference-alignment causes larger drops on multi-hop datasets where external guidance is more critical, while removing self-consistency affects general QA more where the model’s own planning suffices.

Model	RL	General QA			Multi-Hop QA				Avg.	Internal
Model	RL	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
Qwen2.5-3B-Inst.	GRPO	53.0	74.5	47.9	36.2	36.6	13.3	28.8	41.5	79.5
Qwen2.5-3B-Inst.	PPO	51.6	73.1	46.8	34.8	35.4	12.6	27.6	40.3	78.1
Llama-3.2-3B-Inst.	GRPO	50.2	71.8	45.4	33.9	34.1	11.8	26.0	39.0	75.8
Llama-3.2-3B-Inst.	PPO	48.9	70.2	44.2	32.8	33.0	10.9	24.8	37.8	74.2

Appendix

Table 9: Detailed accuracy (%) across different base models and RL algorithms on all datasets. Qwen2.5 consistently outperforms Llama-3.2, and GRPO achieves slightly higher accuracy than PPO across all datasets.

Model	Method	General QA			Multi-Hop QA				Avg.	Internal
Model	Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
7B	w/o Soft Scoring	55.4	77.5	46.0	39.2	36.5	18.5	40.8	44.8	77.4
	w/ Soft Scoring	56.6	78.6	47.5	42.9	39.8	21.8	44.0	47.3	86.2
	$\Delta$	+1.2	+1.1	+1.5	+3.7	+3.3	+3.3	+3.2	+2.5	+8.8
3B	w/o Soft Scoring	51.8	73.2	46.5	32.6	33.1	10.4	24.5	38.9	68.5
	w/ Soft Scoring	53.0	74.5	47.9	36.2	36.6	13.3	28.8	41.5	79.5

Appendix

Table 10: Effect of soft outcome scoring (ACC %). Multi-hop QA datasets benefit more from soft scoring (+3.0–3.7%) compared to general QA datasets (+1.1–1.5%), while the internal AD-QA dataset shows the largest improvement (+8.8–11.0%), confirming that complex enterprise queries benefit most from partial credit signals.

$\lambda_{p}$	General QA			Multi-Hop QA				Avg.	Internal
$\lambda_{p}$	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
0.2	55.5	77.8	46.8	41.8	38.5	20.8	44.5	46.5	83.5
0.3	56.6	78.6	47.5	42.9	39.8	21.8	44.0	47.3	86.2
0.4	55.8	78.0	47.2	42.2	39.2	21.5	43.8	46.8	85.0

Appendix

Table 11: Effect of path reward weight

\lambda_{p}

on performance (ACC %, Qwen2.5-7B). The optimal value is

\lambda_{p}=0.3

, which achieves the best average performance. While

\lambda_{p}=0.2

shows slight advantage on Bamboogle,

\lambda_{p}=0.3

provides the best overall balance.

$\lambda_{a}$	General QA			Multi-Hop QA				Avg.	Internal
$\lambda_{a}$	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
0.6	56.6	78.6	47.5	42.9	39.8	21.8	44.0	47.3	86.2
0.8	56.0	78.2	47.0	42.5	39.2	22.2	44.8	47.1	85.5
1.0	54.8	76.8	45.8	41.2	38.0	20.8	43.2	45.8	83.2

Appendix

Table 12: Effect of outcome accuracy weight

\lambda_{a}

on performance (ACC %, Qwen2.5-7B). The optimal value is

\lambda_{a}=0.6

, which achieves the best average performance. While

\lambda_{a}=0.8

shows slight advantages on MuSiQue and Bamboogle,

\lambda_{a}=0.6

provides better overall results.

LLM Evaluator	General QA			Multi-Hop QA				Avg.	Internal
LLM Evaluator	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg.	AD-QA
HY 2.0-Instruct	56.6	78.6	47.5	42.9	39.8	21.8	44.0	47.3	86.2
Qwen3-32B	55.5	77.5	46.2	41.8	38.8	20.8	42.5	46.2	84.2
Qwen3-8B	52.8	74.2	43.5	39.2	35.8	17.8	39.8	43.3	79.5

Appendix

Table 13: Detailed accuracy (%) across LLM evaluators (Qwen2.5-7B + GRPO). Qwen3-32B shows modest degradation (

-

1.1 Avg.), while Qwen3-8B exhibits larger drops on multi-hop tasks where step coverage evaluation is more challenging.

Explore similar work

AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing
Aug 30, 2026Xinke Jiang, Yue Fang, Zhibang Yang +12Retrieval-Augmented Generation Agentic RAG
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +3
APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation
Mar 14, 2026Kun Chen, Qingchao Kong, Zhao Feifei +1Agentic RAG Multi-Hop QA
University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Wenge Technology Co., Ltd
MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG
Aug 25, 2026Qiuyi Qi, Tian Liang, Jiamu Wang +7Agentic RAG RL for LLM Agents
Zhejiang University · Ant Group · City University of Hong Kong

Aug 30, 2026J/K move · Enter open · S save

AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing

Xinke Jiang, Yue Fang, Zhibang Yang, Jiaran Gao+11

National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China+3

Retrieval-Augmented Generation Agentic RAG Long-Horizon LLM Agents RL for LLM Agents Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) improves the factuality of large language models (LLMs), yet existing RAG systems often struggle with complex, multi-step reasoning that requires adaptive retrieval and continuous revision of intermediate contexts. Recent reinforcement learning (RL)-based agentic RAG methods partially alleviate this issue, but typically rely on coarse-grained action spaces and trajectory-level rewards, resulting in weak reward assignment and a bias toward short-horizon, stereotyped reasoning template. To address, we propose AgenticRag-R1, a RL framework that deeply integrates reasoning, retrieval, and memory via a memory stack and fine-grained action space, supported by hierarchical action-aware rewards and an information-aware trajectory rejection strategy to enable effective long-horizon learning. Experiments across a diverse set of multi-hop, open-domain, and agentic reasoning benchmarks, spanning multiple backbone model sizes, demonstrate that AgenticRag-R1 consistently outperforms strong baselines. Moreover, AgenticRag-R1 learns more robust, interpretable, and memory-aware reasoning behaviors, highlighting the effect of fine-grained action modeling and information-aware optimization for long-horizon reasoning. Our code is anonymous available at https://github.com/jiangxinke/Harness-RL/tree/AgenticRAG-R1-Whitebox.

Abstract

Explore similar work

AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing

APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation

MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG