APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation

Kun Chen; Qingchao Kong; Zhao Feifei; Wenji Mao

cs.CLMar 14, 2026

APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation

Authors: Kun Chen, Qingchao Kong, Zhao Feifei, Wenji Mao

Organizations: University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Wenge Technology Co., Ltd

Abstract

Retrieval-augmented generation (RAG) connects large language models (LLMs) to external knowledge, but single-round retrieval is often insufficient for complex multi-hop questions. To enhance search capabilities for complex tasks, most existing works integrate multi-round iterative retrieval with reasoning processes via end-to-end training. While these approaches improve problem-solving performance, they still face challenges in task reasoning and model training, especially ambiguous retrieval execution paths and sparse rewards in end-to-end reinforcement learning (RL), which can lead to inaccurate retrieval results and lower performance. We attribute these failures to hierarchical credit entanglement: a single final reward updates planning and execution together, so the model cannot clearly separate plan errors from retrieval errors. We propose APEX-Searcher, which uses a Refining Credit Assignment paradigm: planning is optimized by RL with a plan-level reward, while execution is learned by SFT. Extensive experiments show consistent gains in both multi-hop RAG and task planning across benchmarks.

Figures & tables

Figure 1: Overview of the APEX-Searcher framework. The architecture utilizes RL-driven agentic planning in Stage I to decompose a complex question into a multi-step plan. Subsequently, Stage II employs SFT-guided execution to solve each sub-question using an iterative retrieval loop that features dynamic continuation decisions and context management for final synthesis. See Figure 2 for an example.

Figure 2: An example of two stage walkthrough of the APEX-Searcher pipeline. A complete walkthrough of the multi-hop reasoning trace is provided in Figure 6 (Appendix F )

Methods	Type	HotpotQA		2Wiki		MuSiQue		Bamboogle		Avg.
Methods	Type	7B	3B	7B	3B	7B	3B	7B	3B	7B	3B
Direct Inference	Non-Retrieval	0.183	0.149	0.250	0.244	0.031	0.020	0.120	0.024	0.146	0.109
CoT	Non-Retrieval	0.092	0.021	0.111	0.021	0.022	0.002	0.232	0.000	0.114	0.011
RAG	Standard RAG	0.299	0.255	0.235	0.226	0.058	0.047	0.208	0.080	0.200	0.152
IRCoT	Iterative RAG	0.133	0.164	0.149	0.171	0.072	0.067	0.224	0.240	0.145	0.161
Search-o1	Agentic RAG	0.187	0.221	0.176	0.218	0.058	0.054	0.296	0.320	0.179	0.203

Table 1: Main results comparison between Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct across different benchmarks. The highest score for each Benchmark is marked in bold.

Model	Method (Components)			HotpotQA	2Wiki	MuSiQue	Bamboogle
Model	Plan	Plan RL	Exec SFT	HotpotQA	2Wiki	MuSiQue	Bamboogle
Qwen-2.5-7B-Instruct	✗	-	✗	35.11	31.27	8.61	35.20
	✗	-	✓	35.18	35.69	9.56	36.00
	✓	✗	✗	33.52	42.92	11.50	33.60
	✓	✓	✗	36.79	46.30	12.99	39.20
	✓	✗	✓	35.76	48.72	12.12	37.60

Table 2: Ablation study of different components on 7B and 3B models. Plan : Planning module; RL : Planning with RL; Exec SFT : Execution via Supervised Fine-Tuning.

Figure 3: Parameter Sensitivity Analysis on APEX-Searcher. The curves illustrate the impact of the number of retrieved documents (#Doc) and the maximum allowed reasoning hops (#Hop) on model accuracy across four benchmarks. The asterisk (

\star

) denotes the optimal parameter configuration selected for this study and its corresponding performance.

Panel of Figure 4 — Figure 4: (a) shows the Plan score improvements over base models, (b-c) show the reward score convergence during training for the 3B and 7B variants, and (d) shows the optimization of response length over training steps.

Case 1: Genealogy
Question	Who is the maternal grandfather of Antiochus X Eusebes?
Baseline	• Who is the mother of Antiochus X Eusebes? • Who is the father of Antiochus X Eusebes’s mother ? • What is the relationship of the answer to #2 to Antiochus X Eusebes?
Ours	- Who is the mother of Antiochus X Eusebes? - Who is the father of #1? (Who is the father of Cleopatra IV?)

Table 3: Case Study Comparison of Question Decomposition

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Comparison of question decomposition strategies. Our method produces more concise decompositions by effectively using reference notation ( #1 , #2 ) to chain sub-questions, while the baseline often generates redundant or incomplete decompositions.

Panel of Figure 6 — Figure 6: Complete interaction trace showing the multi-hop reasoning process. The system decomposes the complex question, iteratively retrieves information for each sub-question, and synthesizes the final answer through structured prompting.

Explore similar work

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training
Feb 26, 2026Tianle Xia, Ming Xu, Lingxiang Hu +7Agentic RAG LLM Agent Training
Tencent
Plan Before Search: Search Agents Need Plan
May 27, 2026Zhipeng Qian, Zihan Liang, Yufei Ma +7Multi-Hop QA Tool-Augmented Language Model Agents
1Kuaishou Technology · 2Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · 3VCIP, CS, Nankai University
AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning
Apr 19, 2026Jingbo Sun, Wenyue Chong, Songjun Tu +7Agentic Retrieval Agentic RAG
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

Feb 26, 2026J/K move · Enter open · S save

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun+6

Tencent

Agentic RAG LLM Agent Training Reward Shaping

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge, yet traditional single-round retrieval struggles with complex multi-step reasoning. Agentic RAG addresses this by enabling LLMs to dynamically decide when and what to retrieve, but current RL-based training methods suffer from sparse outcome rewards that discard intermediate signals and low sample efficiency where failed samples contribute nothing. We propose Search-P1, a framework that introduces path-centric reward shaping for agentic RAG training, comprising two key components: (1) Path-Centric Reward, which evaluates the structural quality of reasoning trajectories through order-agnostic step coverage and soft scoring that extracts learning signals even from failed samples, and (2) Dual-Track Path Scoring with offline-generated reference planners that assesses paths from both self-consistency and reference-alignment perspectives. Experiments on multiple QA benchmarks demonstrate that Search-P1 achieves significant improvements over Search-R1 and other strong baselines, with an average accuracy gain of 7.7 points.

Abstract

Explore similar work

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Plan Before Search: Search Agents Need Plan

AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning