MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG

Qiuyi Qi; Tian Liang; Jiamu Wang; Jinjian Zhang; Wei Zhou; Pengcheng Zhu; Linjian Mo; Ming Kong

cs.AIAug 25, 2026

MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG

Authors: Qiuyi Qi, Tian Liang, Jiamu Wang, Jinjian Zhang, Wei Zhou, Pengcheng Zhu, Linjian Mo, Ming Kong, +2 more

Organizations: Zhejiang University · Ant Group · City University of Hong Kong

Abstract

Agentic retrieval-augmented generation (RAG) requires language models to decide when to continue searching and when to answer. Existing RL-based methods rely on external supervision and overlook the agent's internal belief about whether the current evidence is sufficient. To address this problem, we reformulate the search decision quality as belief-action alignment and propose MetaRAG, a belief-action aligned policy optimization framework for agentic RAG. MetaRAG uses Verify-first Action Generation to elicit an explicit verification process before each actual action, and Internal Belief Probing to estimate the policy model's own answerability belief from the same question-history context. Based on these, MetaRAG derives a consistency reward that is further gated by answer correctness, avoiding reinforcement of internally consistent but incorrect trajectories. The belief probe is used only during training and introduces no inference-time overhead. Experiments on seven public QA benchmarks show that MetaRAG consistently improves the accuracy-efficiency trade-off over strong RL-based agentic RAG baselines, with gains that transfer to deep research settings, different optimizers, and multiple model backbones.

Figures & tables

Figure 1: Comparison of three supervision paradigms in RL-based agentic RAG. MetaRAG provides lightweight decision-level supervision by aligning each Search/Answer action with the policy model’s own belief about evidence sufficiency, improving boundary calibration.

Figure 2: Overview of MetaRAG . At each decision step, the policy model produces an Actual Action through Verify-first Action Generation, while Internal Belief Probing estimates a Belief Score from a separate yes/no Forward Pass. Belief-Action Gap Diagnosis maps the relation between the Belief Score and the Actual Action into a Step-level Diagnosis, and Consistency Reward Calculation combines $R_{{outcome}}$ with $R_{{consist}}$ into $R_{{total}}$ for training.

Method	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Qwen2.5-3B-Instruct
Search-R1	39.7	1.55	56.5	1.53	39.1	1.75	33.1	2.78	31.0	3.38	12.4	3.39	23.2	2.79	33.6	2.45
HiPRAG	43.0	1.01	59.8	1.01	42.0	1.02	36.0	2.14	40.5	2.48	10.8	2.55	24.0	2.02	36.6	1.75
GiGPO ‡	45.1	1.00	60.9	0.99	44.9	1.17	38.0	1.24	37.1	1.55	14.5	1.81	37.9	1.49	39.8	1.32

Table 1: Main results on seven public QA benchmarks.

\dagger

and

\star

indicate in-domain and out-of-domain datasets, respectively. Acc. denotes Exact Match (EM) accuracy (%), and Searches denotes the average number of search calls per question. Best and second-best accuracy values are bold and underlined . Results marked with

\ddagger

are reproduced by us using the same retrieval setup; the remaining baseline results are taken from Xia et al. (2026) .

Figure 3: Training dynamics on Qwen2.5-3B-Instruct. Solid lines denote exponential moving averages in all panels except the Accuracy panel. From left to right, the panels show accuracy, rewards, average search count, and under-search rate. MetaRAG initially trails GiGPO without SFT warm-up for the new verify-before-action reasoning pattern, but later surpasses it in both accuracy and outcome reward. Its consistency reward increases steadily, and its under-search rate remains mostly lower after about 100 steps.

Core Components
Variant	Acc.	Searches
w/o Consistency Reward	41.9	2.14
w/o Verify-before-Action	40.4	1.38
Diagnostic Signal
Over-Search Only	41.3	1.33
Under-Search Only	42.4	2.56

Table 2: Ablation study on Qwen2.5-3B-Instruct. Results are averaged over seven QA benchmarks.

Figure 4: Hyperparameter sensitivity on Qwen2.5-3B-Instruct. Left: consistency reward weight $\lambda$ . Right: margin $m$ . Dashed lines indicate GiGPO.

Figure 5: Perplexity-based knowledge-boundary awareness attribution on Qwen2.5-7B-Instruct. We report average AUROC and PRR over GSM8K, SciQ, and TriviaQA. Higher values indicate stronger separation between answerable and non-answerable queries.

Method	Acc. (%)	Rec. (%)	Searches
Qwen3-32B
Base	3.61	3.12	0.92
Qwen2.5-7B-Instruct
Search-R1	1.45	2.78	2.38
HiPRAG	1.69	2.71	2.62
GiGPO	2.65	3.10	2.32

Table 3: Zero-shot transfer to BrowseComp-Plus with BM25 retriever. Acc. denotes answer accuracy, Rec. denotes evidence recall, and Searches denotes the average number of search calls per question. Best and second-best Acc./Rec. values are bold and underlined .

Setting	Method	Acc.	Searches
RL Optimizer on Qwen2.5-3B-Instruct
GRPO	GiGPO	39.8	1.32
GRPO	MetaRAG	41.7	1.60
DAPO	GiGPO	44.8	2.31
DAPO	MetaRAG	45.7	2.51
PPO	MetaRAG	41.1	1.45

Table 4: Robustness across training settings. Results are averaged over seven QA benchmarks. Llama3.2-3B denotes Llama-3.2-3B-Instruct.

Belief Probe	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Qwen2.5-3B-Instruct
$P(\mathrm{Yes})-P(\mathrm{No})$	45.6	1.17	62.8	1.19	48.0	1.27	41.3	1.58	39.1	1.87	15.9	2.24	39.5	1.90	41.7	1.60
Internal Confidence	48.0	1.47	62.5	1.54	48.7	1.58	42.3	1.79	41.2	1.83	16.6	2.26	37.9	1.94	42.5	1.77

Table 5: Robustness to alternative belief probes on Qwen2.5-3B-Instruct. We compare the default

P(\mathrm{Yes})-P(\mathrm{No})

belief score with Internal Confidence, a stronger uncertainty estimator that aggregates yes/no self-evaluation signals across internal layers and token positions. For Internal Confidence, we use answerability threshold

\eta=0.5

. Acc. denotes Exact Match (EM) accuracy (%), and Searches denotes the average number of search calls per question. Best accuracy values are bold .

Method	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Qwen2.5-3B-Instruct
$\beta$ -GRPO	39.3	1.28	59.3	0.90	35.7	1.50	31.6	2.45	39.1	2.85	10.7	2.50	34.4	2.15	35.7	2.15
MetaRAG	45.6	1.17	62.8	1.19	48.0	1.27	41.3	1.58	39.1	1.87	15.9	2.24	39.5	1.90	41.7	1.60

Table 6: Comparison with confidence-threshold search training on Qwen2.5-3B-Instruct.

\beta

-GRPO uses search-query token confidence to reward high-certainty search behavior, while MetaRAG uses answerability belief-action alignment to supervise Search/Answer decisions. Acc. denotes Exact Match (EM) accuracy (%), and Searches denotes the average number of search calls per question. Best accuracy values are bold .

Method	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Qwen2.5-3B-Instruct
GiGPO	45.1	1.00	60.9	0.99	44.9	1.17	38.0	1.24	37.1	1.55	14.5	1.81	37.9	1.49	39.8	1.32
MetaRAG (GRPO)	45.6	1.17	62.8	1.19	48.0	1.27	41.3	1.58	39.1	1.87	15.9	2.24	39.5	1.90	41.7	1.60
MetaRAG (GiGPO)	47.3	1.30	63.0	1.31	48.8	1.40	43.2	1.73	42.9	2.08	16.4	2.10	40.1	1.74	43.1	1.67

Table 7: Compatibility with GiGPO step-level credit assignment. Results use the same setting as Table 1 .

\dagger

and

\star

Method	External Under-Search Rate (%)
Qwen2.5-3B-Instruct
GiGPO	35.6
MetaRAG	30.1

Table 8: External LLM-as-judge validation of under-search rate on Qwen2.5-3B-Instruct 400-step checkpoints. We evaluate the full set of test trajectories using GLM-5.1. Lower is better. The judge is used only for post-hoc assessment and is not involved in reward computation, training, or model selection.

Variant	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Core Components
w/o Consistency Reward	46.5	2.03	62.4	2.04	48.1	2.08	41.7	2.12	41.0	2.20	16.0	2.34	37.9	2.14	41.9	2.14
w/o Verify-before-Action	45.6	1.09	60.4	1.12	45.0	1.21	40.1	1.38	39.0	1.60	15.4	1.70	37.5	1.57	40.4	1.38
Diagnostic Signal

Table 9: Detailed ablation study results on Qwen2.5-3B-Instruct.

\dagger

and

\star

indicate in-domain and out-of-domain datasets, respectively. Acc. denotes Exact Match (EM) accuracy (%), and Searches denotes the average number of search calls per question.

Method	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Qwen2.5-3B-Instruct
GiGPO	45.1	1.00	60.9	0.99	44.9	1.17	38.0	1.24	37.1	1.55	14.5	1.81	37.9	1.49	39.8	1.32
GiGPO w/ VF-Inf	44.7	1.04	60.8	1.03	44.4	1.17	38.4	1.28	36.6	1.63	13.3	1.85	37.2	1.52	39.3	1.36
MetaRAG	45.6	1.17	62.8	1.19	48.0	1.27	41.3	1.58	39.1	1.87	15.9	2.24	39.5	1.90	41.7	1.60

Table 10: Effect of verify-first inference on Qwen2.5-3B-Instruct. VF-Inf denotes using Verify-first Action Generation at inference time. Acc. denotes Exact Match (EM) accuracy (%), and Searches denotes the average number of search calls per question.

Setting	Single-Hop QA						Multi-Hop QA								Avg.
	NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
Consistency Reward Weight $\lambda$
$\lambda=0.0$	46.5	2.03	62.4	2.04	48.1	2.08	41.7	2.12	41.0	2.20	16.0	2.34	37.9	2.14	41.9	2.14
$\lambda=0.1$ (default)	45.6	1.17	62.8	1.19	48.0	1.27	41.3	1.58	39.1	1.87	15.9	2.24	39.5	1.90	41.7	1.60
$\lambda=0.2$	44.5	1.22	61.9	1.20	45.3	1.38	40.4	1.60	38.9	2.08	15.7	2.15	40.3	1.80	41.0	1.63

Table 11: Detailed hyperparameter sensitivity results on Qwen2.5-3B-Instruct.

\dagger

and

\star

indicate in-domain and out-of-domain datasets, respectively. Acc. denotes Exact Match (EM) accuracy (%), and Searches denotes the average number of search calls per question.

Perplexity
Method	GSM8K		SciQ		TriviaQA		Avg.
Method	AUROC	PRR	AUROC	PRR	AUROC	PRR	AUROC	PRR
Base	56.0	16.1	60.1	22.7	62.5	22.6	59.5	20.5
Search-R1	54.4	10.1	59.8	22.3	61.4	21.2	58.5	17.9
HiPRAG	52.2	2.7	57.9	18.9	62.4	24.4	57.5	15.3
GiGPO	56.2	13.8	56.8	14.6	63.7	26.7	58.9	18.4

Table 12: Detailed attribution analysis results on Qwen2.5-7B-Instruct. We evaluate knowledge-boundary awareness using perplexity and attentional entropy as diagnostic signals, and report AUROC and PRR on GSM8K, SciQ, and TriviaQA. Avg. denotes the average over the three datasets. Best and second-best values are bold and underlined within each diagnostic-signal block.

Setting	Method	Single-Hop QA						Multi-Hop QA								Avg.
		NQ †		TriviaQA ⋆		PopQA ⋆		HotpotQA †		2Wiki ⋆		MuSiQue ⋆		Bamboogle ⋆		Avg.
		Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches	Acc.	Searches
RL Optimizer on Qwen2.5-3B-Instruct
GRPO	GiGPO	45.1	1.00	60.9	0.99	44.9	1.17	38.0	1.24	37.1	1.55	14.4	1.81	37.9	1.49	39.8	1.32
GRPO	MetaRAG	45.6	1.17	62.8	1.19	48.0	1.27	41.3	1.58	39.1	1.87	15.9	2.24	39.5	1.90	41.7	1.60
DAPO	GiGPO	47.3	2.01	63.6	1.91	48.3	2.13	46.2	2.30	46.6	2.56	19.8	2.78	42.0	2.46	44.8	2.31

Table 13: Detailed robustness analysis results across training settings.

\dagger

and

\star

Method	Train / Step	Resp. Len.	Infer. / Query
Qwen2.5-7B-Instruct
HiPRAG	525s	151.4	0.1899s
GiGPO	249s	104.6	0.1179s
MetaRAG	281s	119.7	0.1453s

Table 14: Training and inference efficiency on Qwen2.5-7B-Instruct. All measurements are averaged over runs on a single node with 8 A100 GPUs. Train / Step denotes the wall-clock time for one training step. Resp. Len. denotes the average response length per turn during agentic RAG inference. Infer. / Query denotes the average end-to-end inference time per query.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Over-search diagnosed by Belief-Action Gap Diagnosis. The correct answer Dead Man’s Shoes is already available after the first retrieval, and the Belief Score is strongly positive, but the agent continues searching and eventually returns an incorrect answer.

Panel of Figure 7 — Figure 7: Under-search diagnosed by Belief-Action Gap Diagnosis. The retrieved evidence identifies Vivian Campbell as the guitarist but does not provide his nationality; despite a negative Belief Score, the agent answers by conflating the band’s nationality with the guitarist’s personal background.

Panel of Figure 8 — Figure 8: The prompt template for Verify-first Action Generation.

Panel of Figure 9 — Figure 9: The prompt template for Internal Belief Probing.

Panel of Figure 10 — Figure 10: The prompt template for External Under-Search Judge.

Panel of Figure 11 — Figure 11: Under-search failure of GiGPO. The agent retrieves that Mold Castle was erected under instructions from Hugh d’Avranches, Earl of Chester, but prematurely returns the title “Earl of Chester” instead of searching for the requested alias.

Panel of Figure 12 — Figure 12: MetaRAG avoids under-search on the same question. The agent recognizes that the first retrieval identifies the Earl but does not yet resolve the requested alias, leading to a targeted follow-up search and the correct answer.

Explore similar work

GRASP: GRanularity-Aware Search Policy for Agentic RAG
Jul 11, 2026Varun Gandhi, Jaewook Lee, Shantanu Todmal +4Multi-Hop QA Retrieval-Augmented Generation
University of Massachusetts Amherst · Adobe Research
Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training
Feb 26, 2026Tianle Xia, Ming Xu, Lingxiang Hu +7Agentic RAG LLM Agent Training
Tencent
AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing
Aug 30, 2026Xinke Jiang, Yue Fang, Zhibang Yang +12Retrieval-Augmented Generation Agentic RAG
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +3

Jul 11, 2026J/K move · Enter open · S save

GRASP: GRanularity-Aware Search Policy for Agentic RAG

Varun Gandhi, Jaewook Lee, Shantanu Todmal, Franck Dernoncourt+3

University of Massachusetts Amherst · Adobe Research

Multi-Hop QA Retrieval-Augmented Generation Adaptive RAG Agentic RAG RL for LLM Agents

Open paper AskPDF arXiv

Agentic retrieval-augmented generation (RAG) extends static RAG by allowing language models to iteratively reason, generate search queries, retrieve evidence, and predict answers. However, it remains challenging for models to decide when to retrieve, whether to use lexical matching or semantic similarity, and how to control context granularity to prevent irrelevant tokens from interfering with agent reasoning. In this paper, we introduce GRASP, a reinforcement learning (RL) framework for training agents to adaptively coordinate complementary retrieval tools during multi-step reasoning. GRASP provides the agent with semantic search, keyword search, and paragraph-reading actions, enabling it to retrieve sentence-level evidence and expand further context only when needed. We train the policy with a reward that jointly accounts for answer accuracy, grounded reading, complementary search, and turn efficiency. Experiments on multi-hop reasoning benchmarks show that GRASP improves both retrieval recall and downstream question answering performance compared with single-step retrieval, prompting-based agentic RAG, and RL-based retrieval baselines. Qualitative and ablation analyses show that the learned policy develops interpretable skimming and scanning behavior: it uses semantic search for broad exploration, paragraph reading for local verification, and keyword search for entity-specific evidence. These results suggest that learning to coordinate retrieval signals and context granularity is critical for agent's correct reasoning.

Abstract

Explore similar work

GRASP: GRanularity-Aware Search Policy for Agentic RAG

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing