Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

Saahil Mathur; Ryan David Rittner; Vedant Ajit Thakur; Daniel Stuart Schiff; Tunazzina Islam

cs.CLMar 25, 2026

Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

Authors: Saahil Mathur, Ryan David Rittner, Vedant Ajit Thakur, Daniel Stuart Schiff, Tunazzina Islam

Organizations: Department of Computer Science, Purdue University, West Lafayette, IN 47907 · Department of Political Science, Purdue University, West Lafayette, IN 47907

Abstract

Retrieval-augmented generation (RAG) systems are increasingly used to analyze complex policy documents, but achieving sufficient reliability for expert usage remains challenging in domains characterized by dense legal language and evolving, overlapping regulatory frameworks. We study the application of RAG to AI governance and policy analysis using the AI Governance and Regulatory Archive (AGORA) corpus, a curated collection of 947 AI policy documents. Our system combines a ColBERT-based retriever fine-tuned with contrastive learning and a generator aligned to human preferences using Direct Preference Optimization (DPO). We construct synthetic queries and collect pairwise preferences to adapt the system to the policy domain. Through experiments evaluating retrieval quality, answer relevance, and faithfulness, we find that domain-specific fine-tuning improves retrieval metrics but does not consistently improve end-to-end question answering performance. In some cases, stronger retrieval counterintuitively leads to more confident hallucinations when relevant documents are absent from the corpus. These results highlight a key concern for those building policy-focused RAG systems: improvements to individual components do not necessarily translate to more reliable answers. Our findings provide practical insights for designing grounded question-answering systems over dynamic regulatory corpora.

Figures & tables

Figure 1: Overview of the AI policy RAG system.

Retriever/Generator	Relevancy	Accuracy
Base Mistral (w/o RAG)	0.733	0.581
Base ColBERT/Base Mistral	0.746	0.605
CL ColBERT/Base Mistral	0.744	0.580
Base ColBERT/DPO Mistral	0.747	0.601
CL ColBERT/DPO Mistral	0.700	0.559
GPT-5.4	0.744	0.777

Table 1: End-to-end QA performance.

Retriever	MRR	Recall@5	Recall@10	Recall@20	MAP@5	MAP@10	MAP@20
ColBERTv2-base	0.641757	0.220146	0.347234	0.585305	0.456694	0.441440	0.481813
Mined negatives	0.748333	0.289443	0.384164	0.483517	0.584472	0.561838	0.561443
Labeled negatives	0.727095	0.267648	0.397876	0.506035	0.504367	0.517485	0.502683
Mixed negatives	0.725857	0.300559	0.401189	0.484138	0.528367	0.492509	0.476172

Table 2: Retriever performance.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Data distribution.

Figure 6: Generator DPO fine-tuning pipeline.

Panel of Figure 7 — Figure 7: Generator optimization prompts.

Panel of Figure 8 — Figure 8: Retriever fine-tuning prompts.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG
Aug 25, 2026Qiuyi Qi, Tian Liang, Jiamu Wang +7Agentic RAG RL for LLM Agents
Zhejiang University · Ant Group · City University of Hong Kong
How Much Do Legal RAG Systems Still Hallucinate?
Aug 14, 2026Souvick Das, Sallam Abualhaija, Domenico BianculliHallucination in Language Models Retrieval-Augmented Generation
SnT, University of Luxembourg Luxembourg
Carolina Guide: A Multi-Agent RAG System with Institutional Guardrails for Academic Policy Assistance
Jun 11, 2026Ben Torsion, Jun ZhouMulti-Agent LLM Systems LLM Guardrails
University of South Carolina, Columbia, South Carolina, USA · University of South Carolina, Columbia, USA

Aug 25, 2026J/K move · Enter open · S save

MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG

Qiuyi Qi, Tian Liang, Jiamu Wang, Jinjian Zhang+6

Zhejiang University · Ant Group · City University of Hong Kong

Agentic RAG RL for LLM Agents Knowledge-Intensive QA RL for IR

Open paper AskPDF arXiv

Agentic retrieval-augmented generation (RAG) requires language models to decide when to continue searching and when to answer. Existing RL-based methods rely on external supervision and overlook the agent's internal belief about whether the current evidence is sufficient. To address this problem, we reformulate the search decision quality as belief-action alignment and propose MetaRAG, a belief-action aligned policy optimization framework for agentic RAG. MetaRAG uses Verify-first Action Generation to elicit an explicit verification process before each actual action, and Internal Belief Probing to estimate the policy model's own answerability belief from the same question-history context. Based on these, MetaRAG derives a consistency reward that is further gated by answer correctness, avoiding reinforcement of internally consistent but incorrect trajectories. The belief probe is used only during training and introduces no inference-time overhead. Experiments on seven public QA benchmarks show that MetaRAG consistently improves the accuracy-efficiency trade-off over strong RL-based agentic RAG baselines, with gains that transfer to deep research settings, different optimizers, and multiple model backbones.

Abstract

Retrieval-Augmented Generation

Explore similar work

MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG

How Much Do Legal RAG Systems Still Hallucinate?

Carolina Guide: A Multi-Agent RAG System with Institutional Guardrails for Academic Policy Assistance