Offline RL for Adaptive Policy Retrieval in Prior Authorization

Ruslan Sharifullin; Maxim Gorshkov; Hannah Clay

cs.IRApr 6, 2026

Offline RL for Adaptive Policy Retrieval in Prior Authorization

Authors: Ruslan Sharifullin, Maxim Gorshkov, Hannah Clay

Organizations: Stanford University.

Abstract

Prior authorization (PA) requires interpretation of complex and fragmented coverage policies, yet existing retrieval-augmented systems rely on static top- $K$ strategies with fixed numbers of retrieved sections. Such fixed retrieval can be inefficient and gather irrelevant or insufficient information. We model policy retrieval for PA as a sequential decision-making problem, formulating adaptive retrieval as a Markov Decision Process (MDP). In our system, an agent iteratively selects policy chunks from a top- $K$ candidate set or chooses to stop and issue a decision. The reward balances decision correctness against retrieval cost, capturing the trade-off between accuracy and efficiency. We train policies using Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Direct Preference Optimization (DPO) in an offline RL setting on logged trajectories generated from baseline retrieval strategies over synthetic PA requests derived from publicly available CMS coverage data. On a corpus of 186 policy chunks spanning 10 CMS procedures, CQL achieves 92% decision accuracy (+30 percentage points over the best fixed- $K$ baseline) via exhaustive retrieval, while IQL matches the best baseline accuracy using 44% fewer retrieval steps and achieves the only positive episodic return among all policies. Transition-level DPO matches CQL's 92% accuracy while using 47% fewer retrieval steps (10.6 vs. 20.0), occupying a "selective-accurate" region on the Pareto frontier that dominates both CQL and BC. A behavioral cloning baseline matches CQL, confirming that advantage-weighted or preference-based policy extraction is needed to learn selective retrieval. Lambda ablation over step costs $λ\in \{0.05, 0.1, 0.2\}$ reveals a clear accuracy-efficiency inflection: only at $λ= 0.2$ does CQL transition from exhaustive to selective retrieval.

Figures & tables

Panel of Figure 1 — Figure 1: System architecture. The agent iteratively retrieves policy chunks from a Sentence-BERT-indexed corpus and decides when to stop. The oracle evaluates the PA request using only retrieved chunks.

Procedure	Chunks	Shared	Train	Test
45378 (Colonoscopy)	30	0	201	23
70450 (CT Head)	56	54	195	27
70486 (CT Maxfac.)	43	43	199	19
70553 (MRI Brain)	54	54	208	28
71260 (CT Chest)	44	42	205	20
72148 (MRI Lumbar)	55	49	178	20

Table 1: Corpus and dataset composition per procedure. Shared = chunks appearing in multiple procedures’ retrieval pools.

Policy	Accuracy	Return	Steps
DPO (transition)	92.0%	$\mathbf{-0.12}$	10.6
CQL	92.0%	$-1.06$	20.0
BC (imitation)	92.0%	$-1.06$	20.0
IQL	62.5%	$+0.01$	3.4
FixedK( $k=5$ )	62.0%	$-0.26$	6.0
FixedK( $k=3$ )	54.0%	$-0.22$	4.0

Table 2: On-policy evaluation (200 episodes, 186 chunks)

$\lambda$	Accuracy	Steps	Behavior
0.05	92.0%	20.0	Exhaustive
0.1 (default)	92.0%	20.0	Exhaustive
0.2	91.5%	14.9	Selective

Table 3: CQL lambda ablation (200 episodes)

Figure 2: Lambda ablation: accuracy, mean steps, and mean return for CQL across step cost values

\lambda\in\{0.05,0.1,0.2\}

. At

\lambda=0.2

, CQL transitions from exhaustive to selective retrieval.

$\beta$	Epochs	Accuracy	Return	Steps
0.5	500	79.5%	$-0.45$	11.4
1.0	1000	85.5%	$-0.31$	11.2
3.0	2000	92.0%	$\mathbf{-0.12}$	10.6

Table 4: DPO beta ablation (200 episodes, transition-level)

Figure 3: Training curves for CQL (200 epochs) and IQL (1000 epochs). Left: CQL total loss and TD loss. Right: IQL Q-network TD loss and policy loss. All metrics converge without divergence.

Figure 4: DPO training curves. Left: BC warmup cross-entropy loss (200 epochs), converging from 2.38 to 0.88. Right: DPO loss (blue) and preference accuracy (red) over 2000 epochs. Preference accuracy reaches 85%, indicating the policy reliably assigns higher probability to preferred actions.

Policy	WIS	FQE Mean $\hat{Q}$
CQL	$-0.055$	$-2.22$
DPO	$\mathbf{-0.012}$	$\mathbf{-0.34}$
IQL	$-0.021$	$-0.17$
BC	$-0.084$	$-2.41$

Table 5: Off-policy evaluation estimates

Comparison	$\Delta$ Acc	$p$ -value	95% CI
DPO vs CQL	0.0pp	–	[0.0, 0.0]
DPO vs IQL	+29.5pp	$8.1\times 10^{-17}$	[23.5, 36.0]
DPO vs BC	0.0pp	–	[0.0, 0.0]
CQL vs IQL	+29.5pp	$8.1\times 10^{-17}$	[23.5, 36.0]
CQL vs BC	0.0pp	–	[0.0, 0.0]
CQL vs FixedK(3)	+38.0pp	$2.0\times 10^{-22}$	[31.5, 45.0]

Table 6: Statistical significance: pairwise comparisons

Figure 5: Accuracy vs. retrieval steps for all seven policies. DPO achieves (10.6, 92%) on the Pareto frontier, dominating CQL and BC at (20, 92%). IQL achieves (3.4, 62.5%).

Figure 6: Procedure-level retrieval frequency (mean chunks retrieved per episode). CQL/BC retrieve broadly across all procedures; DPO retrieves selectively while maintaining CQL-level accuracy; IQL retrieves sparsely, focusing on high-relevance chunks.

Figure 7: Per-procedure accuracy breakdown for all seven policies. CQL, BC, and DPO achieve 100% on 7/10 procedures; three imaging procedures with high cross-procedure semantic interference remain challenging for all policies.

Adaptive Retrieval

Adaptive retrieval adjusts what to retrieve, which strategy to use, and when to stop based on query needs and available evidence.

See all 37 papers

Jun 22, 2026

When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model. We test this proxy for pre-action policy classification in

τ

-bench using Qwen2.5-3B/7B classifiers. Under gold-policy conditioning, a compact structured state improves macro-F1 over raw trajectories by

0.20

after tuning at 3B, with the same ordering at 7B under shared hyperparameters. We then replace the benchmark-designated governing rule with the top-ranked benchmark assertion retrieved from decision-time context. Although the exact governing rule is retrieved at rank 1 for only

7\%

of airline states, the primary 3B classifier obtains macro-F1

0.58

with retrieved assertions versus

0.60

with the gold rule (

Δ=-0.02

, task-cluster 95% CI

[-0.23,+0.21]

); random non-gold and no-assertion controls score

0.32

and

0.21

. We do not detect a macro-F1 difference between retrieved assertions and the gold rule in this configuration, although the interval remains too wide to establish non-inferiority. The same qualitative pattern appears with a second retriever and at 7B, while varying across fine-tuning configurations. These results show that exact-match recovery of the benchmark-designated rule can underestimate the downstream utility of retrieved benchmark assertions in this setting. Retrieval should therefore be evaluated inside the classification loop rather than by exact-match recall alone.

Tianyu Ding, Juan Pablo De la Cruz Weinstein

Amazon Web Services

Tool-Use Evaluation Retrieval-Augmented Classification Long-Horizon Agent Evaluation AI Agent Benchmarks IR Evaluation

Ask PDF

Aug 31, 2026

Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval

In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.

Shaowei Wei, Chong Huang, Songtao Fang +3

Alibaba Group Hangzhou, China

Semantic Search RL for IR RL Fine-Tuning Dense Retrieval

Ask PDF

May 29, 2026

Retriever Portfolios: A Principled Approach to Adaptive RAG

Retrieval-augmented generation (RAG) systems typically rely on a single retriever and a single set of hyperparameters, despite facing highly heterogeneous queries that range from simple factoid questions to complex multi-hop reasoning. We propose a method that automatically selects a small, diverse subset of retrievers (a portfolio) from a large pool of candidates, to cover different regions of the target query distribution. We formalize this setting via an expected best-of-

k

objective over the query distribution and show that it admits an efficient portfolio construction algorithm with near-optimal guarantees. Across multiple QA benchmarks, our learned portfolios and router pipeline consistently outperform single-retriever and naive multi-retriever baselines on both retrieval metrics and answer quality. In addition, compared to inference-time hyperparameter tuning approaches, fixed portfolios enable parallel retrieval and LLM calls, achieving comparable (and sometimes better) accuracy with substantially lower latency and token cost.

Miltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi +1

1EPFL · 2Google Research

Retrieval-Augmented Generation Adaptive RAG Adaptive Retrieval

Ask PDF

Abstract

Adaptive Retrieval

Explore similar work

When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval

Retriever Portfolios: A Principled Approach to Adaptive RAG