cs.CLAug 25, 2026

Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting

Authors: Weibin Cai, Reza Zafarani

Organizations: Data Lab, EECS Department Syracuse University Syracuse, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Existing methods for improving Retrieval-Augmented Generation (RAG) efficiency mainly optimize downstream LLM generation, such as context compression or serving optimization. However, RAG is an end-to-end system, and its bottleneck can shift between upstream reranking and downstream generation under different serving loads and reranking budgets.In this paper, we first empirically characterize this shifting-bottleneck behavior and show that upstream reranking can become the dominant bottleneck under high query rates or large reranking budgets. Reducing the reranking budget can relieve this bottleneck, but it may also drop supporting evidence and degrade recall. To address this problem, we propose PACE (Prioritized Adaptive Coverage of Evidence), a training-free framework that combines evidence frontloading with pressure-adaptive budgeting. PACE first reorders candidates by marginal evidence coverage, prioritizing documents that are query-relevant, complementary, and useful for forming multi-hop evidence chains. We show that this objective is monotone submodular, giving greedy selection a (1−1/e)(1-1/e) approximation guarantee. PACE then dynamically adjusts the reranking budget according to the relative pressure of the reranker and the LLM. Experiments on three multi-hop QA datasets and online serving simulations show that PACE improves evidence recall, reduces p95 latency under ranking-heavy workloads. More importantly, the two components together reveal that less can be more: an evidence-dense top-ranked candidates enable higher final recall with fewer reranked documents.

Figures & tables

Figure 1. How PACE improves RAG effectiveness and efficiency. (1) Evidence frontloading makes top-ranked candidates more evidence-dense, achieving higher evidence recall at a smaller reranking budget DD . (2) Pressure-adaptive budgeting then reduces DD under ranking-heavy workloads to lower latency. (3) Together, PACE enables denser reranker input and higher final recall with fewer reranked documents.
Figure 2. Shifting RAG bottlenecks under different configurations and serving loads. Here, DD denotes reranking budget and KK denotes the number of documents sent to the LLM. In (a), we vary QPS across two rerankers: trecdl22-crossencoder-debertav3 , and ms-marco-MiniLM-L12-v2 , and two LLMs: Qwen2.5-3B-Instruct , and Qwen2.5-7B-Instruct . We report log⁡2(P95 reranker queue timeP95 LLM queue time)\log_{2}\left(\frac{\text{P95 reranker queue time}}{\text{P95 LLM queue time}}\right) , where positive values indicate ranking-heavy serving and negative values indicate generation-heavy serving. In (b) and (c) we use Qwen2.5-3B-Instruct and trecdl22-crossencoder-debertav3 pair and vary DD with and without context compressor.
SymbolMeaning
PRP_{R}Number of unprocessed document pairs in the reranker queue
μ^R\hat{\mu}_{R}Real-time estimated reranker throughput
WRW_{R}Estimated time to clear the reranker queue
∣QL∣|Q_{L}|Number of queries waiting in the LLM queue
BLB_{L}LLM batch size
BRB_{R}Reranker batch size
Table 1. Notation for pressure-adaptive budgeting.
Figure 3. Complete evidence recall@D across three multi-hop QA datasets. PACE tends to cover all required evidence with smaller reranking budgets than training-free baselines.
Figure 4. Supporting evidence recall@D across three multi-hop QA datasets. PACE retrieves a larger fraction of supporting evidence with small reranking budgets.
Figure 5. Ablation study of the marginal evidence coverage objective in Eq. 1 . Removing query relevance, anchor relevance, or both weakens complete evidence recall, showing that direct query relevance and soft-anchor document dependencies jointly contribute to effective evidence frontloading.
Figure 6. Latency comparison under ranking-heavy workloads. PACE substantially reduces p95 end-to-end latency by lowering reranker queueing time with a smaller adaptive reranking budget. Although LLM queueing time may increase as more requests pass through the reranker, but the dominant upstream bottleneck is greatly relieved. The fixed baseline uses D=100D=100 throughout.
Figure 7. Evidence recall under pressure-adaptive budgeting on HotpotQA. The first two subfigures report recall@D before reranking, and the last two report recall@5 after reranking, where DD is dynamically selected and K=5K=5 documents are sent to the LLM. Less can be more when the top-ranked candidates are evidence-dense : as QPS increases, PACE reduces the reranking budget but maintains higher complete and supporting evidence recall than baselines and ablation variants.
Figure 8. Average selected reranking budget under increasing QPS. PACE reduces DD as serving pressure grows, but maintains higher evidence recall than other methods at comparable selected budgets.
DatasetMethodDDComp. R@ DDComp. R@ KK
HotpotQAStandard Dense10010041.13
Dartboard8095.3140.48 (−0.65)(-0.65)
5083.1738.18 (−2.95)(-2.95)
2052.9933.22 (−7.91)(-7.91)
PACE8097.2560.81 (+19.68)(+19.68)
5091.3662.38 (+21.25)(+21.25)
Table 2. Less can be more . Complete evidence recall under fixed reranking budgets. Fixed D=100D=100 denotes the full-budget baseline, and R@KR@K with K=5K=5 measures final evidence recall after reranking. Values in parentheses indicate absolute percentage-point changes in R@KR@K over the full-budget baseline on the same dataset.
More in this topic

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 766 papers

Explore similar work

CardsList
Sep 10, 2026

REVA: Reusable Evidence View Aggregation for Context-Efficient RAG Serving

Retrieval-augmented generation (RAG) improves knowledge-intensive large language model (LLM) applications by conditioning generation on retrieved documents, but longer contexts increase latency, key-value (KV) cache memory, and token cost. Post-retrieval compression can reduce this cost, yet existing compressors often operate independently for each query, rely on auxiliary models or rewriting, and introduce online overhead that can offset the benefit of shorter prompts. We revisit RAG compression from a data-mining perspective by aggregating historical query--document--model interactions into reusable evidence views. We first show that modern compressors have unstable gains over simple truncation and can add substantial inference-time latency. We then propose Reusable Evidence View Aggregation (REVA), a framework that mines the target generator's historical attention traces into a document-keyed, budget-agnostic score store. REVA maps token-level attention to readable word units, aggregates importance across repeated document accesses, and renders budget-specific plain-text views that preserve document order and the standard RAG interface. Across four representative benchmarks and modern LLMs, REVA improves generation quality by 1.0--5.8 points over existing advances, while reducing compression overhead by a factor of 5.3 to 15.6, adding less than 40 ms of latency.

Tuan Nguyen, Qiran Hu, Banruo Liu +3

VinUni-Illinois Smart Health Center, VinUniversity, Vietnam · University of Illinois Urbana-Champaign, USA

LLM Inference EfficiencyRetrieval-Augmented Generation
AskPDF
Jun 1, 2026

When Knowledge Is Not Free: Cost-Aware Evidence Selection in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) typically assumes that external knowledge is free, but many high-quality sources are paywalled, licensed, restricted, or otherwise costly to access. We introduce cost-aware RAG, a setting where retrieved evidence is assigned access-cost tiers and systems must answer under an explicit evidence-access budget. We instantiate this setting by augmenting MS MARCO v2.1 with access-friction tiers and evaluate budgeted evidence selection across general-domain and domain-specific QA benchmarks. Our results show that static selection is brittle: no fixed selector uniformly dominates, and larger budgets do not reliably improve answer quality, even when costly evidence is domain-matched. We then study agentic cost-aware RAG, where an LLM decides when to retrieve, which tier to access, and when to stop. Agents show strong promise as adaptive evidence-acquisition controllers, but their behavior remains highly model- and task-dependent. These findings suggest that cost-aware evidence acquisition is a central challenge for the next generation of RAG systems. All code and data are available at https://github.com/Mignonmy/Cost-Aware.

Mingyan Wu, Han Yang, Omer Ben-Porat +1

Northeastern University · Technical University of Munich · GESIS – Leibniz Institute for the Social Sciences +3

Retrieval-Augmented GenerationEvidence SelectionAgentic RAG
AskPDF
Jun 29, 2026

Know Before You Fetch: Calibrated Retrieval-Budget Allocation for Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) typically retrieves a fixed number of passages for every query. This is wasteful when the reader already knows the answer, and it can be harmful when irrelevant or partially relevant passages distract the reader. We formulate adaptive RAG as calibrated retrieval-budget allocation: given a query, decide whether to answer closed-book, retrieve a compact context (k=1), retrieve a full context (k=5), or abstain. The contribution is a probability interface rather than a new raw uncertainty signal. We calibrate sequence log-probability and prefix-logit uncertainty signals into probabilities of correctness, then use these probabilities for graded context selection, selective abstention, and explicit latency/token trade-offs. Across core QA experiments on TriviaQA, Natural Questions, and MS MARCO, with auxiliary PopQA motivation and Qwen/Llama family checks, diagnostic out-of-fold calibration improves probability quality dramatically: for sequence log-probability, ECE drops from 0.275 to 0.062 on TriviaQA, 0.643 to 0.009 on NQ, and 0.711 to 0.031 on MS MARCO. Graded retrieval improves full-context and passage-budget frontiers for both our signal and TARG-style prefix entropy/margin, while retrieval-call AUC remains essentially tied with binary gating because k=1 is still a retrieval call. Held-out train/validation/test threshold experiments report deployable operating points. At matched-accuracy frontier operating points, a measured cost model reveals that gating is not universally faster: it increases latency by about 27% on Qwen3-8B but saves about 8% on Qwen3-32B. These results support a nuanced view of adaptive RAG: calibrated confidence is best understood as a reusable interface for allocating retrieval budget under task and system constraints.

Zhe Dong, Fang Qin, Manish Shah +1

University of Maine at Presque Isle · Stanford University · Independent Researcher

Retrieval-Augmented GenerationSelective PredictionAdaptive RAGAdaptive RetrievalLanguage Model Calibration
AskPDF