Retrieval-augmented generation (RAG) faces a fundamental three-way tension: deeper retrieval improves factual grounding but inflates token costs and end-to-end latency. Static retrieval configurations cannot resolve this tension across heterogeneous query workloads -- simple definitional queries waste budget on unnecessary context, while complex analytical prompts are underserved by shallow retrieval. This paper introduces Cost-Aware RAG (CA-RAG), a per-query routing framework that selects from a discrete catalog of strategy bundles -- each coupling a retrieval depth (from retrieval-free direct inference to top-k=10 dense retrieval) with a fixed generation profile -- by maximizing a scalar utility that linearly combines an estimated quality prior with normalized penalties for predicted latency and total billed tokens. CA-RAG is implemented with FAISS-backed dense retrieval and OpenAI chat/embedding APIs, and evaluated on a 28-query benchmark spanning four bundles. The router dynamically exercises all bundles, achieving 26% fewer billed tokens than always-heavy retrieval and 34% lower mean latency than always-direct inference while maintaining equivalent answer quality. Per-query delta analysis reveals that savings are non-uniform and concentrated in simpler queries, motivating complexity-aware guardrails. Sensitivity analysis confirms that the same bundle catalog supports multiple cost-latency-quality operating points through weight adjustment alone. All results are generated directly from logged CSV artifacts for full reproducibility. CA-RAG provides a transparent, auditable foundation for cost-conscious LLM deployments.
Figures & tables
Bundle
k
Skip retr.
Qual. prior
Lat. prior (ms)
direct_llm
0
✓
0.52
8
light_rag
3
✗
0.66
45
medium_rag
5
✗
0.74
60
heavy_rag
10
✗
0.82
95
TABLE I: Strategy bundle catalog: retrieval depth and quality/latency priors with shared generation profile.
Metric
Value
Queries
28
Unique strategies
4
Corpus lines (benchmark)
15
Index embedding tokens (API)
262
TABLE II: Benchmark corpus and FAISS index statistics.
Fig. 1: Bundle selection frequency across 28 benchmark queries. All four bundles are exercised, confirming genuine routing diversity.
Fig. 2: Selection utility vs. total billed tokens per query. Each strategy occupies a distinct region of the cost–utility space.
Fig. 3: End-to-end latency distributions by strategy. direct_llm shows the highest variance; retrieval bundles exhibit tighter, more predictable profiles.
Fig. 4: Cumulative billed tokens in query-log order. Slope changes correspond to strategy transitions.
Fig. 5: Mean prompt, completion, and embedding token shares by strategy. Prompt tokens scale with retrieval depth; completion tokens remain stable.
Fig. 6: Histogram of per-query selection utilities. The right-skewed distribution reflects higher quality priors for simple queries where cost and latency penalties are minimal.
Fig. 7: Realized utility distributions by strategy (post-hoc scoring with observed metrics). medium_rag achieves the highest and most consistent realized utility.
Fig. 8: Max retrieval confidence scores when retrieval executes. The bimodal distribution identifies a subset of queries with poor corpus coverage.
Fig. 9: Heuristic query complexity vs. end-to-end latency. Marker color encodes billed tokens. Complexity is a weak predictor of cost ( r=0.21 ).
Fig. 10: Total billed tokens per query in run order. Per-query cost heterogeneity motivates adaptive rather than fixed retrieval depth.
Policy
cost(tok)
lat(ms)
qual.
U
router_default
252.4
2927
0.80
0.192
router_latency_sensitive
256.0
2165
0.81
-0.291
router_cost_sensitive
231.8
2536
0.81
0.117
fixed_direct
249.9
4457
0.80
-0.367
fixed_light
197.3
2091
0.82
0.167
fixed_medium
239.5
1906
0.82
0.177
TABLE III: Policy-level comparison: mean billed tokens, latency (ms), lexical quality proxy, and selection utility.
Baseline
P(cost win)
P(lat win)
P(qual win)
fixed_direct
0.43
0.71
0.32
fixed_light
0.07
0.36
0.00
fixed_medium
0.32
0.18
0.00
fixed_heavy
0.82
0.25
0.07
TABLE IV: Per-query win-rates of the default router vs. fixed baselines. Lower is better for cost/latency; higher is better for quality proxy.
Fig. 11: Mean billed tokens by policy. The router reduces cost by 26% vs. fixed-heavy.
Fig. 12: Mean end-to-end latency (ms) by policy. The router reduces latency by 34% vs. fixed-direct.
Fig. 13: Mean lexical quality proxy by policy. Quality is maintained within 0.01–0.02 of the best fixed baselines.
Fig. 14: Normalized mean cost, latency, and quality under default, latency-sensitive, and cost-sensitive weight settings.
Fig. 15: Per-query token cost delta vs. fixed-heavy. Savings are concentrated in definitional queries.
Fig. 16: Per-query latency delta vs. fixed-heavy.
Fig. 17: Per-query quality proxy delta vs. fixed-heavy. Quality parity is maintained across all query types.
Fig. 18: Strategy selection distribution under default, latency-sensitive, and cost-sensitive weight settings.
Variable
mean
std
min
max
cost
258.5
94.5
161.0
530.0
latency
2512.3
1675.6
1151.6
8253.4
utility
0.3
0.1
0.1
0.4
quality_proxy
0.8
0.1
0.6
1.0
TABLE V: Run-wide descriptive statistics over 28 queries.
Strategy
mean cost
mean latency
mean U
direct_llm
236.0 ± 56.0
4274 ± 900
0.333 ± 0.123
light_rag
203.7 ± 84.9
2490 ± 1703
0.263 ± 0.092
medium_rag
216.7 ± 45.9
1714 ± 588
0.250 ± 0.086
heavy_rag
367.0 ± 74.0
2869 ± 2298
0.220 ± 0.057
TABLE VI: Per-strategy means ± std. dev. Cost and latency scale monotonically with retrieval depth; quality gains diminish beyond medium_rag .
cost
lat.
U
cplx.
cost
1.00
0.66
-0.50
0.22
lat.
0.66
1.00
-0.19
0.13
U
-0.50
-0.19
1.00
-0.48
cplx.
0.22
0.13
-0.48
1.00
TABLE VII: Pearson correlations among per-query logged scalars. Weak complexity–cost correlation ( r=0.21 ) identifies a key limitation of heuristic routing signals.