Persistent Retrieval for Conversational Agents: A Study of Retrieval Efficiency

Karthik Ravula; Sathwik Kowtha; Rishabh Verma; Kala Mehak Jain; Gunjit Mittal

cs.AINov 5, 2026

Persistent Retrieval for Conversational Agents: A Study of Retrieval Efficiency

Authors: Karthik Ravula, Sathwik Kowtha, Rishabh Verma, Kala Mehak Jain, Gunjit Mittal

Organizations: ServiceNow

Abstract

Conversational agents can retrieve information directly or delegate retrieval to a subagent. This design choice affects how retrieval history is maintained, what context each model call receives, and what information must be re-retrieved in follow-up questions. We compare three designs: skill-based retrieval in the main agent, a non-persistent retrieval subagent that generates a findings response, and our persistent retrieval subagent, which maintains retrieval history across turns and returns retrieved evidence directly to the main agent. Our approach also allows the main agent to request corrections from the same subagent. We evaluate the three designs on enterprise questions posed independently, within shared conversations, and as dependent follow-ups. Across five multi-turn settings, our approach uses 43.3% fewer recorded agent-model tokens and incurs 45.0% lower estimated API cost than skill-based retrieval, while reducing average response time by 9.1%. Compared with non-persistent retrieval, it reduces response time by 15.6 seconds per question, while using 65.1% more tokens and incurring 29.1% higher estimated API cost. Our approach shifts the retrieval subagent from producing synthesized responses to maintaining retrieval state and supplying evidence that the main agent can directly evaluate and use for answer generation.

Figures & tables

Figure 1: Retrieval designs compared in our evaluation. Skill keeps retrieval in the orchestrator. A non-persistent subagent generates a findings response. Our method retains retrieval history, returns evidence directly, and resumes the same session for corrections.

Set	Method	Accuracy (%)	Response time (s)	Mean tokens per conversation				Est. cost ($)
				Input	Cached	Uncached	Output
ITSM	Non-persistent	96.49	66.7	107.1k	92.2k	14.9k	1.3k	0.080
	Skill	98.25	52.1	166.4k	139.2k	27.3k	0.7k	0.113
	Our method	98.25	54.1	96.9k	81.2k	15.7k	0.6k	0.068
HRSD	Non-persistent	93.75	78.3	108.0k	93.8k	14.2k	1.4k	0.080
	Skill	95.83	52.9	160.7k	134.5k	26.2k	0.8k	0.111

Table 1: Single-turn results. Accuracy and response time use 57 ITSM and 48 HRSD questions per method (each single-turn conversation consists of one question); tokens are cumulative conversation means. Token values use thousands (k) and are rounded to one decimal place. Estimated API cost is in USD per conversation for recorded agent-model calls ( Sec. 3.3 ).

Set	Method	Accuracy (%)	Response time (s)	Mean tokens per question				Est. cost ($)
				Input	Cached	Uncached	Output
ITSM ( $\leq 20$ )	Non-persistent	94.74	73.0	142.8k	115.0k	27.8k	1.5k	0.120
	Skill	96.49	68.2	461.8k	379.6k	82.1k	0.6k	0.309
	Our method	96.49	66.4	246.5k	188.0k	58.4k	0.5k	0.201
HRSD ( $\leq 20$ )	Non-persistent	89.58	71.1	147.3k	126.1k	21.2k	1.4k	0.105
	Skill	91.67	60.3	264.6k	238.7k	25.9k	0.8k	0.136

Table 2: Multi-turn results. ITSM and HRSD use 57 and 48 questions per setting; ITSM Complex uses 69 ( Sec. A.2 ). Tokens (k) sum parent and child calls per question, including cached input. Estimated API cost is in USD per question for recorded agent-model calls ( Sec. 3.3 ).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Configuration	Accuracy (%)	Response time (s)	Total tokens	Uncached input	Schema calls	Cypher calls	Est. cost ($)
No persistence	89.85	58.6	191.8k	32.0k	49	71	0.1302
No direct return	89.85	72.0	302.1k	35.4k	31	80	0.1712
Our method	89.85	54.3	211.3k	27.0k	23	59	0.1205

Appendix

Table 5: Component ablation. Persistent retrieval state and direct evidence return on ITSM Complex. Call counts are averaged across runs and rounded to the nearest integer; time, tokens, and cost are per-question means. Tokens use thousands (k).

Agentic Retrieval

Agentic retrieval lets AI agents iteratively search, inspect, and verify evidence to answer complex information needs.

See all 103 papers

Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents
Mar 24, 2026Xunzhuo Liu, Bowei He, Xue Liu +3Memory-Augmented Language Model Agents Conversational Memory
vLLM Semantic Router Project · MBZUAI · McGill University +3
Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval
Oct 17, 2026Yongjie Zhou, Shuai Wang, Bevan Koopman +1LLM Agent Memory Long-Term Conversational Memory
The University of Queensland, Brisbane, Australia · CSIRO · Google
VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures
Mar 2, 2026Jielin Qiu, Jianguo Zhang, Zixiang Chen +12Agentic RAG Conversational Agents
Salesforce AI Research

Mar 24, 2026J/K move · Enter open · S save

Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents

Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo+2

vLLM Semantic Router Project · MBZUAI · McGill University+3

Memory-Augmented Language Model Agents Conversational Memory LLM Inference Efficiency LLM Routing

Open paper AskPDF arXiv

Production AI agents frequently receive user-specific queries that are highly repetitive, with up to 47% being semantically similar to prior interactions, yet each query is typically processed with the same computational cost. We argue that this redundancy can be exploited through conversational memory, transforming repetition from a cost burden into an efficiency advantage. We propose a memory-augmented inference framework in which a lightweight 8B-parameter model leverages retrieved conversational context to answer all queries via a low-cost inference path. Without any additional training or labeled data, this approach achieves 30.5% F1, recovering 69% of the performance of a full-context 235B model while reducing effective cost by 96%. Notably, a 235B model without memory (13.7% F1) underperforms even the standalone 8B model (15.4% F1), indicating that for user-specific queries, access to relevant knowledge outweighs model scale. We further analyze the role of routing and confidence. At practical confidence thresholds, routing alone already directs 96% of queries to the small model, but yields poor accuracy (13.0% F1) due to confident hallucinations. Memory does not substantially alter routing decisions; instead, it improves correctness by grounding responses in retrieved user-specific information. As conversational memory accumulates over time, coverage of recurring topics increases, further narrowing the performance gap. We evaluate on 152 LoCoMo questions (Qwen3-8B/235B) and 500 LongMemEval questions. Incorporating hybrid retrieval (BM25 + cosine similarity) improves performance by an additional +7.7 F1, demonstrating that retrieval quality directly enhances end-to-end system performance. Overall, our results highlight that memory, rather than model size, is the primary driver of accuracy and efficiency in persistent AI agents.

Abstract

Agentic Retrieval

Explore similar work

Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents

Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval

VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures