VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

Jielin Qiu; Jianguo Zhang; Zixiang Chen; Liangwei Yang; Ming Zhu; Juntao Tan; Haolin Chen; Wenting Zhao

cs.SDMar 2, 2026

VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

Authors: Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang, Ming Zhu, Juntao Tan, Haolin Chen, Wenting Zhao,

Organizations: Salesforce AI Research

Abstract

We present VoiceAgentRAG, an open-source dual-agent memory router that decouples retrieval from response generation. A background Slow Thinker agent continuously monitors the conversation stream, predicts likely follow-up topics using an LLM, and pre-fetches relevant document chunks into a FAISS-backed semantic cache. A foreground Fast Talker agent reads only from this sub-millisecond cache, bypassing the vector database entirely on cache hits.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture of VoiceAgentRAG . The Slow Thinker (left) continuously monitors the conversation stream, predicts follow-up topics, retrieves from the vector store, and populates the semantic cache. The Fast Talker (right) checks the cache first ( $<$ 1ms), bypassing the vector store on hits. On misses, it falls back to direct retrieval (dashed red) and caches the results for future queries.

Metric	Traditional RAG	VoiceAgentRAG
Retrieval latency (avg)	110.4 ms	0.35 ms
Retrieval speedup	1 $\times$	316 $\times$
Cache hit rate (overall)	—	75% (150/200)
Cache hit rate (warm, turn $\geq$ 2)	—	79% (150/190)
Saved per cache-hit query	—	110.1 ms
Total retrieval time saved	—	16,508 ms

Table 1: Overall performance across 200 queries on 10 conversation scenarios using Qdrant Cloud. All measurements are from real network round-trips, with no simulated delays.

Scenario	Hit Rate	Avg. Retrieval	Pattern
S8: Feature comparison	95% (19/20)	118.4 ms	Broad tour
S2: Pricing deep-dive	85% (17/20)	107.8 ms	Single topic
S3: API integration	85% (17/20)	108.7 ms	Single topic
S5: Onboarding walkthrough	85% (17/20)	119.2 ms	Sequential
S7: Integration questions	85% (17/20)	108.0 ms	Related topics
S6: Troubleshooting	80% (16/20)	113.6 ms	Problem-solving

Table 2: Cache hit rate by conversation scenario (20 turns each), sorted by hit rate. Scenarios with sustained topics achieve higher hit rates than those with frequent topic shifts. Avg. retrieval shows Qdrant Cloud search latency.

Turn Range	Queries	Hit Rate	Note
Turns 1–4	50	58% (29/50)	Cold start + warming
Turns 5–9	50	86% (43/50)	Warm cache
Turns 10–14	50	78% (39/50)	Stable
Turns 15–19	50	82% (41/50)	Saturated

Table 3: Cache hit rate by conversation depth (aggregated across all 10 scenarios, 200 queries total). Turn 1 always misses (cold start). The cache reaches 86% hit rate by turns 5–9.

Threshold $\tau$	Hit Rate	Note
0.55	15%	Too strict, most queries score 0.30–0.55
0.50	30%	Captures some matches
0.45	55%	Good precision, moderate recall
0.40	75%	Best balance (default)
0.35	82%	Higher recall, risk of irrelevant context
0.30	90%	High recall but low precision

Table 4: Impact of cache similarity threshold

\tau

on hit rate. Query-to-document cosine similarity with OpenAI text-embedding-3-small typically ranges 0.30–0.55. A threshold of 0.40 balances precision and recall.

System	Proactive	Predictive	Voice	Bkgd.	Open
Traditional RAG [ 12 ]	✗	✗	✗	✗	—
GPTCache [ 21 ]	✗	✗	✗	✗	✓
Stream RAG [ 1 ]	✓	✓	✓	✗	✗
MemGPT/Letta [ 15 ]	✗	✗	✗	✗	✓
DPT-Agent [ 19 ]	✓	✗	✗	✓	✗
LiveKit+RAG [ 13 ]	✗	✗	✓	✗	✓

Table 5: Comparison of VoiceAgentRAG with related approaches. VoiceAgentRAG is the only system combining proactive (predictive) caching with a dedicated background agent for voice-optimized retrieval.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Component	Parameter	Default
Cache	Max size	2000
Cache	TTL	300s
Cache	Similarity threshold $\tau$	0.40
Cache	Dedup threshold	0.95
Slow Thinker	Prediction strategy	LLM
Slow Thinker	Max predictions per turn	5

Appendix

Table 6: Default configuration parameters for VoiceAgentRAG .

Agentic RAG

Agentic retrieval-augmented generation lets language models plan searches, use tools, and refine evidence before producing grounded answers.

See all 246 papers

Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval
Oct 17, 2026Yongjie Zhou, Shuai Wang, Bevan Koopman +1LLM Agent Memory Long-Term Conversational Memory
The University of Queensland, Brisbane, Australia · CSIRO · Google
Cognis: Context-Aware Memory for Conversational AI Agents
Mar 27, 2026Parshva Daftari, Khush Patel, Shreyas Kapale +2Agent Memory Management Memory-Augmented Language Model Agents
Lyzr Research
AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory
May 24, 2026Aojie Yuan, Haiyue Zhang, Shahin NazarianLexical Retrieval Long-Term Conversational Memory
University of Southern California Los Angeles, CA, USA

Oct 17, 2026J/K move · Enter open · S save

Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval

Yongjie Zhou, Shuai Wang, Bevan Koopman, Guido Zuccon

The University of Queensland, Brisbane, Australia · CSIRO · Google

LLM Agent Memory Long-Term Conversational Memory Retrieval-Augmented Generation LLM Agent Evaluation Conversational QA Memory Agent Benchmarks

Open paper AskPDF arXiv

Explicit memory systems transform conversation history into persistent representations intended to improve future access, often at substantial construction cost. Yet it remains unclear whether this added complexity yields better answer accuracy than retrieving directly from raw history. This study compares six state-of-the-art explicit memory systems with raw-history RAG on LongMemEval-S and BEAM-100K under original and controlled retrieval settings. We find only localized gains for particular question types and tight run-time context budgets, with no consistent advantage over strong raw-history RAG baselines. Oracle retrieval shows that some systems fail to preserve answer-supporting information during explicit memory construction, while others preserve it but fail to retrieve it effectively, despite construction costs reaching hours per conversation. Our findings suggest that while many existing memory systems for AI agents emphasize increasingly elaborate explicit memory construction, they are often evaluated without strong controlled comparisons to raw-history RAG and may underemphasize retrieval. Retrieval remains a major bottleneck when useful information is preserved, suggesting that improving access to existing information may be at least as important as constructing richer explicit memory.

Abstract

Agentic RAG

Explore similar work

Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval

Cognis: Context-Aware Memory for Conversational AI Agents

AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory