Production AI agents frequently receive user-specific queries that are highly repetitive, with up to 47% being semantically similar to prior interactions, yet each query is typically processed with the same computational cost. We argue that this redundancy can be exploited through conversational memory, transforming repetition from a cost burden into an efficiency advantage. We propose a memory-augmented inference framework in which a lightweight 8B-parameter model leverages retrieved conversational context to answer all queries via a low-cost inference path. Without any additional training or labeled data, this approach achieves 30.5% F1, recovering 69% of the performance of a full-context 235B model while reducing effective cost by 96%. Notably, a 235B model without memory (13.7% F1) underperforms even the standalone 8B model (15.4% F1), indicating that for user-specific queries, access to relevant knowledge outweighs model scale. We further analyze the role of routing and confidence. At practical confidence thresholds, routing alone already directs 96% of queries to the small model, but yields poor accuracy (13.0% F1) due to confident hallucinations. Memory does not substantially alter routing decisions; instead, it improves correctness by grounding responses in retrieved user-specific information. As conversational memory accumulates over time, coverage of recurring topics increases, further narrowing the performance gap. We evaluate on 152 LoCoMo questions (Qwen3-8B/235B) and 500 LongMemEval questions. Incorporating hybrid retrieval (BM25 + cosine similarity) improves performance by an additional +7.7 F1, demonstrating that retrieval quality directly enhances end-to-end system performance. Overall, our results highlight that memory, rather than model size, is the primary driver of accuracy and efficiency in persistent AI agents.
Figures & tables
Fig. 1 : Compound pipeline. Solid arrows: inference path; dashed arrow: cross-model memory accumulation. When c≥τ , the small model’s response is accepted (cheap path); otherwise the query escalates. Both paths store the turn-pair for future retrieval.
Condition
S
M
O
T
ALL
Warm (compound)
46.4
20.8
25.5
10.3
30.5
Warm (memory only)
45.7
20.8
25.5
10.3
30.1
Cold compound
16.7
6.2
9.6
13.1
13.0
Cold 8B
17.5
8.8
24.7
14.1
15.4
Cold 235B
19.8
7.6
16.1
6.4
13.7
Full-ctx 235B
61.9
47.0
19.1
16.1
43.9
TABLE I : F1 scores (%) on LoCoMo by category (152 questions). S = single-hop (70), M = multi-hop (40), O = open (12), T = temporal (30). Cold compound = routing without memory ( τ=0.50 , 96% on 8B). “—” = unreported in source.
F1 (%)
% on 8B
EffCost
No memory
No routing (Cold 8B)
15.4
100
15K
Routing (Cold compound)
13.0
96.1
∼ 15K
With memory
No routing (Warm mem-only)
30.1
100
110K
Routing (Warm compound)
30.5
100
22K
TABLE II: 2 × 2 factorial: memory × routing (152 questions, Qwen3-8B, τ=0.50 ). “% on 8B” = fraction of queries served by the small model. Memory does not change routing behavior; instead it changes answer quality .
Condition
In
Out
EffCost
Cold 8B
9.6K
1.4K
15K
Cold compound
∼ 10K
∼ 1.4K
∼ 16K
Cold 235B
9.6K
1.4K
443K
Warm (mem. only)
103K
1.7K
110K
Warm (compound)
16K
1.5K
22K
Full-ctx 235B
2,314K
1.8K
68,179K
TABLE III : Token usage and effective cost (152 questions). EffCost = (in+4×out)×P/8B , where the 4× output multiplier reflects costlier autoregressive decode and P is model parameters. All Warm conditions run on 8B ( P/8B=1 ); 235B conditions cost ≈ 29 × more per token.
Fig. 2 : F1 by question category. Memory injection provides the largest gains for factual-recall categories (single-hop, multi-hop) and the smallest for categories requiring external knowledge or temporal reasoning.
Retrieval
F1 (%)
BLEU-1 (%)
Cosine-only
36.5
34.9
Hybrid (BM25 + cosine)
44.2
42.0
TABLE IV: End-to-end QA on LongMemEval (100 stratified questions, 8B model). Hybrid = BM25 + cosine fusion. Best per column in bold.
Question type
n
Cosine
Hybrid
Δ
single-session-user
14
57.9
76.9
+19.0
single-session-assistant
11
64.8
61.8
− 3.0
single-session-preference
6
2.7
5.8
+3.1
multi-session
27
29.0
26.8
− 2.2
temporal-reasoning
26
28.7
34.4
+5.7
knowledge-update
16
36.3
63.0
+26.7
TABLE V: Per-type F1 (%) on LongMemEval: cosine-only vs. hybrid retrieval. Δ = absolute F1 gain from adding BM25.
Strategy
F1
¢/query
Personal
Enterprise
All large (no memory)
13.7
0.325
$22.75
$9,750
All small (no memory)
15.4
0.020
$1.37
$585
Compound (ours)
30.5
0.025
$1.72
$738
Compound is 92% cheaper than all-large with 2 × the quality.
TABLE VI : Projected monthly API cost for different strategies. Personal = 7K queries/mo (OpenClaw single-agent profile [ 24 ] ); Enterprise = 3M queries/mo (100 agents). Assumes 500 input + 200 output tokens/query; compound adds 1.68 × input from memory injection. Pricing: small 0.15/0.60, large 2.50/10.00 per M tokens (GPT-4o-mini / GPT-4o). F1 from Table I .
LLM agents lack persistent memory, causing conversations to reset each session and preventing personalization over time. We present Lyzr Cognis, a unified memory architecture for conversational AI agents that addresses this limitation through a multi-stage retrieval pipeline. Cognis combines a dual-store backend pairing OpenSearch BM25 keyword matching with Matryoshka vector similarity search, fused via Reciprocal Rank Fusion. Its context-aware ingestion pipeline retrieves existing memories before extraction, enabling intelligent version tracking that preserves full memory history while keeping the store consistent. Temporal boosting enhances time-sensitive queries, and a BGE-2 cross-encoder reranker refines final result quality. We evaluate Cognis on two independent benchmarks -- LoCoMo and LongMemEval -- across eight answer generation models, demonstrating state-of-the-art performance on both. The system is open-source and deployed in production serving conversational AI applications.
Explicit memory systems transform conversation history into persistent representations intended to improve future access, often at substantial construction cost. Yet it remains unclear whether this added complexity yields better answer accuracy than retrieving directly from raw history. This study compares six state-of-the-art explicit memory systems with raw-history RAG on LongMemEval-S and BEAM-100K under original and controlled retrieval settings. We find only localized gains for particular question types and tight run-time context budgets, with no consistent advantage over strong raw-history RAG baselines. Oracle retrieval shows that some systems fail to preserve answer-supporting information during explicit memory construction, while others preserve it but fail to retrieve it effectively, despite construction costs reaching hours per conversation. Our findings suggest that while many existing memory systems for AI agents emphasize increasingly elaborate explicit memory construction, they are often evaluated without strong controlled comparisons to raw-history RAG and may underemphasize retrieval. Retrieval remains a major bottleneck when useful information is preserved, suggesting that improving access to existing information may be at least as important as constructing richer explicit memory.
Yongjie Zhou, Shuai Wang, Bevan Koopman +1
The University of Queensland, Brisbane, Australia · CSIRO · Google
Large Language Model (LLM) agents require persistent memory to maintain personalization, factual continuity, and long-horizon reasoning, yet standard context-window and retrieval-augmented generation (RAG) pipelines degrade over multi-session interactions. We present MemMachine, an open-source memory system that integrates short-term, long-term episodic, and profile memory within a ground-truth-preserving architecture that stores entire conversational episodes and reduces lossy LLM-based extraction. MemMachine uses contextualized retrieval that expands nucleus matches with surrounding context, improving recall when relevant evidence spans multiple dialogue turns. Across benchmarks, MemMachine achieves strong accuracy-efficiency tradeoffs: on LoCoMo it reaches 0.9169 using gpt4.1-mini; on LongMemEvalS (ICLR 2025), a six-dimension ablation yields 93.0 percent accuracy, with retrieval-stage optimizations -- retrieval depth tuning (+4.2 percent), context formatting (+2.0 percent), search prompt design (+1.8 percent), and query bias correction (+1.4 percent) -- outperforming ingestion-stage gains such as sentence chunking (+0.8 percent). GPT-5-mini exceeds GPT-5 by 2.6 percent when paired with optimized prompts, making it the most cost-efficient setup. Compared to Mem0, MemMachine uses roughly 80 percent fewer input tokens under matched conditions. A companion Retrieval Agent adaptively routes queries among direct retrieval, parallel decomposition, or iterative chain-of-query strategies, achieving 93.2 percent on HotpotQA-hard and 92.6 percent on WikiMultiHop under randomized-noise conditions. These results show that preserving episodic ground truth while layering adaptive retrieval yields robust, efficient long-term memory for personalized LLM agents.