How Much Conversation History Does an AI Agent Need?
Authors: Danny Kim, Fernando Koch
Organizations: Pembroke Pines Charter High School Boca Raton, FL · Department of Electrical Eng and Computer Sciences Florida Atlantic University Boca Raton, FL
How does using less conversational history change reader-token use and benchmark scores? We examine this question on 1,100 BEAM questions. We held the retrieval ranking and reader constant across five fixed passage budgets, and mean assigned scores rose as the passage budget grew. The score gain per additional token became smaller at the upper tested budgets. Answerable questions generally received higher scores with more context. Pooled abstention scores sometimes rose as the passage budget shrank. We also tested message retention in a controlled task with explicit conversational-state targets and compared retained user messages with token-matched recency. Qwen matched the full state more often for direct requirements when user messages were retained. Its full-state matching declined when answers depended on references to earlier offers. Mistral produced near-floor results across the retention conditions. A scoring audit leaves the correctness of BEAM answers unresolved. These experiments characterize the token and score consequences of the tested memory budgets. Reliable budget selection depends on retaining the evidence each question requires and validating the measure used to assess answer quality.
Figures & tables
Figure 2: A dependency removed by retention. The rows isolate the decisive messages in two versions of a frozen task; columns show the full source, retained text and visible item. A: e14 names offer e13, so removing e13 leaves the item unresolved. A later offer of the same value cannot resolve that reference. B: e14 states the value directly, so it survives. Both variants retain place and duration. Oracle restoration repairs the source dependency but also changes padding and position. This schematic describes task grammar; model outcomes are measured separately.
Figure 3: Where passage budgets help and hurt assigned scores. A: All 40 category–budget contrasts as paired means on one zero-centered color scale; signed cell values give score-point changes from ten passages. Outlines mark top-3. Each cell uses 104–110 complete pairs from 110 planned questions. Categories are descriptive, not significance tests. B: Top-3/top-10 subgroup effects with saved 95% source-group intervals; labels give paired counts. C: Planned interactions with adjusted 97.5% intervals: 100K, +15.29[−1.96,32.88] ; 500K, +20.64[9.22,33.36] points. The tiers contain 20 and 33 source groups, respectively.
Figure 4: Evaluation limits and score sensitivity. A–B: Selected AI-review outcomes and synthetic qualification cases passing all required checks. Passing cases comprise 18/21 reused regressions and 4/10 fresh probes; this is not an original-BEAM error-rate estimate. C: All 53 single-group deletion estimates, the original interval, equal-group weighting, and unrestricted missing-pair bounds. Only the row labeled 95% is a confidence interval. D: Saved worst-case bounds when both scores on selected complete pairs may change arbitrarily; all 16 missing pairs are unrestricted throughout. The threshold of 22 pairs is a sensitivity result, not an observed error count or error rate.
Figure 5: All controlled retention outcomes on a common scale. Marks and counts show correct completed full-source states among 64 planned cases in 16 families; correct guesses need not be supported by retained text. Light ticks show source sufficiency, a property of retained text. The open diamond preserves one missing response; its range is not a confidence interval. Row groups separate the main policies, guidance/placebo, decoder and oracle controls. At matched tokens, oracle restoration changes Qwen correctness from 3/64 to 42/64 on reference cases and from 50/64 to 41/64 on direct cases. Position changes and padding prevent an isolated mechanism claim.
Figure 6: Post-hoc diagnostics from saved responses. A: Qwen’s 64 reference-retention requests partition into five groups. The 15 partial-state matches return a null item and correct visible place/duration; eight other null-item outputs miss at least one of those slots. Two full-state matches cannot establish support for the unresolved item. One response is missing. B: Mistral’s full-history outputs, partitioned by slot into target match, null and other value. Only one output matches all three slots. These counts explain what the original all-slot metric combines; they do not replace it or establish a causal explanation.
Explicit memory systems transform conversation history into persistent representations intended to improve future access, often at substantial construction cost. Yet it remains unclear whether this added complexity yields better answer accuracy than retrieving directly from raw history. This study compares six state-of-the-art explicit memory systems with raw-history RAG on LongMemEval-S and BEAM-100K under original and controlled retrieval settings. We find only localized gains for particular question types and tight run-time context budgets, with no consistent advantage over strong raw-history RAG baselines. Oracle retrieval shows that some systems fail to preserve answer-supporting information during explicit memory construction, while others preserve it but fail to retrieve it effectively, despite construction costs reaching hours per conversation. Our findings suggest that while many existing memory systems for AI agents emphasize increasingly elaborate explicit memory construction, they are often evaluated without strong controlled comparisons to raw-history RAG and may underemphasize retrieval. Retrieval remains a major bottleneck when useful information is preserved, suggesting that improving access to existing information may be at least as important as constructing richer explicit memory.
Yongjie Zhou, Shuai Wang, Bevan Koopman +1
The University of Queensland, Brisbane, Australia · CSIRO · Google
Long conversations with an AI agent create a simple problem for one user: the history is useful, but carrying it verbatim is expensive. We study personalized agent memory: one user's conversation history with an agent, distilled into a compact retrieval layer for later search. Each exchange is compressed into a compound object with four fields (exchange_core, specific_context, thematic room_assignments, and regex-extracted files_touched). The searchable distilled text averages 38 tokens per exchange. Applied to 4,182 conversations (14,340 exchanges) from 6 software engineering projects, the method reduces average exchange length from 371 to 38 tokens, yielding 11x compression. We evaluate whether personalized recall survives that compression using 201 recall-oriented queries, 107 configurations spanning 5 pure and 5 cross-layer search modes, and 5 LLM graders (214,519 consensus-graded query-result pairs). The best pure distilled configuration reaches 96% of the best verbatim MRR (0.717 vs 0.745). Results are mechanism-dependent. All 20 vector search configurations remain non-significant after Bonferroni correction, while all 20 BM25 configurations degrade significantly (effect sizes |d|=0.031-0.756). The best cross-layer setup slightly exceeds the best pure verbatim baseline (MRR 0.759). Structured distillation compresses single-user agent memory without uniformly sacrificing retrieval quality. At 1/11 the context cost, thousands of exchanges fit within a single prompt while the verbatim source remains available for drill-down. We release the implementation and analysis pipeline as open-source software.
Conversational agents can retrieve information directly or delegate retrieval to a subagent. This design choice affects how retrieval history is maintained, what context each model call receives, and what information must be re-retrieved in follow-up questions. We compare three designs: skill-based retrieval in the main agent, a non-persistent retrieval subagent that generates a findings response, and our persistent retrieval subagent, which maintains retrieval history across turns and returns retrieved evidence directly to the main agent. Our approach also allows the main agent to request corrections from the same subagent. We evaluate the three designs on enterprise questions posed independently, within shared conversations, and as dependent follow-ups. Across five multi-turn settings, our approach uses 43.3% fewer recorded agent-model tokens and incurs 45.0% lower estimated API cost than skill-based retrieval, while reducing average response time by 9.1%. Compared with non-persistent retrieval, it reduces response time by 15.6 seconds per question, while using 65.1% more tokens and incurring 29.1% higher estimated API cost. Our approach shifts the retrieval subagent from producing synthesized responses to maintaining retrieval state and supplying evidence that the main agent can directly evaluate and use for answer generation.