How Much Conversation History Does an AI Agent Need?
Authors: Danny Kim, Fernando Koch
Organizations: Pembroke Pines Charter High School Boca Raton, FL · Department of Electrical Eng and Computer Sciences Florida Atlantic University Boca Raton, FL
How does using less conversational history change reader-token use and benchmark scores? We examine this question on 1,100 BEAM questions. We held the retrieval ranking and reader constant across five fixed passage budgets, and mean assigned scores rose as the passage budget grew. The score gain per additional token became smaller at the upper tested budgets. Answerable questions generally received higher scores with more context. Pooled abstention scores sometimes rose as the passage budget shrank. We also tested message retention in a controlled task with explicit conversational-state targets and compared retained user messages with token-matched recency. Qwen matched the full state more often for direct requirements when user messages were retained. Its full-state matching declined when answers depended on references to earlier offers. Mistral produced near-floor results across the retention conditions. A scoring audit leaves the correctness of BEAM answers unresolved. These experiments characterize the token and score consequences of the tested memory budgets. Reliable budget selection depends on retaining the evidence each question requires and validating the measure used to assess answer quality.
Figures & tables
Figure 2: A dependency removed by retention. The rows isolate the decisive messages in two versions of a frozen task; columns show the full source, retained text and visible item. A: e14 names offer e13, so removing e13 leaves the item unresolved. A later offer of the same value cannot resolve that reference. B: e14 states the value directly, so it survives. Both variants retain place and duration. Oracle restoration repairs the source dependency but also changes padding and position. This schematic describes task grammar; model outcomes are measured separately.
Figure 3: Where passage budgets help and hurt assigned scores. A: All 40 category–budget contrasts as paired means on one zero-centered color scale; signed cell values give score-point changes from ten passages. Outlines mark top-3. Each cell uses 104–110 complete pairs from 110 planned questions. Categories are descriptive, not significance tests. B: Top-3/top-10 subgroup effects with saved 95% source-group intervals; labels give paired counts. C: Planned interactions with adjusted 97.5% intervals: 100K, +15.29[−1.96,32.88] ; 500K, +20.64[9.22,33.36] points. The tiers contain 20 and 33 source groups, respectively.
Figure 4: Evaluation limits and score sensitivity. A–B: Selected AI-review outcomes and synthetic qualification cases passing all required checks. Passing cases comprise 18/21 reused regressions and 4/10 fresh probes; this is not an original-BEAM error-rate estimate. C: All 53 single-group deletion estimates, the original interval, equal-group weighting, and unrestricted missing-pair bounds. Only the row labeled 95% is a confidence interval. D: Saved worst-case bounds when both scores on selected complete pairs may change arbitrarily; all 16 missing pairs are unrestricted throughout. The threshold of 22 pairs is a sensitivity result, not an observed error count or error rate.
Figure 5: All controlled retention outcomes on a common scale. Marks and counts show correct completed full-source states among 64 planned cases in 16 families; correct guesses need not be supported by retained text. Light ticks show source sufficiency, a property of retained text. The open diamond preserves one missing response; its range is not a confidence interval. Row groups separate the main policies, guidance/placebo, decoder and oracle controls. At matched tokens, oracle restoration changes Qwen correctness from 3/64 to 42/64 on reference cases and from 50/64 to 41/64 on direct cases. Position changes and padding prevent an isolated mechanism claim.
Figure 6: Post-hoc diagnostics from saved responses. A: Qwen’s 64 reference-retention requests partition into five groups. The 15 partial-state matches return a null item and correct visible place/duration; eight other null-item outputs miss at least one of those slots. Two full-state matches cannot establish support for the unresolved item. One response is missing. B: Mistral’s full-history outputs, partitioned by slot into target match, null and other value. Only one output matches all three slots. These counts explain what the original all-slot metric combines; they do not replace it or establish a causal explanation.