A persistent agent must decide both what to retain as information arrives and what to surface once a query appears, yet memory evaluations can confound these decisions by comparing methods that differ in both retention and selection. We build a streaming-recall benchmark crossing retention and selection rules and evaluate every condition on the same 300 seeded episodes. Holding access fixed, query-aware selection improves required-fact recall by 15.5 percentage points (95% CI: 12.8 to 18.2), whereas a mixed comparison that also changes history access reports a 68.7-point advantage, of which 53.2 points are attributable to access. Under bounded retention, query-aware, dense, and oracle selection reach the retention ceiling, and all 319 observed failures in the bounded recency condition are caused by eviction rather than ranking errors. Recall falls to 0% as targets recede sufficiently far into the past. Repeating the evaluation on SQuAD preserves the retention ceiling while showing that dense retrieval can outperform lexical retrieval on natural text. These results show that bounded-memory evaluations should hold access fixed and report retention and selection separately.
Figures & tables
Retention rule
random
recency
query
dense
oracle
Retention
unbounded
31.5
31.3
100.0
100.0
100.0
100.0
recency
27.0
31.3
46.8
46.8
46.8
46.8
random
27.8
30.8
46.8
46.8
46.8
46.8
frequency
26.8
30.2
45.8
45.8
45.8
45.8
type-oracle †
43.7
41.2
70.3
70.3
70.3
70.3
Table 1: Required-fact recall (%) under informative queries, K=5 , k=3 , 600 queries per cell on the same 300 episodes. “Retention” is the fraction of queries whose target survived retention and upper-bounds every entry in its row. † type-oracle reads the generator’s labels and is privileged, not implementable by an agent. The 100% entries are a construction ceiling, not a performance result: an informative query token-matches exactly one fact, so selection is exactly solvable when nothing was discarded.
Figure 1: Required-fact recall under query-aware selection; the lexical and dense scorers are identical throughout under informative queries, so one set of curves serves for both. (a) vs. memory budget K ; selection is inert at K≤k=3 . (b) vs. delay (facts observed after the target; n=188/159/253 ): recency retention falls to zero : query-aware retrieval offers no protection against eviction. (c) The apparent advantage at K=5 : adding query-aware selection at matched access gives +15.5 pp, adding full-history access a further +53.2 pp.
Retention rule
K=2
K=3
K=5
K=10
K=20
unbounded
100.0
100.0
100.0
100.0
100.0
recency
22.7
31.3
46.8
100.0
100.0
random
21.7
30.8
46.8
53.0
53.0
frequency
18.0
26.7
45.8
100.0
100.0
type-oracle †
30.2
41.2
70.3
100.0
100.0
Table 2: Recall (%) under query-aware selection across memory budgets K , informative queries, k=3 . At K≤k nothing retained is withheld, so selection is inert and every rule ties.
Retention rule
0–2
3–5
6–9
unbounded
100.0
100.0
100.0
recency
100.0
58.5
0.0
random
52.1
57.2
36.4
frequency
93.1
45.3
11.1
type-oracle †
100.0
97.5
31.2
Table 3: Recall (%) under query-aware selection by delay, measured as the number of facts observed after the target, K=5 , informative queries ( n=188/159/253 queries per bucket). Selection is oracle-equivalent in every row, so the decline is retention loss, not retrieval failure.
Retention rule
d=0.1
d=0.3
d=0.5
d=0.7
unbounded
100.0
100.0
100.0
100.0
recency
47.3
46.8
46.5
50.5
random
44.8
46.8
41.0
44.2
frequency
47.5
45.8
35.3
23.5
type-oracle †
54.0
70.3
100.0
100.0
Table 4: Recall (%) under query-aware selection as distractor density d rises, K=5 , informative queries. Higher density benefits the privileged type-oracle, since more of the stream is filtered before capacity is consumed, while recency retention is roughly unchanged.
Retention rule
random
recency
query
dense
oracle
Retention
unbounded
31.5
31.3
31.3
42.0
100.0
100.0
recency
27.0
31.3
31.3
35.8
46.8
46.8
random
27.8
30.8
30.8
39.3
46.8
46.8
frequency
26.8
30.2
30.2
35.2
45.8
45.8
type-oracle †
43.7
41.2
41.2
39.7
70.3
70.3
Table 5: The same grid under uninformative queries, which name no fact. The query column collapses onto recency exactly: with no discriminative token every fact ties and the rule degenerates to its tie-break. dense does not collapse, but gains nothing from the query either (§ 4.3 ).
Retention
Distractors
recency
query
oracle
unbounded
unmarked
31.3
31.3
100.0
unbounded
marked
31.3
41.2
100.0
recency
unmarked
31.3
31.3
46.8
recency
marked
31.3
40.3
46.8
Table 6: Distractor marking control, uninformative queries only. Prefixing distractors with a literal marker string, as the original generator did, lifts the query-aware rule by +9.8 pp [+7.7,+12.2] and +9.0 pp [+7.0,+11.3] while leaving every other rule unchanged. The scorer is detecting the marker, not matching the query.
Context given to the model
Recall
Answer acc.
Acc. ∣ present
No memory
0.0
0.0
—
Bounded ( K=5 ), recency selection
30.7
23.3
70.7
Bounded ( K=5 ), query selection
47.3
36.3
73.2
Full context (all 10 facts)
100.0
73.7
73.7
Required fact alone
100.0
74.7
74.7
Table 7: Answer accuracy (%) with FLAN-T5-large at the end of the pipeline, greedy decoding, 300 queries per condition drawn from the same seeded episodes, normalized exact match. “Acc. ∣ present” conditions on the required fact having reached the model and is close to constant across conditions, which is what licenses recall as the primary metric.
Retention rule
random
recency
query
dense
oracle
Retention
unbounded
31.7
31.3
91.7
97.7
100.0
100.0
recency
29.3
31.3
51.7
53.3
53.3
53.3
random
25.3
29.3
39.3
39.7
40.0
40.0
frequency
40.7
42.3
63.3
64.3
65.3
65.3
type-oracle †
64.0
69.3
86.3
87.7
89.3
89.3
Table 8: The same grid on real SQuAD text : recall (%) of the answer-bearing sentence, 300 questions from SQuAD v1.1 dev, K=5 , k=3 . Each stream mixes the target paragraph’s sentences with sentences from unrelated paragraphs, so nothing marks the distractors. The retention ceiling holds in all 25 cells. Unlike the synthetic benchmark dense now beats query , yet under a bounded buffer it still only reaches the ceiling, never past it.