A stored experience may answer one future question and offer little help with another. We measure this task-conditioned utility rather than assigning each memory a single importance score. A task-blind writer stores a memory without seeing the later task; a frozen reader then answers that task with and without the memory, using identical filler. A cost ledger records the resources used to form, store and expose the memory. On single-call factual-recall tasks built from LongMemEval-V2 web-agent memories, the same memory helps more when it contains information needed for the answer. Its mean paired reward gain on these answer-bearing task pairs exceeds that on matched-control pairs by +0.33 (95% interval +0.21 to +0.46). The tested task-blind summaries and structured records yield smaller gains than Raw, including at fixed total context; a budget sweep is consistent with retention of answer-bearing content contributing to the difference. The bearing--control contrast remains positive in a new session with randomized memory position and under a second reader, although individual memory--task labels change. Generic retrieval performs above chance but misses most screen-defined answer-bearing memories. The factual-recall gains do not extend to the tested software-engineering and scientific-coding benchmarks, where even raw memories have no measurable effect. We provide a public, versioned Conditional Memory Utility Dataset of paired measurements and costs. The practical takeaway is to evaluate memory compression by the task utility it preserves alongside its costs. Task-conditioned measurements make this comparison possible; their stability must also be checked.
Figures & tables
Figure 1. Conditional memory utility evaluation. (a) Page-visit memory units extracted from past episodes are stored as Raw, Summary or Structured representations without exposing the writer to future evaluation tasks. (b) For a memory–task pair, a frozen reader is evaluated with identical filler F , either with or without the target memory m ; the paired reward difference defines ΔU(m,t) under the recorded evaluation conditions. (c) Repeating this comparison across tasks, representations and replicates yields a task-conditioned utility profile. Formation, storage, retrieval and exposure costs are recorded alongside the measurements (the cost ledger). A three-panel diagram. Panel a: a memory unit (one page visit) passes to a writer that produces Raw, Summary or Structured text, stored in a frozen bank; the writer never observes the later task. Panel b: the same later task and identical filler feed two arms, without and with the target memory, each answered by the same frozen reader; their rewards give Delta U equals R with minus R without. Panel c: repeating this gives a profile of Delta U over tasks, with a cost ledger of formation, storage, retrieval and exposure.
run
memories
pairs × conditions
excluded
LongMemEval-V2
screened design
60
1,157 pairs
0 rows
representations
57
277 × 3 representations
3 memories
length control
57
277 × 8 arms
6 rows
second reader
60
275 of 285 pairs
20 rows
gold-free screen
–
32 + 42 pairs
–
Table 1. Every run a reported number comes from. Counts are memory–task pairs times the conditions each pair was evaluated under (representations, arms or replicates, as named). A dataset row is one replicate of one pair under one condition and records both arms. Excluded rows had no producible reward and are dropped, never scored as zero.
Figure 2. Answer bearing is sparse at every screen cap (exploratory). (a) Number of memory units (of 1,889 ) that bear on each count of screened future tasks, at the paper’s cap of eight units per task (log axis). (b) Share of bearing memories that bear on exactly one task when the cap is 4, 8, 16 or 32 units; n is the number of bearing memories. The median breadth is one at every cap. Two panels. Top: a log-scale histogram of how many tasks each memory bears on, with 1,724 memories at zero and 119 at one, falling quickly after. Bottom: at screen caps of 4, 8, 16 and 32, between 64 and 72 percent of bearing memories bear on exactly one task.
Figure 3. The bearing–control contrast remains positive across tested conditions (LongMemEval-V2). (a) Bearing minus control, paired within memory, in the original run, in a new session with the memory’s position redrawn per replicate, and under a second reader (95% intervals over memories). (b) Bearing-cell ΔU of each representation at its own length ( n=135 ) and with every arm padded to the Raw arm’s token count ( n=77 , excluding cells exposed to a cache-key defect); task-clustered intervals. Pre-registered: the original run and the second reader in (a); exploratory: the new session in (a) and panel (b). Two panels. Top: three interval estimates of the bearing-minus-control effect, for the original run, a new session with random position, and a second reader, all well above zero. Bottom: Raw near 0.35 and Summary and Structured near zero, both at their own length and at a fixed total context.
stratum
both-arms-fail
Wilson 95%
mean ΔU
bearing
0.53
[0.44, 0.61 ]
+0.3453
matched control
0.79
[0.71, 0.85]
+0.0274
random
0.70
[0.67, 0.73]
+0.0642
Table 2. Screened run on LongMemEval-V2: the same 60 memories paired with three strata of tasks. Both-arms-fail is the fraction of cells in which the reader fails with and without the memory; such cells cannot show a memory effect. The criterion for a material reduction ( bearing ’s Wilson upper bound below the comparator’s point estimate) was fixed in code before these numbers existed.
Figure 4. What carries utility (LongMemEval-V2). (a) Bearing-cell ΔU as Raw is cut to its last N tokens ( 135 cells, task-clustered intervals); labels give the share of cells whose text still contains the answer. (b) Share of a task’s bearing units that a retriever ranks in its top 1, 5 and 10 of the domain pool ( 75 tasks; task-clustered interval shown at top 10); the dashed line is chance. Both panels are exploratory; the matched-length contrast in (a) was pre-registered. Two panels. Top: bearing-cell utility rising with the number of Raw tokens kept, with the share of cells keeping the answer rising from 29 to 100 percent. Bottom: recall at ten for BM25, prefix and chunked dense retrieval, from 0.21 to 0.40, far above a near-zero chance line.
Figure 5. What single cells can and cannot show (exploratory). (a) Per-cell labels (helped, unchanged, hurt) in the original run against a new session at the same memory position ( 156 replicate cells; diagonal agrees). At a new position κ falls to 0.65 , and between two readers it is 0.58 . (b) Help among cells the reader got wrong without the memory (filled) and harm among cells it got right (hollow), by stratum, Wilson 95% intervals: help differs sharply, harm does not detectably. Two panels. Top: Cohen's kappa of 0.78, 0.65 and 0.58 for a new session at the same position, a new session at a new position, and a second reader, with 91, 87 and 82 percent raw agreement. Bottom: help rates of about 41 percent for bearing against 8 and 12 percent for control and random, while harm rates overlap across strata.
Figure 6. Position affects per-pair measurements; the aggregate bearing–control contrast remains positive after position standardization (exploratory). Per-cell ΔU by the memory’s position among the five context units in the second-session run, with task-clustered 95% bands: the bearing effect dips in the middle and recovers at the end, while the control stays near zero at every position. A line chart over five positions. The bearing line is about 0.44 and 0.42 at the first two positions, about 0.29 and 0.30 in the middle, and 0.43 at the last, with a wide band; the control line stays near zero throughout.
Figure S2. Length-control run on LongMemEval-V2: ΔU on the 135 bearing cells, task-clustered 95% intervals. (a) Raw truncated to its last 4,096 and 1,024 tokens and to each cell’s Summary length (line), against Summary, Structured and the first- N window at that length; percentages give the share of cells whose text still contains the bearing content. (b) The same arms split by whether their text kept the bearing content (filled) or lost it (hollow). Left: bearing-cell utility rising with exposed tokens from about 0.08 at 280 tokens to 0.34 at 8,000, with Summary, Structured and first-N near zero. Right: for each arm, utility near 0.25 to 0.37 when the bearing content is kept and near zero when it is lost, except Structured whose kept estimate is near zero with a wide interval.
arm
median tokens
kept
ΔU
ΔU∣ kept
ΔU∣ lost
Raw (full)
8,344
100%
+0.34 [+0.21, +0.49]
+0.34
–
last 4,096
4,096
84%
+0.31 [+0.19, +0.45]
+0.36
+0.05
last 1,024
1,024
48%
+0.19 [+0.09, +0.31]
+0.37
+0.03
last N (matched)
282
29%
+0.08 [-0.01, +0.18]
+0.31
-0.01
Summary
282
18%
+0.03 [-0.06, +0.12]
+0.25
-0.02
Structured
216
13%
-0.04 [-0.13, +0.04]
+0.00
-0.05
Table S1. Length-control run on LongMemEval-V2: bearing-cell ΔU by arm ( 135 cells, task-clustered 95% intervals), share of cells whose text still contains the bearing content, and ΔU split by whether it does. Exploratory beyond the three pre-registered contrasts.
Figure S3. Exposure efficiency ( ΔU per thousand charged tokens) against the reuse factor, on the 277 representation cells of LongMemEval-V2. Lines use each representation’s paired bearing − control effect; bands use its 95% interval over memories. Raw has no writer call and does not depend on r . The vertical line marks the reuse factor of this study. Line chart of efficiency against reuse factor on a log axis: Raw is flat near 0.034, Summary rises and crosses Raw near r of 5.5 with a wide band spanning zero, Structured is below zero.
contrast
benchmark
n
mean
95% CI
MDE
Summary − Raw
LongMemEval-V2
277
-0.1408
[-0.2211, -0.0601]
0.12
SciCode
122
+0.0246
[-0.0342, +0.0991]
0.10
SWE-bench
40
-0.1000
[-0.2250, +0.0000]
0.16
Structured − Raw
LongMemEval-V2
277
-0.1625
[-0.2454, -0.0798]
0.12
SciCode
122
+0.0328
[-0.0342, +0.1200]
0.11
SWE-bench
40
-0.0500
[-0.1750, +0.0750]
0.18
Table S2. Per-cell representation contrasts: ΔU under one representation minus ΔU under Raw on the same memory and task, with task-clustered 95% intervals. Only LongMemEval-V2 separates from zero. Every SciCode and SWE-bench interval includes zero (SWE-bench’s Summary − Raw only at its upper bound), and doubling the SciCode cell count separated no contrast. MDE is the minimum detectable contrast at 80% power and two-sided α=0.05 , computed from each interval’s width. Raw’s own with-versus-without ΔU is −0.01[−0.05,+0.02] on SciCode and −0.05 on SWE-bench, so on these benchmarks no memory effect exists for a representation to preserve, and the contrasts cannot test the ordering.
Table S3. Screened run on LongMemEval-V2: transitions between the no-memory and with-memory outcome (binary reward). Accuracy columns give the share of cells answered correctly without and with the memory. Hurt is conditioned on cells correct without the memory, help on cells wrong without it; Wilson 95% intervals in brackets.
Figure S4. Sign reversal on LongMemEval-V2, screened design (1,157 cells, 60 memories). (a) Help rate among cells the reader got wrong without the memory (filled) and harm rate among cells it got right (hollow), with Wilson 95% intervals; counts are in Table S3 . Harm does not differ detectably across strata; help does. (b) Flips among the 35 memories with at least two bearing cells when each memory’s random cells are subsampled to its own bearing -cell count (20,000 resamples); the line marks the observed bearing count. Left: dot plot of help and harm rates by stratum; help is about 41 percent for bearing against 8 and 12 percent elsewhere, harm overlaps across strata at 12 to 29 percent with wide intervals. Right: histogram of flip counts at matched cell counts, with zero flips in about 63 percent of resamples.
analysis
anchor
recorded deviation
screened design and its reduction criterion
criterion fixed in code before the numbers existed
none recorded
reversal rate (headline)
pre-registration, section 5
comparison with an additive null added later, as a diagnostic
variance decomposition
pre-registration, section 6
not identifiable here (zero execution floor); in earlier arms the mixed model did not converge and the registered fallback was used
gold-free screens
gold-free screen pre-commitment
none recorded
length control
frozen at a recorded commit, before execution
3 of 27 descriptive figures do not re-derive: bearing quartile means by Raw length; none appears in the paper
second reader
written before any design-cell call
cross-session estimate superseded by a session-matched run
Table S4. Pre-registered analyses, their anchors, and recorded deviations.
arm
excluding exposed cells
all cells
Raw
+0.35 [+0.23, +0.49]
+0.37 [+0.25, +0.51]
Summary
+0.04 [-0.04, +0.12]
+0.03 [-0.03, +0.09]
Structured
+0.01 [-0.04, +0.07]
-0.01 [-0.07, +0.03]
matched truncation
+0.04 [-0.01, +0.11]
+0.11 [+0.03, +0.20]
Summary − Raw
-0.31 [-0.44, -0.18]
-0.34 [-0.48, -0.21]
Structured − Raw
-0.34 [-0.47, -0.21]
-0.39 [-0.52, -0.26]
Table S5. Bearing-cell ΔU at a fixed total context (task-clustered 95% intervals).
method
recall@1
recall@10
MRR
AUROC
BM25
0.03
0.21 [0.14, 0.28]
0.18
0.86
dense, prefix
0.11
0.35 [0.26, 0.44]
0.36
0.82
dense, chunks
0.14
0.40 [0.31, 0.50]
0.41
0.85
Table S6. Retrieval of bearing units (task-clustered 95% intervals; chance recall@10 is about 0.01). MRR is the mean reciprocal rank of the first bearing unit; AUROC is the area under the ROC curve over bearing and non-bearing units.
Figure S5. Overview of further results on LongMemEval-V2 (a–d, f) and across benchmarks (e). (a) Future tasks each of the 1,889 memory units bears on, over 75 discriminative tasks (log axis). (b) Share of cells where the memory helped, cell-level Wilson 95% intervals (memory-resampled ones in Section S7 ), for the same 60 memories in three strata. (c) Share of bearing units a retriever ranks in its top ten of the domain pool (75 tasks, task-clustered 95% intervals); the dashed line is chance. (d) Paired bearing − control effect per representation ( 277 cells, 57 memories, intervals over memories); hollow markers count exposed tokens, filled ones add formation cost amortized over 4.86 cells. (e) Per-cell ΔU minus Raw’s, task-clustered intervals; filled markers exclude zero. (f) Two readers’ labels on 275 Raw cells in one session; the hatched cell (zero under both) cannot disagree. Six panels: a log histogram of bearing tasks per memory unit peaking at one; bar chart of helped-cell rates by stratum with bearing highest; similarity sextile lines nearly flat; utility against tokens showing Raw above zero and compressed representations near zero; cross-benchmark contrast intervals, only LongMemEval-V2 excluding zero; a three by three reader agreement table.