Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression

A language model's memory can be worse than no memory at all. Give a model a memory that kept a wrong conclusion but dropped the work behind it, and it re-emits the stale value as a confident answer; give the same model an empty memory, and it abstains. We call this failure brittle memory. The information loss behind it is definitional (an answer cannot be recomputed once its inputs are gone), so the loss is only the setup; the finding is behavioral. Whether a model turns the lost source into a confident error or an abstention is set by disposition, not capability: four of eight models we test emit, and the four that abstain escape only by an interface affordance -- forced through a mandatory structured-output field, as production tool calls are, they commit the inherited wrong value. We measure correctability with reclaim evaluation: induce a known drift, compress the interaction at a fixed budget, deliver a correction that names the error, and score exact recovery of the known answer, judge-free. Correctability is bottlenecked not by capability but by whether the memory kept a re-derivation basis (the source) rather than the answer, so an 8B model and a frontier one wall in the same place. A one-line source-first policy -- keep the recomputable source, drop the re-derivable conclusion -- restores correctability at equal budget wherever the source is compact and identifiable, with a length-matched control that rules out 'more text' and a deployable one-prompt form weaker than the oracle. We map where the fix fails (source size, noise, a silent truncation mode a completeness tag makes loud), show the failure compounds through memory loops, and replicate on three deployed memory systems and on real dialogue (MultiWOZ, where the checkable value is present by construction). We release the harness, the paired memory conditions, and validators built to come out false.

Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the resulting loss in accuracy, but do not distinguish irreversible losses caused by eviction from recoverable retrieval failures. We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context and reruns the same reader. Combining the change in correctness with whether the evidence was retained after eviction classifies each oracle-answerable error as recoverable, irreversible, or residual; in the residual case, the answer remains incorrect after restoration. We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes, using GPT-4o-mini as the primary reader and judge and GPT-5.4-mini as a robustness reader. Under top-k retrieval at an 80k-token budget, the irreversible share among errors corrected by restoration is 0.67-0.73 for FIFO, random, and redundancy-aware eviction, compared with 0.60 for LLM-importance. At 8k tokens, it reaches 1.00 for all four policies. Recoverable errors occur under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not directly comparable unless the retrieval regime is reported. An exploratory matched-accuracy analysis detects no difference in irreversible rate among accuracy-matched policy pairs at a resolution of 1.2-6 percentage points. The same analysis detects the deliberately destructive control. To our knowledge, this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores on a standard conversational benchmark.

AI systems increasingly retrieve from records that revise themselves: issue threads, encyclopedic histories, policy logs, and long conversations. The challenge is not only finding relevant evidence, but deciding which claims remain in force, which were superseded, and when to abstain. Structured memories promise to solve this with typed edges, temporal updates, and conflict status, yet evaluations often change mechanism and prompt presentation together. We study this as Evidence-State Revision, comparing flat retrieval, coarse edge invalidation, and fine-grained RevisionLedger on 2,907 high-agreement questions from GitHub, multi-repo issue histories, Wikipedia, and DyKnow-style temporal streams. A render-matched control (same layout, deprecation disabled) reveals the central confound: when a value is changed and later restored, RevisionLedger appears to beat a flat baseline by +0.182, but almost all the gain comes from easier presentation; the fine-grained mechanism residual is indistinguishable from zero (+0.021 to +0.025 across two judge families). After presentation is controlled, coarse invalidation is the only mechanism that pays for current-state queries, beating the fine ledger by 0.084; the same query-sufficiency principle says provenance mainly needs retained invalidated evidence, not richer typing. Memory evaluations should hold render fixed, and deprecation-aware systems should deploy the coarsest retained state that covers their queries.