An answer can match the reference for a complete agent history even when the reader's visible history does not determine it. We study this distinction using executable device logs. Our protocol separates reference agreement, certified evidence sufficiency, and context-appropriate behavior. Insufficiency requires two legal histories yielding byte-identical visible contexts but different answers; public chronology and query cutoffs certify sufficient prefixes. On 80 confirmation histories, Qwen3-32B matches 87 hidden references among 203 insufficient raw-prefix instances, with only seven appropriate abstentions. A fixed-input DeepSeek-V4-Flash replication matches 85 and abstains on 21 of the same 203 instances. Each reader makes 1,600 requests. Compact CSV's estimated appropriateness effects remain inconclusive: +3.16 percentage points, clustered 95% interval [0.00,6.57], for Qwen and +2.46, [-1.84,6.94], for DeepSeek. A separate held-out judge study achieves 99.38% correctness but retains support errors and truncations; a deterministic baseline solves all constructed cases. These results support an evaluation diagnosis across two hosted reader configurations, not gains for a general memory system or natural-language evidence assessment.
Figures & tables
Study
Histories / status
Model requests
Interpretation
Outcome masking
40 development
480
Equal-length evidence intervention; four queries per history
Unordered packing
Same 40 development
2,080
Budget and format exploration, including supplement; shared requests
Judge confirmation
120 reserved
2,400
Constructed candidates; no new reader answers
Chronological Qwen
16 development + 80 confirmation
1,920
Five contexts and four queries; 1,600 confirmation requests
DeepSeek replication
Same 80, Qwen-exposed
1,600
Identical visible inputs; second reader configuration
Table 1: Distinct experimental roles. Request counts are successful unique requests, except that the judge count includes explicitly independent stress-test repetitions. They are not sample sizes for statistical inference. The unordered packing count includes its later 1,280-token supplement.
Qwen3-32B
DeepSeek-V4-Flash
Shared evidence counts
Context
Reference
Appropriate
Reference
Appropriate
Partial S
I
U
Raw prefix
181/320
90/307
172/320
98/307
104
203
13
Whole-record raw
189/320
89/307
172/320
94/307
104
203
13
CSV prefix
195/320
108/298
167/320
111/298
119
179
22
Full raw
166/320
166/320
198/320
198/320
—
0
0
Full CSV
188/320
188/320
196/320
196/320
—
0
0
Table 2: Qwen confirmation and fixed-input DeepSeek replication: the same 80 histories, four queries, five contexts, 1,600 requests per reader. S, I, U denote sufficient, insufficient, and unclassified; full-source cases are sufficient. Appropriate denominators exclude U. Raw and CSV prefixes use 768 Qwen context tokens; whole-record raw averages 728.05. Visible texts are identical across readers.
Certified raw-prefix subset
Qwen
DeepSeek
Sufficient: correct
83/104
77/104
Sufficient: abstains
4/104
23/104
Insufficient: concrete
196/203
182/203
Insufficient: abstains
7/203
21/203
Insufficient: ref. hit
87/203
85/203
Table 3: Conditional raw-prefix behavior on shared inputs. Insufficient reference hits fail the evidence-sensitive contract. Other prefix arms appear in Appendix C .
Evaluator
Coverage
Joint correct
False accepts
Finite program
1,680
1,680
0/480
Extended program
1,920
1,920
0/480
Reference only
1,680
1,320
120/480
LLM low
1,918
1,908
0/480
Program-first + low
1,920
1,920
0/480
Table 4: Judge confirmation over 1,920 constructed cases from 120 reserved histories. Coverage and correctness have denominator 1,920. False accepts refer to 480 scalar negative controls; they exclude the distinct two-axis support errors. Uncovered or invalid cases do not receive credit.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Context
Operator
Reference
Appropriate
Raw prefix
count
40/80
13/76
first
63/80
40/79
last
43/80
19/76
state
35/80
18/76
Whole-record
count
40/80
13/76
first
63/80
40/79
Appendix
Table 5: Qwen operator breakdown on chronological confirmation.
Reader
Certified subset
Raw
Whole
CSV
Qwen
S: correct
83/104
87/104
99/119
S: abstain
4/104
0/104
1/119
I: concrete
196/203
201/203
170/179
I: abstain
7/203
2/203
9/179
I: ref. hit
87/203
91/203
75/179
DeepSeek
S: correct
77/104
74/104
86/119
Appendix
Table 6: Both readers on the same certified subsets. S and I denote partial sufficient and certified insufficient.
DeepSeek contrast
n
Δ pp
95% CI
CSV–raw, appropriate
285
+2.46
[−1.84,6.94]
CSV–raw, reference
320
−1.56
[−5.94,2.81]
Whole–raw, appropriate
307
−1.30
[−4.82,2.66]
Full CSV–raw, reference
320
−0.62
[−5.00,3.44]
Appendix
Table 7: Prespecified secondary contrasts for the second reader. n counts paired queries, not independent histories. The full-source point estimate is exactly −0.625 pp.
Reference agreement
Context appropriateness
Context
n
Δ pp
95% CI
n
Δ pp
95% CI
Raw prefix
320
−2.81
[−8.12,2.19]
307
+2.61
[−2.73,7.92]
Whole-record raw
320
−5.31
[−10.00,−0.94]
307
+1.63
[−3.90,7.12]
CSV prefix
320
−8.75
[−13.75,−4.06]
298
+1.01
[−4.91,7.02]
Full raw
320
+10.00
[1.25,18.44]
320
+10.00
[1.56,18.44]
Full CSV
320
+2.50
[−5.31,10.62]
320
+2.50
[−5.62,10.31]
Appendix
Table 8: DeepSeek minus historical Qwen on shared texts; 80-history paired bootstrap. Date and service changes are confounded with reader identity.
LLM agents that interact with a user across many sessions accumulate histories that exceed their context window, so they store past interactions in an external memory and answer each question from a small set of retrieved records. Existing memory systems rank records by lexical or embedding relevance, yet the top-ranked memories can each be relevant while jointly omitting a complementary fact that the answer requires, especially for multi-session and temporal questions. Drawing on the distinction between relevance and sufficiency in legal evidence scholarship, we recast memory retrieval as constructing a sufficient memory set. To operationalize this view, we introduce a blinded LLM judgment over the retrieved set, together with Gold Hit and Turn Hit as evidence-coverage proxies. We then propose Budgeted Flat Reconstruction (BFR), which builds sufficient sets over a fixed flat memory store in two stages. Specifically, we first apply Formal Concept Analysis for Memory Selection (FCA-MS) to decompose the question into information requirements and select a compact candidate subset that jointly covers them. Then, we repeatedly acquire unseen records through deeper text search or complementary entity and session views, stopping when the budget is exhausted. Experiments on LoCoMo and LongMemEval-S show that BFR outperforms same-store adaptations of recent agent-memory systems in both answer quality and evidence coverage. Specifically, on LongMemEval-S it raises judged accuracy from 72.4% to 82.2% and Turn Hit to 91.4%.
Yufeng Li, Shuxin Li, Zhenhua Xu +7
East China Normal University, Shanghai, China · Nanyang Technological University, Singapore · Zhejiang University, Hangzhou, China +3
A memory can answer a current query correctly while discarding distinctions required by a later update. We investigate this failure with a paired-history audit: two histories have the same current answer, receive a shared future update, and require different subsequent answers. A pilot evaluates 24 history pairs across six synthetic mechanisms, 12 memory conditions, two repeats, and two model backends. A deterministic frontier selector obtains strict reveal accuracy of 96/96 on DeepSeek and 82/96 on GLM; a structured writer obtains 62 successes with one unresolved outcome and 56/96. The configured four-outcome joint contrast has finite-sample identification intervals of [0.521, 0.542] and [0.292, 0.313], not confidence intervals. A record-level audit distinguishes retained-state adequacy, response delivery, and answer-schema compliance without changing those original scores. It finds 26 and 25 well-formed but semantically wrong structured reveal memories, while all 14 GLM frontier reveal failures contain correct values in the wrong wrapper. Tombstone removal produces 16/16 exact replay failures in the targeted mechanism. Identifier renaming then exposes a separate flaw: original frontier late-reference adequacy falls from 8/8 to 94/320 transformed instances. We provide and test a label-equivariant repair, but it preserves only 2/8 original late-reference answers: eliminating a naming shortcut does not solve unknown future relevance. These results support a scoped evaluation methodology and reproducible failure analysis, not general superiority of the repaired algorithm. Paid pilot evidence, retrospective diagnostics, and new offline tests are reported separately; no independent held-out or natural-task validation is claimed.
Long-running LLM agents compress past interactions into persistent memories that may be reused as premises for later tasks. This creates a distinct derivation problem: whether the memory actually follows from what the interaction history supports. Relevant evidence may be scattered across earlier interactions, while compression can introduce relations or event status that the history never established. A valid memory may therefore appear unsupported because its citations omit relevant evidence, while individually supported facts may be composed into a stronger statement the history never established. We characterize this problem through three coupled requirements: (1) Evidence scope; (2) Compositional validity; (3) Admission reliability. We therefore ask whether the interaction history available at write time supports what enters persistent memory. We introduce DerivAudit, a framework for auditing whether a memory is actually supported by the history available when it was written. The audit separates three questions: whether supporting evidence lies beyond writer-provided citations, whether the composed memory introduces unsupported meaning, and how write-time admission decisions affect later memory use. Across two natural memory corpora, audits using broader pre-write history recover support for nearly 60% of memories that appear unsupported from citations alone, while 17-21% remain unsupported after expansion. Yet broader evidence does not by itself make admission reliable: unsupported memories are still frequently admitted across verification models, and evidence expansion alone worsens it on two backbones.