Organizations: Department of Computer and Information Sciences, University of Delaware, USA · Department of Biomedical Informatics and Data Science, University of Alabama at Birmingham, USA · Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, USA · Department of Computer Science and Engineering, BRAC University, Bangladesh · Department of Computer Science, Iowa State University, USA
Agents that answer questions from compressed or retrieved memory must recognize when the evidence a query needs is no longer in memory. Benchmarks for this task usually create insufficient-evidence examples by deleting supporting passages. We show that this construction leaks the label through memory size: on MuSiQue, a classifier that only counts paragraphs reaches an area under the ROC curve (AUROC) of 0.979 for detecting unsafe memory, higher than the lexical estimator we initially evaluated. We propose a size-matched construction that provably removes this shortcut, and use it to study MemSafe, an estimator that cross-encodes the query with each memory unit and aggregates the units with a set transformer. Across three multi-hop question answering datasets and five seeds, MemSafe reaches 0.968 and 0.983 AUROC on MuSiQue and HotpotQA, 0.26 to 0.39 above a lexical baseline, while the third dataset, 2WikiMultiHopQA, is saturated. A frozen pretrained cross-encoder with a logistic head already closes 41% of the MuSiQue gap between the lexical baseline and MemSafe. At the same time, MemSafe degrades more than a weak baseline on the unanswerable questions released with MuSiQue, reaches only 0.639 AUROC on SQuAD~2.0, and needs several thousand clinical training examples before it outperforms a feature-based estimator. Used as a gate for a 7B reader, it reduces the error rate on answered questions from 0.850 to 0.631 at 5% coverage, outperforming both reader confidence and, on average, the ground-truth integrity label, although a 7B LLM judge is the better gate at 10% coverage. These results indicate that the way insufficient evidence is constructed matters as much as the estimator that detects it.
Figures & tables
Figure 1: Overview of MemSafe . Stage I fixes the memory size at ∣D∣ for every integrity state, so size carries no label information (Proposition 1 ). Stage II cross-encodes each (q,ui) pair and a bridge-conditioned pair (q∥utop,ui) (Stage A), and a two-layer set transformer (Stage B) predicts the unsafe probability, the integrity state, per-unit evidence presence si , and the number of missing evidence items g . At test time , a threshold τ chosen on validation data at a 5% risk budget decides whether to answer. Values are MuSiQue test-split means.
Figure 2: Size matching removes the memory-size shortcut. Left: MuSiQue test scores under the deletion construction (taken from the pre-fix run ledger) and under size matching. Right: unsafe AUROC of the surface controls and LexSet under size matching. The dotted line marks chance.
MuSiQue
HotpotQA
2WikiMultiHopQA
Model
AUROC ↑
Macro F1 ↑
AUROC ↑
Macro F1 ↑
AUROC ↑
Macro F1 ↑
MemSafe (ours)
0.9682
0.7960
0.9831
0.8600
0.9999
0.9894
Encoder + mean pool
0.9484
0.7665
0.9765
0.8423
0.9997
0.9880
Relevance aggregation
0.9215
0.7136
0.9612
0.8146
0.9947
0.9758
Concatenated cross-encoder
0.8912
0.5811
0.9590
0.7575
0.9983
0.9579
8k joint encoder (three seeds; exploratory)
0.9019
0.5974
0.9559
0.7534
n/a
n/a
Table 1: Test-split unsafe AUROC and integrity macro F1, averaged over five seeds (three for the concatenated and 8k joint encoders). Seed-level standard deviations of the learned models are at most 0.0031 ; deterministic models have none. n/a: not run. Bold marks the best value in each column; ↑ means higher is better.
Variant
Unsafe AUROC ↑ ( Δ )
Macro F1 ↑ ( Δ )
Full MemSafe
0.9674±0.0008
0.7929±0.0022
w/o encoder fine-tuning (frozen encoder)
0.9121 ( −0.0553 )
0.6333 ( −0.1596 )
w/o bridge-conditioned encoding
0.9423 ( −0.0251 )
0.7551 ( −0.0377 )
w/o set model (mean pooling)
0.9502 ( −0.0173 )
0.7664 ( −0.0265 )
w/o lexical and position features
0.9637 ( −0.0038 )
0.7839 ( −0.0090 )
w/o localization heads
0.9677 ( +0.0002 )
0.7915 ( −0.0014 )
Table 2: Single-component ablations on MuSiQue (test split, three Stage B seeds sharing one encoder). Each row removes one component from the full model, except the last, which removes both fine-tuning and the set model. Bold marks the best value in each column; ↑ means higher is better.
Figure 3: Left: clinical learning curve on MIMIC-IV OMR at a fixed event source and patient density ( 1/16 of patients, three seeds per point). LexSet leads at the smallest scale, and MemSafe overtakes it with about 2.5 times as many training variants; realized patient counts also grow with the cap. Right: selective answering with the 7B reader. Gating on the predicted unsafe score yields lower error than answering everything, than reader confidence, and, on average over three estimator seeds, than the ground-truth integrity label at every coverage level shown; the LLM judge is the better gate at 10% coverage.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
H
Interaction history
M , Mz
Memory retained from H ; memory constructed for integrity state z
q
Query
E(q)
Evidence items required to answer q
ui , n
The i -th memory unit; number of units in M
z
Integrity state: complete , missing , relation-lost , or stale
Appendix
Table 3: Notation.
Dataset
Raw records
Base questions
Excluded
Variants
MuSiQue
44,710
21,902
22,808
65,297
HotpotQA
97,852
97,406
446
292,218
2WikiMultiHopQA
192,606
180,030
12,576
540,090
Appendix
Table 4: Processed evidence-QA datasets. Every raw record is used or excluded for a recorded reason, and every run uses the complete processed dataset.
Figure 4: The pinned 8k ModernBERT joint encoder improves on the original truncated concatenated baseline on MuSiQue but remains below MemSafe and the matched mean-pooling comparator on both datasets. Bars show three-seed means with sample standard-deviation error bars.
Figure 5: Independent-construction transfer from MuSiQue to SQuAD 2.0. MemSafe retains positive discrimination and exceeds the controls, but the absolute performance gap exposes the generalization limitation.
Figure 6: Unsafe AUROC on the test split for the attribution ladder (five-seed means with seed-level standard deviations; the dotted line marks chance). The first jump combines architecture and fine-tuning; Figure 7 and Table 2 separate them on MuSiQue.
Figure 7: Fine-tuning and the set model provide overlapping gains (MuSiQue, test split). Arrows show the gain from adding the set model: +0.171 unsafe AUROC with a frozen encoder and +0.046 with a fine-tuned one. Filled points average three Stage B seeds that share one encoder (Table 2 ); the hollow point is the five-seed comparator from Table 1 , so the figure mixes two replication scopes. The dashed line marks LexSet .
Figure 8: Left: unsafe AUROC on the size-matched family A and on MuSiQue-Full’s native contrasts (family B). Right: change in unsafe AUROC when evaluation is restricted to questions whose gold evidence never appears in training. The lexical baselines lose accuracy; the proposed estimator does not.
Dataset
Model
AURC ↓
Coverage at 5% target ↑
Realized test risk
MuSiQue
MemSafe
0.3197
0.2479
0.0553
Encoder + mean pool
0.3359
0.1334
0.0463
Relevance aggregation
0.3626
0.0206
0.0627
LexSet
0.6088
0.0002
0.7333
HotpotQA
MemSafe
0.3120
0.2922
0.0505
Encoder + mean pool
0.3166
0.2750
0.0514
Appendix
Table 5: Selective prediction with an evidence-aware risk definition. AURC is lower-is-better; coverage is the test fraction accepted at a threshold chosen on validation for a 5% risk target, and the last column is the risk realized at that threshold. n/a: no realized risk is reported for this row. Bold marks the best value within each dataset; ↓ and ↑ mean lower and higher is better. Realized risk should be close to the target rather than minimal, so it is not bolded. 2WikiMultiHopQA values are not bolded because both encoder models sit at the prevalence limits (coverage 0.3509 , AURC 0.30046 ; Appendix E.5 ).
Figure 9: Risk-coverage curves on the test split (seed 17). The dotted line marks a 5% evidence-aware risk budget. Curves that reach further right before crossing it support higher usable coverage.
Error among answered questions ↓
Gate
5% coverage
10%
25%
50%
MemSafe unsafe score
0.6306±0.0206
0.6824±0.0039
0.7084±0.0069
0.7687±0.0027
LLM judge (zero-shot, 7B)
0.6419
0.6700
0.7389
0.7786
Oracle integrity label
0.7230
0.7205
0.7133
0.7793
Reader sequence confidence
0.7027
0.7744
0.8398
0.8264
Flat TF-IDF logistic
0.7838
0.7845
0.8008
0.8176
Appendix
Table 6: Selective answering on MuSiQue with the 7B reader: error among answered questions at fixed coverage (lower is better). Answering everything is wrong 85.03% of the time. MemSafe and LexSet use three independently trained estimator seeds (mean ± sample standard deviation); every other row is deterministic given the reader and the subset, so it has no seed variance. The oracle row gates on the true integrity label, which is not an oracle for answer correctness. The LLM judge is the better gate at 10% coverage. Bold marks the lowest error in each column.
Figure 10: Reader gating under lower unsafe-memory prevalence (MuSiQue, 7B reader, one estimator seed). Complete memories stay fixed and unsafe memories are subsampled; 66% is the construction’s native rate. Error among answered questions at 5% (left) and 10% (right) coverage: the MemSafe gate lies below both the oracle integrity label and answering everything at every prevalence tested.
Setting
Patients
Train variants
MemSafe AUROC ↑
LexSet AUROC ↑
Δ AUROC (seed wins)
Δ Macro F1
OMR, 1/64 patients, 1k cap
683
2,780
0.9220
0.9649
−0.0429 (0/5)
+0.0880
OMR, 1/16 patients, 60k cap
3,934
29,920
0.9960
0.9725
+0.0236 (5/5)
+0.1475
Laboratory events, 1/16 , 60k cap
7,869
166,560
0.9908
0.9567
+0.0341 (5/5)
+0.0480
Appendix
Table 7: MIMIC-IV test-split results over five seeds. The first two rows share an event source but differ in patient sampling fraction and question cap, so the change in sign points to a scale effect rather than a modality effect without isolating training-data quantity from cohort composition. Positive differences favor MemSafe . Bold marks the better of the two estimators in each row; ↑ means higher is better.
Cap
Patients
Train variants
MemSafe AUROC ↑
LexSet AUROC ↑
Δ AUROC (seed wins)
Δ Macro F1
1,000
929
2,788
0.9462
0.9626
−0.0163 (1/3)
+0.0991
2,500
2,002
6,844
0.9911
0.9648
+0.0263 (3/3)
+0.1774
5,000
3,113
13,724
0.9935
0.9704
+0.0231 (3/3)
+0.1733
10,000
3,906
27,788
0.9955
0.9736
+0.0219 (3/3)
+0.1614
20,000
3,934
29,920
0.9957
0.9725
+0.0232 (3/3)
+0.1181
Appendix
Table 8: Locating the sample-efficiency crossover: a five-point learning curve on MIMIC-IV OMR test-split results, holding the event source and patient sampling fraction fixed at 1/16 and varying only the base-question cap (three seeds per point, fewer than the five seeds used elsewhere); realized patient counts grow with the cap. At 1/16 sampling, the 20,000 and 60,000 caps both exceed the available questions and yield the identical data selection (same selection hash), so the 20,000 -cap row reproduces the scale-matched OMR setting of Table 7 on a seed subset ( 0.9957 versus 0.9960 unsafe AUROC for MemSafe ; LexSet matches at 0.9725 in both). Positive differences favor MemSafe . Bold marks the better of the two estimators in each row; ↑ means higher is better.
Model
MuSiQue ↓
HotpotQA ↓
2WikiMultiHopQA ↓
MemSafe
0.0003
0.0003
0.0000
Encoder + mean pool
0.0018
0.0002
0.0000
Relevance aggregation
0.0008
0.0001
0.0005
Concatenated cross-encoder
0.0009
0.0009
0.0001
LexSet
0.0009
0.0002
0.0031
Appendix
Table 9: Seed-level standard deviation of test unsafe AUROC for learned models (five seeds; three for the concatenated cross-encoder). Lower means more stable across seeds ( ↓ ); bold marks the lowest value in each column.
Long-term memory enables language models to use past interactions in future conversations. However, evidence needed to answer a question may be scattered across distant turns, while the question itself omits clues needed to locate it. Retrieved facts can reveal these clues, motivating retrieval decisions conditioned on evidence already found. We introduce MERA (Missing-Evidence Retrieval Augmentation), which separates globally searchable memory from a question-specific evidence state. Verified evidence guides subsequent retrieval without restricting access to the global memory. We train a lightweight planner through reinforcement learning, rewarding queries that recover previously missing evidence. MERA achieves strong answer accuracy across Qwen3-30B and GPT-4o-mini backbones. With Qwen3-30B for evidence processing and answer generation, the trained 0.6B planner achieves 77.40% accuracy on LoCoMo and 71.29% on LongMemEval-S, exceeding a 30B planner without retrieval-grounded training by 4.10% and 3.96%, respectively. On LoCoMo, later retrieval rounds increase cumulative evidence recall from 55.5% to 80.5%.
LLM agents that interact with a user across many sessions accumulate histories that exceed their context window, so they store past interactions in an external memory and answer each question from a small set of retrieved records. Existing memory systems rank records by lexical or embedding relevance, yet the top-ranked memories can each be relevant while jointly omitting a complementary fact that the answer requires, especially for multi-session and temporal questions. Drawing on the distinction between relevance and sufficiency in legal evidence scholarship, we recast memory retrieval as constructing a sufficient memory set. To operationalize this view, we introduce a blinded LLM judgment over the retrieved set, together with Gold Hit and Turn Hit as evidence-coverage proxies. We then propose Budgeted Flat Reconstruction (BFR), which builds sufficient sets over a fixed flat memory store in two stages. Specifically, we first apply Formal Concept Analysis for Memory Selection (FCA-MS) to decompose the question into information requirements and select a compact candidate subset that jointly covers them. Then, we repeatedly acquire unseen records through deeper text search or complementary entity and session views, stopping when the budget is exhausted. Experiments on LoCoMo and LongMemEval-S show that BFR outperforms same-store adaptations of recent agent-memory systems in both answer quality and evidence coverage. Specifically, on LongMemEval-S it raises judged accuracy from 72.4% to 82.2% and Turn Hit to 91.4%.
Yufeng Li, Shuxin Li, Zhenhua Xu +7
East China Normal University, Shanghai, China · Nanyang Technological University, Singapore · Zhejiang University, Hangzhou, China +3
Long-term conversational memory requires retrieving evidence scattered across multiple sessions, yet single-pass retrieval fails on temporal and multi-hop questions. Existing iterative methods refine queries via generated content or document-level signals, but none explicitly diagnoses the evidence gap, namely what is missing from the accumulated retrieval set, leaving query refinement untargeted. We present EviMem, combining IRIS (Iterative Retrieval via Insufficiency Signals), a closed-loop framework that detects evidence gaps through sufficiency evaluation, diagnoses what is missing, and drives targeted query refinement, with LaceMem (Layered Architecture for Conversational Evidence Memory), a coarse-to-fine memory hierarchy supporting fine-grained gap diagnosis. On LoCoMo, EviMem improves Judge Accuracy over MIRIX on temporal (73.3% to 81.6%) and multi-hop (65.9% to 85.2%) questions at 4.5x lower latency. Code: https://github.com/AIGeeksGroup/EviMem.