cs.CLSep 26, 2026
SaveBefore Answering: Evidence Sufficiency under Size-Matched Memory Construction
Organizations: Department of Computer and Information Sciences, University of Delaware, USA · Department of Biomedical Informatics and Data Science, University of Alabama at Birmingham, USA · Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, USA · Department of Computer Science and Engineering, BRAC University, Bangladesh ·
Abstract
Agents that answer questions from compressed or retrieved memory must recognize when the evidence a query needs is no longer in memory. Benchmarks for this task usually create insufficient-evidence examples by deleting supporting passages. We show that this construction leaks the label through memory size: on MuSiQue, a classifier that only counts paragraphs reaches an area under the ROC curve (AUROC) of for detecting unsafe memory, higher than the lexical estimator we initially evaluated. We propose a size-matched construction that provably removes this shortcut, and use it to study MemSafe, an estimator that cross-encodes the query with each memory unit and aggregates the units with a set transformer. Across three multi-hop question answering datasets and five seeds, MemSafe reaches and AUROC on MuSiQue and HotpotQA, to above a lexical baseline, while the third dataset, 2WikiMultiHopQA, is saturated. A frozen pretrained cross-encoder with a logistic head already closes of the MuSiQue gap between the lexical baseline and MemSafe. At the same time, MemSafe degrades more than a weak baseline on the unanswerable questions released with MuSiQue, reaches only AUROC on SQuAD~2.0, and needs several thousand clinical training examples before it outperforms a feature-based estimator. Used as a gate for a 7B reader, it reduces the error rate on answered questions from to at coverage, outperforming both reader confidence and, on average, the ground-truth integrity label, although a 7B LLM judge is the better gate at coverage. These results indicate that the way insufficient evidence is constructed matters as much as the estimator that detects it.
Figures & tables
Figure 1: Overview of MemSafe . Stage I fixes the memory size at for every integrity state, so size carries no label information (Proposition 1 ). Stage II cross-encodes each pair and a bridge-conditioned pair (Stage A), and a two-layer set transformer (Stage B) predicts the unsafe probability, the integrity state, per-unit evidence presence , and the number of missing evidence items . At test time , a threshold chosen on validation data at a risk budget decides whether to answer. Values are MuSiQue test-split means.
Figure 2: Size matching removes the memory-size shortcut. Left: MuSiQue test scores under the deletion construction (taken from the pre-fix run ledger) and under size matching. Right: unsafe AUROC of the surface controls and LexSet under size matching. The dotted line marks chance.
| MuSiQue | HotpotQA | 2WikiMultiHopQA | ||||
|---|---|---|---|---|---|---|
| Model | AUROC | Macro F1 | AUROC | Macro F1 | AUROC | Macro F1 |
| MemSafe (ours) | ||||||
| Encoder + mean pool | ||||||
| Relevance aggregation | ||||||
| Concatenated cross-encoder | ||||||
| 8k joint encoder (three seeds; exploratory) | n/a | n/a | ||||
Table 1: Test-split unsafe AUROC and integrity macro F1, averaged over five seeds (three for the concatenated and 8k joint encoders). Seed-level standard deviations of the learned models are at most ; deterministic models have none. n/a: not run. Bold marks the best value in each column; means higher is better.
| Variant | Unsafe AUROC ( ) | Macro F1 ( ) |
|---|---|---|
| Full MemSafe | ||
| w/o encoder fine-tuning (frozen encoder) | ( ) | ( ) |
| w/o bridge-conditioned encoding | ( ) | ( ) |
| w/o set model (mean pooling) | ( ) | ( ) |
| w/o lexical and position features | ( ) | ( ) |
| w/o localization heads | ( ) | ( ) |
Table 2: Single-component ablations on MuSiQue (test split, three Stage B seeds sharing one encoder). Each row removes one component from the full model, except the last, which removes both fine-tuning and the set model. Bold marks the best value in each column; means higher is better.
Figure 3: Left: clinical learning curve on MIMIC-IV OMR at a fixed event source and patient density ( of patients, three seeds per point). LexSet leads at the smallest scale, and MemSafe overtakes it with about times as many training variants; realized patient counts also grow with the cap. Right: selective answering with the 7B reader. Gating on the predicted unsafe score yields lower error than answering everything, than reader confidence, and, on average over three estimator seeds, than the ground-truth integrity label at every coverage level shown; the LLM judge is the better gate at coverage.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| Interaction history | |
| , | Memory retained from ; memory constructed for integrity state |
| Query | |
| Evidence items required to answer | |
| , | The -th memory unit; number of units in |
| Integrity state: complete , missing , relation-lost , or stale |
Table 3: Notation.
| Dataset | Raw records | Base questions | Excluded | Variants |
|---|---|---|---|---|
| MuSiQue | 44,710 | 21,902 | 22,808 | 65,297 |
| HotpotQA | 97,852 | 97,406 | 446 | 292,218 |
| 2WikiMultiHopQA | 192,606 | 180,030 | 12,576 | 540,090 |
Table 4: Processed evidence-QA datasets. Every raw record is used or excluded for a recorded reason, and every run uses the complete processed dataset.
Figure 4: The pinned 8k ModernBERT joint encoder improves on the original truncated concatenated baseline on MuSiQue but remains below MemSafe and the matched mean-pooling comparator on both datasets. Bars show three-seed means with sample standard-deviation error bars.
Figure 5: Independent-construction transfer from MuSiQue to SQuAD 2.0. MemSafe retains positive discrimination and exceeds the controls, but the absolute performance gap exposes the generalization limitation.
Figure 6: Unsafe AUROC on the test split for the attribution ladder (five-seed means with seed-level standard deviations; the dotted line marks chance). The first jump combines architecture and fine-tuning; Figure 7 and Table 2 separate them on MuSiQue.
Figure 7: Fine-tuning and the set model provide overlapping gains (MuSiQue, test split). Arrows show the gain from adding the set model: unsafe AUROC with a frozen encoder and with a fine-tuned one. Filled points average three Stage B seeds that share one encoder (Table 2 ); the hollow point is the five-seed comparator from Table 1 , so the figure mixes two replication scopes. The dashed line marks LexSet .
Figure 8: Left: unsafe AUROC on the size-matched family A and on MuSiQue-Full’s native contrasts (family B). Right: change in unsafe AUROC when evaluation is restricted to questions whose gold evidence never appears in training. The lexical baselines lose accuracy; the proposed estimator does not.
| Dataset | Model | AURC | Coverage at target | Realized test risk |
|---|---|---|---|---|
| MuSiQue | MemSafe | |||
| Encoder + mean pool | ||||
| Relevance aggregation | ||||
| LexSet | ||||
| HotpotQA | MemSafe | |||
| Encoder + mean pool |
Table 5: Selective prediction with an evidence-aware risk definition. AURC is lower-is-better; coverage is the test fraction accepted at a threshold chosen on validation for a risk target, and the last column is the risk realized at that threshold. n/a: no realized risk is reported for this row. Bold marks the best value within each dataset; and mean lower and higher is better. Realized risk should be close to the target rather than minimal, so it is not bolded. 2WikiMultiHopQA values are not bolded because both encoder models sit at the prevalence limits (coverage , AURC ; Appendix E.5 ).
Figure 9: Risk-coverage curves on the test split (seed 17). The dotted line marks a evidence-aware risk budget. Curves that reach further right before crossing it support higher usable coverage.
| Error among answered questions | ||||
|---|---|---|---|---|
| Gate | coverage | |||
| MemSafe unsafe score | ||||
| LLM judge (zero-shot, 7B) | ||||
| Oracle integrity label | ||||
| Reader sequence confidence | ||||
| Flat TF-IDF logistic | ||||
Table 6: Selective answering on MuSiQue with the 7B reader: error among answered questions at fixed coverage (lower is better). Answering everything is wrong of the time. MemSafe and LexSet use three independently trained estimator seeds (mean sample standard deviation); every other row is deterministic given the reader and the subset, so it has no seed variance. The oracle row gates on the true integrity label, which is not an oracle for answer correctness. The LLM judge is the better gate at coverage. Bold marks the lowest error in each column.
Figure 10: Reader gating under lower unsafe-memory prevalence (MuSiQue, 7B reader, one estimator seed). Complete memories stay fixed and unsafe memories are subsampled; is the construction’s native rate. Error among answered questions at (left) and (right) coverage: the MemSafe gate lies below both the oracle integrity label and answering everything at every prevalence tested.
| Setting | Patients | Train variants | MemSafe AUROC | LexSet AUROC | AUROC (seed wins) | Macro F1 |
|---|---|---|---|---|---|---|
| OMR, patients, 1k cap | 683 | 2,780 | (0/5) | |||
| OMR, patients, 60k cap | 3,934 | 29,920 | (5/5) | |||
| Laboratory events, , 60k cap | 7,869 | 166,560 | (5/5) |
Table 7: MIMIC-IV test-split results over five seeds. The first two rows share an event source but differ in patient sampling fraction and question cap, so the change in sign points to a scale effect rather than a modality effect without isolating training-data quantity from cohort composition. Positive differences favor MemSafe . Bold marks the better of the two estimators in each row; means higher is better.
| Cap | Patients | Train variants | MemSafe AUROC | LexSet AUROC | AUROC (seed wins) | Macro F1 |
|---|---|---|---|---|---|---|
| 1,000 | 929 | 2,788 | (1/3) | |||
| 2,500 | 2,002 | 6,844 | (3/3) | |||
| 5,000 | 3,113 | 13,724 | (3/3) | |||
| 10,000 | 3,906 | 27,788 | (3/3) | |||
| 20,000 | 3,934 | 29,920 | (3/3) |
Table 8: Locating the sample-efficiency crossover: a five-point learning curve on MIMIC-IV OMR test-split results, holding the event source and patient sampling fraction fixed at and varying only the base-question cap (three seeds per point, fewer than the five seeds used elsewhere); realized patient counts grow with the cap. At sampling, the and caps both exceed the available questions and yield the identical data selection (same selection hash), so the -cap row reproduces the scale-matched OMR setting of Table 7 on a seed subset ( versus unsafe AUROC for MemSafe ; LexSet matches at in both). Positive differences favor MemSafe . Bold marks the better of the two estimators in each row; means higher is better.
| Model | MuSiQue | HotpotQA | 2WikiMultiHopQA |
|---|---|---|---|
| MemSafe | |||
| Encoder + mean pool | |||
| Relevance aggregation | |||
| Concatenated cross-encoder | |||
| LexSet |
Table 9: Seed-level standard deviation of test unsafe AUROC for learned models (five seeds; three for the concatenated cross-encoder). Lower means more stable across seeds ( ); bold marks the lowest value in each column.