cs.AIOct 5, 2026
SaveThe Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory
Organizations: University of Arkansas at Little Rock
Abstract
Long-term-memory agents can retrieve relevant information that is inadmissible for the current request because it belongs to another principal, violates policy, or reflects an incompatible lifecycle state. Recall and final-answer accuracy do not reveal this: a route can appear safe by missing required evidence, while a correct answer may follow inadmissible prompt exposure. We introduce a retrieval-admissibility verification framework that assigns each memory-query pair one of three statuses (admissible, inadmissible, or unresolved), compares routes at matched required-evidence recall with bounds for unresolved cases, and tracks memory IDs through prompt exposure while linking exposure to target-level disclosure. We evaluate its stages on separate, non-pooled populations. A post-hoc top-20 reanalysis of frozen rankings from two public long-term-memory benchmarks, RHELM and MemOps, covers 3,767 queries. All released anchors lie within trusted query namespaces; with within-namespace scores unchanged, off-namespace filtering cannot lower their ranks. Top-20 anchor recall increases from 0.432 to 0.533, 80% recall feasibility from 0.237 to 0.311, and exact similarity evaluations decrease by 98.3%. In a frozen 72-case development diagnostic, a released-metadata reference preserves required evidence, whereas neither text-only verifier detects violations under the 1% required-anchor false-denial limit. Across 1,523 paired benchmark-native cases, namespace routing is associated with judged-accuracy gains of 0.053-0.068 across three readers; recall also changes, so this comparison is observational. In 16 controlled exposure scenarios, only one of four reader-specific 95% confidence intervals excludes zero for relevant-inadmissible literal disclosure (+0.156, 95% CI [0.031, 0.312]). Results motivate separate verification of candidate support, admissibility, prompt exposure, and answer disclosure.
Figures & tables
Figure 1: Audited retrieval-admissibility cases. Wrong-scope, superseded, and explicitly forgotten records are excluded, whereas superseded medication states are retained for a history query. The first row is from RHELM and the remainder from MemOps; text is abridged, but verdicts are not.
Figure 2: Agent-memory retrieval-admissibility verification framework. Query metadata constrain candidate support, semantic retrieval ranks IDs, and policy and lifecycle verification precede agent-prompt assembly. The lower trace follows one memory ID from storage through retrieval and exposure, followed by target-level disclosure scoring; disclosure is observable output, not internal causal use. Each study reports its population and reader-specific estimates separately.
| Question | Population and comparison | Primary outcomes |
|---|---|---|
| RQ1: Support | RHELM/MemOps queries; global, namespace, random, pre/post-filter, routed, and released-field arms; metadata stress | Recall, feasibility, bounds, coverage, constrained loss, typed exposure, work |
| RQ2: Verification | 96 public-development queries with fixed top-20 pools plus controlled scenarios; fields versus text decisions | ROC-AUC, violation precision/recall, anchor false denial, consistency, overflip |
| RQ3: Enforcement | paired benchmark-native reader cases plus controlled pairs per reader; no reader pooling | Answer correctness/quality, non-answer action, protected/stale disclosure, cell effects, selectivity |
Table 1: Experimental map. Each row identifies the population, compared systems, and outcomes; rows are analyzed separately and never pooled.
Figure 3: Three separate evaluations yield different stagewise verdicts. (a) Namespace beats random same-size support; late filtering retains global work. (b) The released-field reference preserves evidence, but the text-only gates do not. (c) Only DeepSeek V4 Pro’s relevant-inadmissible contrast excludes zero; ∗ marks the Claude Opus 5 replication. Populations ( queries, 72 verifier cases, and 192 pairs per reader) are not pooled.
| Method | Visible information | Recall | Feasible | Similarity evals. | ||
|---|---|---|---|---|---|---|
| Global dense | text | .432 | .237 | [.187,.195] | .798 | 90,122 |
| Global recency+dense | text + released order | .452 | .256 | [.198,.206] | .784 | 90,122 |
| Namespace dense | text + trusted namespace | .533 | .311 | [.118,.126] | .713 | 1,551 |
| Namespace global | +.101 | +.074 | — | .085 | 58.1 fewer | |
Table 2: Frozen-route admissibility audit and downstream reader association. (a) Post-hoc admissibility, top-20 results; rankings and settings are unchanged, coverage is .992; similarity evaluations count query–candidate score computations, not wall-clock latency. (b) Equal-source reader averages and paired namespace-minus-global contrasts. (c) Typed matched-prefix exposure; axes can overlap. Readers remain separate; † marks a gated replication.
| Support intervention | Recall | Feasible | Similarity evals. | |
|---|---|---|---|---|
| Global dense | .432 | .237 | .798 | 90,122 |
| Random same-size | .061 | .024 | .978 | 1,540 |
| Namespace pre-filter | .533 | .311 | .713 | 1,551 |
| Global post-filter ( ) | .531 | .311 | .713 | 90,122 |
| Anchor-preserving same-size oracle | .840 | .747 | .390 | 1,551 |
Table 3: Support ablations and attribution controls. Frozen top-20 estimates at . Random and anchor-preserving rows are ten-seed means; the latter uses released anchors. Similarity evaluations count query–candidate score computations, not wall-clock latency.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Support | Known non-usable | Coverage | Lower | Upper |
|---|---|---|---|---|
| Global dense | 0.575 | 0.538 | 0.308 | 0.770 |
| Namespace dense | 0.348 | 0.355 | 0.134 | 0.778 |
Table 4: Historical non-usable diagnostics within feasible matched-recall prefixes. These source-macro quantities do not use the infeasibility penalty and are not pure wrong-scope measures. Namespace support lowers the resolved rate and lower bound but also lowers coverage; the upper bound remains high.
| Axis family | Route | Recall | Feasible | Any violation | ||
|---|---|---|---|---|---|---|
| Scope + policy + lifecycle | Global | .432 | .237 | [.187,.195] | .798 | .528 |
| Scope + policy + lifecycle | Namespace | .533 | .311 | [.118,.126] | .713 | .396 |
| Scope + lifecycle | Global | .432 | .237 | [.102,.111] | .787 | .356 |
| Scope + lifecycle | Namespace | .533 | .311 | [.017,.025] | .693 | .134 |
Table 5: The namespace result does not depend on the low-agreement policy axis. Equal-source macro estimates at target recall .8. For the policy-omitted loss difference, the paired 10,000-sample namespace-group bootstrap gives [ ].
| Route | Known risk | Coverage | Any violation | |
|---|---|---|---|---|
| Global dense | .1893 | .9918 | [.1873,.1955] | .5285 |
| Namespace dense | .1122 | .9913 | [.1105,.1191] | .3721 |
Table 6: The matched-prefix risk reduction persists on an identical feasible-query set. Both routes reach target recall on the same 609 queries. Equal-source macro estimates use the full scope+policy+lifecycle target; the paired namespace-minus-global upper-bound difference is [95% CI ] from 10,000 namespace-group bootstrap replicates.
| Axis | Exact agreement | Uncertain | Evaluable | Reviewer–source | |
|---|---|---|---|---|---|
| Relevance | 0.8937 (185/207) | 0 | 207 | 0.8135 | .797–.802 |
| Scope | 0.9952 (206/207) | 0 | 207 | 0.9808 | .995–1.000 |
| Lifecycle state | 0.8599 (178/207) | 23 | 184 | 0.6703 | .841–.947 |
| Prohibited evidence | 0.8841 (183/207) | 0 | 207 | 0.0829 | .729–.826 |
| Legacy usable composite (derived) | 0.9710 (201/207) | 0 | 207 | 0.9405 | .836–.855 |
Table 7: Scope and relevance labels are reproducible; lifecycle and prohibited status remain the harder axes. Raw independent, unadjudicated agreement on the blinded human-audit sample. Uncertainty counts labels across both reviewers; nominal Krippendorff’s excludes pairs containing an uncertain label. The legacy usable composite is derived rather than independently elicited and excludes lifecycle. Exact agreement with the deterministic released-field construction is reported as a consistency check, not external gold validation.
Figure 4: Relevant records require query-conditioned eligibility checks. Four abridged expert-audited examples underlying the admissibility taxonomy: wrong principal, superseded current state, explicit forgetting, and a historical query that requires superseded states. Packet IDs, memory IDs, released labels, and verdicts are preserved in the example manifest; wording is shortened for legibility. These cases illustrate coverage, not population prevalence.
| Work | Invalidity target | Primary unit / stages | Matched-recall bounds | Paired exposure estimand |
|---|---|---|---|---|
| GateMem [ Ren et al., 2026 ] | Principal access, updates, forgetting | Episode/checkpoint utility and leakage | Not primary | Not primary |
| MemOps [ Hao et al., 2026 ] | Operation target, scope, state transition | Structured operation trace and probes | Not primary | Not primary |
| STALE [ Chao et al., 2026 ] | Implicit invalidation of earlier state | State-resolution and answer probes | Not primary | Not primary |
| A-TMA [ Shi et al., 2026 ] | Current, historical, and transition state | Bank, retrieval, and answer diagnostics | Not primary | Not primary |
| MemConflict [ Tao et al., 2026 ] | Temporal, factual, and contextual conflicts | Retrieval, ranking, and final answers | Not primary | Not primary |
| MemGate [ Zhang et al., 2026b ] | Query-conditioned contextual appropriateness | Learned admission gate, threat, and utility | Not primary | Not primary |
Table 8: This work audits eligibility and downstream stages at the record level. Relationship to nearby evaluation frameworks. “Not primary” describes each paper’s reported estimands, not the capabilities of its system. Our emphasis is one record-level audit across eligibility axes and stages.
Figure 5: More filtering is not monotonically better. Changes from namespace dense in percentage points; right is better. Circles show utility and open squares recall-constrained loss reduction for (a) released metadata on queries and (b) fixed-candidate gates on 72 development cases. Whiskers are pointwise 95% bootstrap intervals; verifiers remain separate. Full corruption brackets appear in Table 14 .
Figure 6: Semantic verifiers recognize eligibility changes but do not apply them selectively. (a) Feasibility and sign-reversed recall-constrained-loss differences versus namespace dense; labels give recall differences. Whiskers are pointwise 95% intervals from 10,000 namespace-group bootstrap replicates stratified by source and query intent over 72 analysis cases. (b) Controlled focal consistency versus stable-control overflip; labels report stable-admissible false-denial rate. The benchmark-native and controlled populations are separate and never pooled.
Figure 7: No tested threshold satisfies the declared anchor-denial constraint while removing violations. Thresholds span 0 to 1 in .05 steps; panels (a)–(b) show the practical region. Open markers are the calibration-selected .95 thresholds. Candidate precision, violation recall, and anchor denial are analysis-candidate micro averages; route outcomes in (c) use the prespecified equal-stratum macro. GPT-5.6 Sol meets the 1% false-denial constraint only by retaining every candidate and detecting no violations; Gemini 3.6 Flash has no tested point below the line. The gray cross is the retain-all namespace route.
| Model | ROC-AUC | Violation precision | Violation recall | Anchor false deny |
|---|---|---|---|---|
| GPT-5.6 Sol | ||||
| Gemini 3.6 Flash |
Table 9: Text-only verification does not yield a low-denial operating point. Candidate-level diagnostics for the benchmark-native text-only verification experiment. Values use the fixed development-selected operating points. Higher ROC-AUC, violation precision, and violation recall are better; lower required-anchor false-denial rate is better.
Figure 8: Trusted namespace support improves retrieval and downstream utility. Across queries, namespace raises (a) recall and (b) feasibility at every budget. (c) All three readers gain accuracy on the frozen cases; whiskers are paired 95% intervals. GPT-5.6 Luna is sequential. Claude Opus 5 was not executed as a reader on this population; Claude Haiku 4.5 served only as the shared blinded answer judge.
Figure 9: Namespace support outperforms arbitrary candidate-count matching. (a) The ten-seed random partition preserves source-level namespace-label counts but breaks provenance assignment; points show recall and feasible rate. (b) Global top- is namespace-filtered and truncated to 20; dashed lines are namespace pre-filtering. (c) Namespace-minus-global penalized risk as the explicit infeasibility cost varies; negative is better. Scope-excluded risk retains only released policy/lifecycle violations and unresolved labels. Across all 87 groups, leave-one-group-out recall deltas are +.097 to +.108; the exact seven-group RHELM sign-flip -value is .0156.
Figure 10: Zero-call diagnostics expose uncertainty and retrieval headroom. (a) Established composite evaluator judgments are hidden after retrieval, so routes, anchors, recall, and feasibility remain fixed. Solid lines report known-label coverage and dashed lines the resulting partial-identification interval width; the callout gives lower and upper risk at 20% hiding. (b) The gold-preserving arm retains every released required anchor and samples non-anchors to exactly the namespace support size. Circles are recall, squares are feasible-query rate, and whiskers show one seed standard deviation where applicable. * The oracle reads released gold and is an attribution diagnostic, not a deployable router.
| Reader | answer accuracy | answer quality | non-answer rate |
|---|---|---|---|
| DeepSeek V4 Pro | +0.053 [+0.020, +0.084] | +0.043 [+0.023, +0.062] | -0.048 [-0.068, -0.031] |
| Gemini 3.6 Flash | +0.068 [+0.047, +0.090] | +0.050 [+0.036, +0.065] | -0.041 [-0.057, -0.025] |
| GPT-5.6 Luna † | +0.066 [+0.039, +0.096] | +0.048 [+0.031, +0.066] | -0.038 [-0.059, -0.015] |
Table 10: Namespace support improves answer utility for every frozen reader. Benchmark-native same-population route-to-reader effects. Entries are namespace dense minus global dense with pointwise 95% paired namespace-group bootstrap intervals. Readers are reported separately and share one blinded Claude Haiku 4.5 judge. † GPT-5.6 Luna is a sequential robustness replication.
| Reader | Source | answer accuracy | non-answer rate |
|---|---|---|---|
| DeepSeek V4 Pro | RHELM | +.021 [ .037,+.075] | .044 [ .075, .016] |
| MemOps | +.084 [+.052,+.116] | .053 [ .077, .030] | |
| Gemini 3.6 Flash | RHELM | +.059 [+.032,+.094] | .042 [ .068, .022] |
| MemOps | +.076 [+.045,+.106] | .039 [ .061, .018] | |
| GPT-5.6 Luna † | RHELM | +.046 [+.004,+.094] | .059 [ .096, .016] |
| MemOps | +.086 [+.053,+.121] | .017 [ .034, .001] |
Table 11: The direction is stable across sources, with wider RHELM uncertainty. Source-specific namespace-dense minus global-dense effects in the benchmark-native reader analysis. Intervals use 10,000 paired namespace-group bootstrap replicates. The equal-source macro result in the main table is prespecified, but these rows expose source heterogeneity: DeepSeek V4 Pro accuracy on RHELM is the only accuracy interval that crosses zero. Non-answer means , not gold-action-aware over-refusal. RHELM intervals resample only seven evaluation namespace groups and should be read with that cluster count in view.
| Reader | answer accuracy | answer quality | non-answer rate |
|---|---|---|---|
| DeepSeek V4 Pro | +.062 [+.033,+.092] | +.050 [+.033,+.068] | .050 [ .069, .032] |
| Gemini 3.6 Flash | +.070 [+.048,+.094] | +.052 [+.037,+.067] | .040 [ .057, .024] |
| GPT-5.6 Luna † | +.072 [+.045,+.101] | +.053 [+.036,+.071] | .032 [ .049, .014] |
Table 12: Case weighting preserves the namespace benefit. Post-hoc case-weighted sensitivity for namespace dense minus global dense. Each evaluable case receives equal weight across RHELM and MemOps; 95% intervals resample namespace groups within source. The prespecified equal-source macro remains primary, and readers remain separate.
| Arm | Selected setting |
|---|---|
| Global BM25 | |
| Global dense | Parameter-free |
| Global BM25+dense RRF | |
| Global recency dense | |
| Namespace dense | Parameter-free |
| Query-agnostic current-only | Remove stale and superseded records for every query |
Table 13: All route settings were selected on development data only. Frozen development-selected settings; all arms use .
| Channel | Reference | Last dominant | First failure |
|---|---|---|---|
| Namespace false deny | Global dense | .10 | .20 |
| Namespace missing | Global dense | .10 | .20 |
| Namespace source swap | Global dense | .10 | .20 |
| Namespace false allow | Global dense | .50 | Not observed |
| Policy false deny | Clean namespace | .00 | .02 |
| Policy false allow | Clean namespace | .30 | .40 |
Table 14: False-deny corruption is the earliest failure mode. Observed metadata-corruption weak-dominance brackets over ten nested seeds. Namespace channels use global dense as the reference; policy, lifecycle, and intent channels use clean namespace dense. Rates are tested grid points, not interpolated thresholds. “Not observed” means no failure through the maximum tested rate .50.
| Arm | Recall | Feasible | Penalized non-usable upper risk | Candidates |
|---|---|---|---|---|
| Global BM25 | 0.448 | 0.284 | 0.913 | 90122 |
| Global dense | 0.717 | 0.539 | 0.873 | 90122 |
| Global BM25+dense RRF | 0.691 | 0.503 | 0.883 | 90122 |
| Global recency dense | 0.728 | 0.550 | 0.864 | 90122 |
| Namespace dense | 0.866 | 0.762 | 0.837 | 1551 |
| Query-agnostic current-only | 0.824 | 0.672 | 0.845 | 1549 |
Table 15: Namespace dense gives the strongest retrieval trade-off among the frozen routes. All nine frozen v1 arms at on the source-macro public evaluation; main-text Table 2 a truncates the same frozen rankings to top-20. The legacy penalized non-usable upper-risk field assigns 1.0 to infeasible or unresolved query outcomes. Candidates scored is a local diagnostic, not a production latency estimate.
Figure 11: Namespace support, rather than the tested routing refinements, separates the frozen retrieval regimes. (a) Recall versus mean candidates scored for all nine arms. (b) Namespace-dense minus global-dense effects under the historical non-usable loss. (c) Conditional non-usable bounds and label coverage. (d) Threshold, cluster, and historical released-field v1 differences versus namespace dense. This figure preserves the original route semantics and cannot be read as the corrected v2 policy/lifecycle attribution.
Figure 12: Observed exposure predicts downstream disclosure but does not identify a causal effect. (a) Exposed-minus-unexposed answer disclosure over discordant checkpoints. (b) Predictive value of exposure on held-out rows. (c) GPT-4o mini G1-minus-G0 leakage, bounded utility, and GateMem bounded over-refusal. Readers are not pooled; route contrasts are non-causal and leakage and utility use different populations. Exact reader snapshots are gpt-4o-mini-2024-07-18 and gpt-4o-2024-08-06 .
Figure 13: Controlled exposure can become disclosure. (a) Paired effects by evidence type; (b) selectivity gaps and relevant-inadmissible effects. Whiskers are pointwise 95% scenario-bootstrap intervals. Readers are separate; Claude Opus 5 is an unpooled replication, and DeepSeek V4 Pro’s result is constructed, not a safety guarantee.
| Support | Evidence recall | Wrong-scope leakage | Non-usable fraction |
|---|---|---|---|
| Global dense | 1.0 | 0.025 | 0.1189 |
| Namespace dense | 1.0 | 0.000 | 0.0824 |
Table 16: The scope intervention behaves as intended in a controlled smoke. Candidate-pool results on 16 curated evaluation packets.
| Layer | Status | Relation to earlier evidence |
|---|---|---|
| Retrieval setting selection | Prespecified development procedure | Selected before evaluation with historical loss; no test retuning. |
| Frozen v1 evaluation | Held-out evaluation of the frozen v1 route family | Preserves the original non-usable status semantics and all nine arms. |
| Admissibility v2 score | Post-hoc correction; frozen routes | Replaces by in scoring only; no reranking or retuning. |
| Released-governance v2 | Post-hoc untuned attribution; frozen routes | Applies policy independently of intent and reports lifecycle separately. |
| Two-reader analysis | Prespecified paired execution | Cases, routes, prompts, and two readers frozen before outcome scoring. |
| GPT-5.6 Luna reader | Gated sequential replication | Repeats the three primary routes after the two-reader continuation criterion. |
Table 17: The paper separates prespecified, corrected, replicated, and post-hoc analyses. Provenance of the main analysis layers. “Frozen route” means that no ranking, threshold, or candidate order was changed for the analysis.