Beyond Marginal Influence: Interaction-Aware Memory Selection for LLM Agents

BHVSP Subrahmanyam; Neeraj Choudhary

cs.LGOct 16, 2026

Beyond Marginal Influence: Interaction-Aware Memory Selection for LLM Agents

Authors: BHVSP Subrahmanyam, Neeraj Choudhary

Organizations: Department of Computer Science and Engineering, Mahindra University, India

Abstract

LLM-agent memory selection often uses retrieval rankings based on record relevance or individual influence. Such rankings cannot identify sufficient subsets under redundant or jointly necessary evidence. We study per-decision memory sufficiency: selecting a small subset that preserves full-memory behavior. To separate search error from model error, we introduce JointCoreBench, a benchmark with nine interaction families, six workflow domains, natural-language decisions, and exact sufficient-set oracles, and JointCore, a relevance cost-ordered group-deletion baseline that tests blocks before individual records. On 6,912 exact conditions, cost-ordered JointCore recovers a minimum-cost sufficient set in 100.0% of cases, reduces subset evaluations from 30.00 to 11.13 (62.9%), and achieves 87.8% active-memory savings. Across four 12B-70B model families and 432 tasks per model, its unweighted mean is 10.28 versus 16.00 model queries and 50.6% versus 43.1% minimum-cost recovery relative to cost aware singleton deletion. Both methods preserve full-memory behavior under the deterministic primary interface. Tests with physical deletion, UNKNOWN masks, PENDING sentinels, counterfactual values, and alternative labels identify interface dependent behavior. JointCore is an efficient benchmark baseline; the study does not establish deployment-ready memory compression.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic JointCore trace on eight ordered records. The search removes $d_{1}{:}d_{4}$ as a block, refines $e_{1}{:}e_{4}$ , and returns $\{e_{1},e_{2}\}$ .

Method	Exact suff.	Min.-cost	Savings	Subset evals.
Random- $B$	0.162	0.001	0.766	0.00
Semantic top- $B$	0.944	0.222	0.775	0.00
Independent leave-one-out	0.667	0.667	0.900	30.00
Cost-aware singleton	1.000	1.000	0.878	30.00
Legacy JointCore	1.000	0.933	0.876	10.96
Cost-ordered JointCore	1.000	1.000	0.878	11.13

Table 1: Exact-stage aggregate over 6,912 conditions. Exact sufficiency and minimum-cost recovery are proportions. Savings are relative to full registered payload cost. Higher is better except for subset evaluations, which exclude the shared full-memory reference.

Checkpoint	Control	Behavior	Exact	Min.-cost	Savings	Queries
Qwen2.5–14B	0.988	1.000/1.000	0.586/0.544	0.465/0.382	0.881/0.860	9.97/16.00
Gemma–4–12B	0.875	1.000/1.000	0.845/0.808	0.572/0.493	0.837/0.807	10.70/16.00
Mistral–3.1–24B NF4	0.847	1.000/1.000	0.660/0.644	0.403/0.296	0.874/0.835	9.56/16.00
Llama–3.3–70B NF4 †	0.993	1.000/1.000	0.706/0.713	0.583/0.551	0.856/0.836	10.87/16.00

Table 2: Primary-interface evaluation over 432 tasks per model. Control is exact full-memory accuracy. Remaining cells are cost-ordered JointCore / cost-aware singleton: behavior is agreement with the model’s full-memory decision, exact is logical sufficiency, min.-cost is exact minimum-cost recovery, and savings use registered payload cost. Higher is better except for queries.

\dagger

Exploratory model result.

Figure 2: Primary-interface comparison over 432 tasks per model. (a) Mean selection queries per task, where lower is better; JointCore whiskers are 95% task-bootstrap intervals and singleton always uses 16 queries. (b) Exact minimum-cost recovery, where higher is better.

\dagger

Exploratory model result.

Figure 3: Model-stage results by interaction family. Panels (a) and (b) show cost-ordered JointCore minus cost-aware singleton in percentage points for logical recovery and active-payload savings; positive values favor JointCore. Panel (c) shows JointCore’s absolute mean query count, with singleton’s 16-query cost as the reference. Values are unweighted means across the four displayed models and include the exploratory Llama results.

Figure 4: Memory and label robustness on the fixed 108-task subset for each model. Each cell is cost-ordered JointCore / cost-aware singleton task success (%); color is their percentage-point difference. Primary uses semantic YES/NO decisions and UNKNOWN masks. $\dagger$ Exploratory model result.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Sufficiency condition	Candidate minimal sets
Singleton	The designated record $r_{1}$ is retained.	$\{r_{1}\}$
Redundant OR	At least one record in $R$ is retained.	Any $\{r_{i}\}$
Synergistic AND	Every record in $R$ is retained.	$R$
XOR	Every XOR flag in $R$ is retained; their fixed values contain exactly one ON flag.	$R$
Threshold	At least $k$ records from $R$ are retained.	Any size- $k$ subset of $R$
Multiple minimum sets	Every record in $R_{A}$ or every record in $R_{B}$ is retained.	$R_{A}$ or $R_{B}$

Appendix

Table 3: Exact interaction families used by JointCoreBench. The condition column specifies when a retained set

S

is logically sufficient.

Method	Retained memory	Cost	Subset evals.	Exact / minimum
Full memory	all 16 records	81	0	yes / no
Semantic top- $4$	$\{m_{0},m_{9},m_{10},m_{11}\}$	20	0	yes / no
Independent leave-one-out	$\emptyset$	0	16	no / no
Cost-aware singleton	$\{m_{10}\}$	3	16	yes / yes
Cost-ordered JointCore	$\{m_{10}\}$	3	9	yes / yes
Exact oracle	$\{m_{0}\}$	3	0	yes / yes

Appendix

Table 4: Exact worked example from the frozen benchmark. Static retrieval uses no selection evaluations; the exact oracle is a reference, not a selector.

Checkpoint	$\Delta$ queries	95% CI	adjusted $p$
Qwen2.5–14B	-6.03	[-6.34, -5.73]	$<.001$
Gemma–4–12B	-5.30	[-5.81, -4.77]	$<.001$
Mistral–3.1–24B NF4	-6.44	[-6.84, -6.00]	$<.001$
Llama–3.3–70B NF4 †	-5.13	[-5.49, -4.74]	$<.001$

Appendix

Table 5: Displayed-cohort paired model-query comparisons. Differences are JointCore minus cost-aware singleton, so negative values favor JointCore. Adjusted

p

uses the registered twelve-test Holm family.

Memory Management for LLM Agents

See all 58 papers

MemRefine: LLM-Guided Compression for Long-Term Agent Memory
Jun 11, 2026Minjae Kim, Jinheon Baek, Soyeong Jeong +1Persistent Memory for Language Models LLM Context Compression
Korea University · KAIST
LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
Jul 17, 2026Jing Yu, Yibo Zhao, Jiaming Zhang +1Persistent Memory for Language Models Long-Term Conversational Memory
School of Data Science and Engineering, East China Normal University
Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
Jul 15, 2026Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11Persistent Memory for Language Models Memory Management for LLM Agents
University of California Los Angeles · University of Washington · Northwestern University

Jun 11, 2026J/K move · Enter open · S save

MemRefine: LLM-Guided Compression for Long-Term Agent Memory

Minjae Kim, Jinheon Baek, Soyeong Jeong, Sung Ju Hwang

Korea University · KAIST

Persistent Memory for Language Models LLM Context Compression Long-Term Conversational Memory Agent Memory Management Memory Management for LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly expected to operate over long-term interactions, where information from past dialogues must be preserved and recalled to support future tasks. However, as interactions accumulate, the memory store grows without bound and fills with redundant entries that inflate storage cost and degrade retrieval by crowding out the most useful evidence. Furthermore, this is especially limiting on resource-constrained platforms with hard memory budgets, motivating us to formulate storage-budgeted memory management, the task of keeping an already constructed memory store within a fixed budget while preserving information useful for future interactions. To this end, we then propose MemRefine, an LLM-guided framework that, since surface similarity poorly reflects factual value, uses similarity only to propose candidate pairs and defers delete, merge, and preserve decisions to an LLM judge based on factual content, iterating until the budget is met. Across multiple memory frameworks and long-term conversation benchmarks, MemRefine consistently meets target budgets while preserving downstream performance and outperforming rule-based baselines under tight budgets.

Abstract

Memory Management for LLM Agents

Explore similar work

MemRefine: LLM-Guided Compression for Long-Term Agent Memory

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents