Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing

Jiahe Geng; Jinpeng Wang; Kun Yuan

cs.AISep 4, 2026

Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing

Authors: Jiahe Geng, Jinpeng Wang, Kun Yuan

Abstract

Many long-horizon LLM deployments face tight prompt budgets: latency, cost, and context limits make full-context prompting impractical as interaction length grows. The key question is then not raw recall alone, but which memory design gives the best quality--token trade-off in the compact-memory regime. We present RSM-full, an online clustered-memory pipeline designed for a strong quality--token Pareto point. RSM-full combines two design choices: a cosine-gated max-member merge write rule and an atom-aware grouped context packer. On AMA-Bench, our primary compact-memory benchmark, it reaches $83%$ of Full-Context quality at $32%$ of the token cost at a $4$ k budget; under four-seed averaging it beats the closest streaming-clustered baseline (Online K-Means) by $+3.5$ -- $6.0$ ,pp ( $p{<}.001$ ) across the whole ${\sim}2.6$ k-- ${\sim}5$ k regime. Three-seed ablations show most of this gain comes from the merge rule ( $+5.7$ ,pp over Online K-Means and matched- $τ$ DP-means) and the grouped packer ( $+5.0$ ,pp over flat concatenation). The pattern reproduces on RealMem, an independent long-horizon persona-memory benchmark: RSM-full improves on Budget-RAG ( $+0.69$ ,pp, $p{=}.006$ ), is on par with BM25-RAG (paired $Δ{=}{+}0.27$ ,pp, $p{=}.47$ ; we do not claim BM25 equivalence in the equivalence-test sense), and significantly outperforms Streaming-Proto ( $+2.97$ ,pp) and the closest reproduced 2025 agentic-memory baseline A-MEM ( $+1.65$ ,pp, $p{<}.001$ ). Across benchmarks the message is consistent: under tight budgets, compact-memory performance is driven mainly by how streaming memories are merged and how retrieved content is assembled. Overall, RSM-full is most useful when answeroughly $2k$ -- $5k$ prompt tokens, where itdefines a strong compact-memory Pareto point; higher-token baselines remain stronger outside this regime.

Figures & tables

Method	Avg (per-ep)	Recall	CausInf	StaUpd	StaAbs	TotalTok ♡	Avg/kTok
Full-Context	0.373	0.430	0.369	0.380	0.249	12,519	$0.030$
BM25-RAG	0.357	0.405	0.366	0.363	0.234	22,648	$0.016$
Full-RAG	0.355	0.415	0.373	0.340	0.234	20,642	$0.017$
RSM-full-int8 (ours, $k{=}8$ ) ∗	0.315	0.373	0.337	0.295	0.196	4,003	$\mathbf{0.079}$
RSM-full (ours, $k{=}8$ )	0.311	0.361	0.342	0.308	0.169	4,001	$\mathbf{0.078}$
Streaming-Proto	0.302	0.337	0.347	0.280	0.203	3,982	$0.076$

Table 1: AMA-Bench end-to-end comparison. Methods above the midline require more than

12

k tokens.

Figure 1: Quality–token trade-off on AMA-Bench. The shaded region marks the compact-memory regime (

{\leq}5

k tokens).

	Multi-seed mean (per-QA)		Mean gain vs. RSM-full ref. (percentage points)
Method	bt= 2048/4096/8192	Seeds	$\sim$ 2.6k	$\sim$ 4k	$\sim$ 5k
RSM-full (ref.)	$.279/.316/.360$	4/4/4	ref	ref	ref
Oja ( $k{=}4$ )	$.232/.248/.265$	4/4/3	$\mathbf{+4.64}$	$\mathbf{+6.78}$	$\mathbf{+9.46}$
Online K-Means	$.242/.281/.299$	4/4/3	$\mathbf{+3.65}$	$\mathbf{+3.50}$	$\mathbf{+6.04}$

Table 2: AMA-Bench multi-seed comparison against the closest vector-index baselines.

	$\lvert\mathbf{v}_{1,m}^{\top}\mathbf{q}\rvert$ retrieval	Centroid cosine retrieval	Retrieval contrast
RSM (cosine gate, no $K$ cap)	A: $\mathbf{0.3142}$	D: $0.3109$	A $-$ D $=+0.33$ (NS, $p{=}.40$ )
Online K-Means ( $K_{\max}{=}16$ )	B: $0.2568$	C: $0.2652$	B $-$ C $=-0.84^{*}$ ( $p{=}.015$ )
DP-means ( $\lambda{=}\sqrt{0.3}{\approx}0.5477$ , $K_{\max}{=}16$ )	E: $0.2597$	F: $0.2628$	E $-$ F $=-0.30$ (NS, $p{=}.38$ )
Clustering contrasts at $v_{1}$ retrieval endpoint :
RSM vs K-Means	A $-$ B $=\mathbf{+5.74^{***}}$ CI $[+4.65,+6.84]$ , $p{<}.0001$
RSM vs DP-means	A $-$ E $=\mathbf{+5.45^{***}}$ CI $[+4.37,+6.53]$ , $p{<}.0001$

Table 3:

2{\times}3

decomposition of RSM, Online K-Means, and matched-

\tau

DP-means on BGE-normalized AMA-Bench.

Seed	Atom-aware ( $k{=}8$ )	Flat ( $k{=}8$ )	$\Delta$ (pp)	Note
seed 43	$0.3141$ ( $784/2496$ )	$0.2600$ ( $649/2496$ )	$\mathbf{+5.41}$	Single-seed bootstrap CI $[+3.53,+7.29]$ , $p{<}.0001$
seed 44	$0.3189$ ( $796/2496$ )	$0.2612$ ( $652/2496$ )	$\mathbf{+5.77}$	Matched-protocol replication
seed 45	$0.3097$ ( $773/2496$ )	$0.2708$ ( $676/2496$ )	$\mathbf{+3.89}$	Matched-protocol replication
3-seed mean	$0.3142$	$0.2640$	$\mathbf{+5.02\pm 1.00}$	$t$ -CI $[+2.54,+7.50]$ ; $p{<}.0001$ per seed
Atom-aware numbers are the 3-seed RSM-full results reported in Table 2 .

Table 4: Three-seed atom-aware packer ablation on the lean AMA-Bench pipeline.

Method	Seed 43	Seed 44	Seed 45	Pool.	$p$
RSM-full ( $v_{1}$ )	$0.3141$	$0.3189$	$0.3097$	$\mathbf{0.3142}$	—
K-Means (centroid)	$0.2700$	$0.2580$	$0.2676$	$0.2652$	—
$\Delta$ (RSM $-$ KM, pp)	$+4.40$	$+6.08$	$+4.21$	$\mathbf{+4.90}$	$\mathbf{<.0001}$
$95\%$ CI	$[+2.56,+6.29]$	$[+4.17,+7.93]$	$[+2.32,+6.09]$	$[+3.79,+5.98]$
Domain breakdown (pooled; RSM $-$ K-Means)
OPENWORLD-QA ( $n{=}1080$ )				$\mathbf{+10.27}$	$\mathbf{<.0001}$

Table 5: K-Means + atom-aware packer ablation on AMA-Bench.

Method	3-seed mean Acc	TotalTok ♡	$\Delta$ vs RSM-full (pp)	$p$ -value
RSM-full ( $k{=}8$ , ours)	$\mathbf{0.4684}$	$3{,}705$	—	—
BM25-RAG	$0.4657$	$3{,}773$	$+0.27$	$0.47$ NS
Budget-RAG	$0.4615$	$3{,}737$	$\mathbf{+0.69^{\ast\ast}}$	$\mathbf{0.006}$
Full-RAG	$0.4569$	$3{,}513$	$\mathbf{+1.15^{\ast\ast}}$	$\mathbf{0.002}$
A-MEM †	$0.4519$	$3{,}753{+}316$	$\mathbf{+1.65^{\ast\ast\ast}}$	$\mathbf{<.001}$
RSM-full-int8	$0.4506$	$3{,}513$	$\mathbf{+1.78^{\ast\ast\ast}}$	$\mathbf{<.001}$

Table 6: RealMem results on the full 10-persona set.

\Delta

denotes RSM-full minus baseline.

Method / slice	Causal ( $n{=}101$ )	State ( $n{=}100$ )	Goal ( $n{=}100$ , new)	Value ( $n{=}100$ , new)
RSM-full ( $v_{1}$ , $k{=}8$ )	$\mathbf{0.396}$ / $3446$	$0.240$ / $3913$	$\mathbf{0.210}$ / $398$	$0.310$ / $4042$
RSM-cent-retr ( $v_{1}$ -free)	$0.327$ / $2319^{*}$	$0.250$ / $3128$	$0.190$ / $314$	$0.370$ / $3894^{\dagger}$
Full-RAG	$0.347$ / $2319$	$0.180$ / $3128^{*}$	$0.180$ / $314$	$0.370$ / $3894^{\dagger}$
Budget-RAG	$0.327$ / $3536^{*}$	$0.250$ / $3701$	$0.150$ / $2765^{\dagger}$	$0.310$ / $3894$
Streaming-Proto	$0.327$ / $2319^{*}$	$0.260$ / $3128$	$0.150$ / $314^{\dagger}$	$0.350$ / $3894^{\dagger}$
A-MEM ‡	$0.149$ / $3783$	$0.110$ / $3879$	$0.050$ / $3435$	$0.270$ / $3144$

Table 7: LoCoMo-Plus grouped Cognitive across all four relation families.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Atoms	$\alpha$ (Basis%)	AvgSize	$\Delta$	Control	Judge
High- $\alpha$ regime ( $\alpha{\geq}50\%$ ): clustered-memory advantage activates
ToolBench-catbal ( $N{=}2{,}000$ )	227	$56.8\%$	8.8	$+0.25$ pp	RSM-centroid	GPT-4o
AMA-Bench (208 ep)	2.3/ep	$62\%^{\dagger}$	10.2	$+4.3$ pp	RSM-centroid	GPT-5.4
Intermediate regime (partially untested)
ToolBench-hard ( $N{=}500$ )	282	$29.8\%$	1.8	$+1.8$ pp (int8)	RSM-centroid	GPT-4o
Low- $\alpha$ regime ( $\alpha{\leq}5\%$ ): RSM-full $\approx$ centroid, graceful degradation

Appendix

Table 8: Atom-membership diagnostics across benchmarks.

Name	Rank ( $k$ )	Retrieval	Context builder	Storage/atom
RSM-full	2–8	$\|\mathbf{v}_{1}^{\top}\mathbf{q}\|$	atom-aware	$(k{+}1)d$ floats
RSM-full-int8	2–8	$\|\mathbf{v}_{1}^{\top}\mathbf{q}\|$	atom-aware	$(k{+}1)d$ int8
RSM-centroid	0	$\cos(\mathbf{q},\mu_{m})$	basic (flat)	$d$ floats

Appendix

Table 9: RSM pipeline variants.

Protocol	RSM-full ( $v_{1}$ , $k{=}8$ )	RSM-cent-retr	Budget-RAG	Streaming- Proto	Full-RAG
flat	$0.175$ / $3100$	$0.150$ / $3106$	$\mathbf{0.225}$ / $3875$	$0.150$ / $3260$	$0.100$ / $3189$
grouped	$\mathbf{0.200}$ / $1914$	$0.100$ / $2650$	$\mathbf{0.225}$ / $3852$	$0.200$ / $2565$	$0.175$ / $2752$
$\Delta$ (grouped $-$ flat)	$+0.025$ / $-1187$	$-0.050$ / $-456$	$+0.000$ / $-23$	$+0.050$ / $-696$	$+0.075$ / $-437$

Appendix

Table 10: Flat-vs-grouped matched-qid control on LoCoMo-Plus (bal40,

n{=}40

Method	Mean score (correct/ $n$ )	$\Delta$ vs RSM-full	$95\%$ CI	$p$
RSM-full ( $k{=}8$ , ambient cosine, ref)	$0.3185$ ( $795/2{,}496$ )	(ref)	—	—
Algorithm E ( $k{=}8$ , subspace-projected)	$0.3109$ ( $776/2{,}496$ )	$-0.76$	$[-2.12,+0.60]$	$0.27$

Appendix

Table 11: Algorithm E vs. RSM-full on the full 208-episode AMA-Bench.

Section	Stored object	Retrieval rule	Write gate	Charged storage
§ F (LLaMA h.s.)	Full subspace $V_{m}$ $+$ centroid ( $k{\in}\{2,4\}$ )	Proj. $\|\mathbf{v}_{1}^{\top}q\|$ (Eq. 2 )	Cosine $\tau{=}0.65$	$(dk{+}d)\!\times\!4$ B (f32); int8 optional
Appendix B.4 (Latency)	Full subspace $V_{m}$ (col-major)	Proj. precomputed $V_{m}^{\top}q$	N/A (isolated profiling)	$dk\!\times\!4$ B
§ F (MMLU seq.)	Full subspace $V_{m}$ $+$ centroid ( $k{=}2$ )	Proj. $\|\mathbf{v}_{1}^{\top}q\|$ (Eq. 2 )	Cosine $\tau{=}0.16$	$(2d{+}d)\!\times\!4$ B (f32); int8 optional
§ 4.1 (AMA main table)	Full subspace $V_{m}$ $+$ centroid ( $k{=}8$ )	Proj. $\|\mathbf{v}_{1}^{\top}q\|$	Cosine $\tau{=}0.85$	$(dk{+}d)\!\times\!4$ B (f32); int8 optional
§ 4.1 (LoCoMo main table)	Full subspace $V_{m}$ $+$ centroid ( $k{=}8$ )	Proj. $\|\mathbf{v}_{1}^{\top}q\|$ (pure dense)	Cosine $\tau{=}0.85$	$(dk{+}d)\!\times\!4$ B (f32); int8 optional

Appendix

Table 12: RSM operating points by experimental section.

$M$	Full-RAG (ms)	RSM-full-Naive (ms)	RSM-full-Opt (ms)	Speedup (Naive/Opt)
10	0.030	3.390	0.176	$19{\times}$
20	0.033	1.213	0.340	$4{\times}$
50	0.085	3.354	0.367	$9{\times}$
100	0.060	10.997	9.889	$1.1{\times}$
200	4.481	14.775	18.699	—

Appendix

Table 13: Median query latency (ms) as a function of memory size

M

Symbol	Description
$H\in\mathbb{R}^{n\times d}$	Trajectory matrix ( $n$ time steps, hidden dimension $d$ )
$V_{m}\in\mathbb{R}^{d\times k}$	Subspace basis matrix of atom $m$ (orthonormal columns)
$P_{V_{m}}=V_{m}V_{m}^{\top}$	Orthogonal projection onto span $(V_{m})$
$c_{m}\in\mathbb{R}^{k}$	Coefficient summary vector of atom $m$
$e_{m}=V_{m}c_{m}$	Reconstruction vector (legacy evaluation proxy only)
$\mathcal{G}(k,d)$	Grassmann manifold of $k$ -planes in $\mathbb{R}^{d}$

Appendix

Table 14: Key notation used throughout the paper.

	bt=2k		bt=4k		bt=8k
Method	4o	5.4	4o	5.4	4o	5.4
RSM-full-int8	$\mathbf{.250}$	$\mathbf{.291}$	$\mathbf{.342}$	$\mathbf{.315}$	$.276$	$.322$
RSM-full	$.241$	$.288$	$.331$	$.311$	$\mathbf{.285}$	$\mathbf{.323}$
Streaming-Proto	$.099$	$.125$	$.229$	$.287$	$.144$	$.174$
RSM-centroid	$.125$	$.157$	$.227$	$.268$	$.161$	$.187$
Budget-RAG	$.084$	$.111$	$.207$	$.259$	$.133$	$.168$

Appendix

Table 15: AMA-Bench compact-memory ranking under the official GPT-4o judge and a GPT-5.4 re-judge.

Setting	RSM-full ( $\|\mathbf{v}_{1}^{\top}q\|$ )	Budget-RAG	Stream-Proto	KMeans ‡	Storage	Compression
Full-RAG (ref.)	$1.000$	—	—	—	25,600 KB	$1\times$
k=4, float32	$\mathbf{0.999}$	$0.996$	$0.999$	$1.000$	2,624 KB	$9.8\times$
k=4, int8	$\mathbf{0.999}$	$0.948$	$0.996$	$1.000$	1,064 KB	$24.1\times$
k=2, float32	$\mathbf{0.999}$	$0.982$	$0.999$	$1.000$	1,616 KB	$15.8\times$
k=2, int8	$\mathbf{0.999}$	$0.890$	$0.974$	$0.996$	812 KB	$\mathbf{31.5\times}$

Appendix

Table 16: ToolBench real hidden-state comparison (

N{=}2000

d{=}4096

; mean over 5 seeds). ‡ Offline.

Budget/Compression	RSM-centroid	Budget-RAG	Stream-Proto	KMeans ‡
$72.2$ ( $22.2\times$ )	$\mathbf{1.000}$	$0.970$	$0.995$	$0.998$
$54.8$ ( $29.2\times$ )	$\mathbf{1.000}$	$0.902$	$0.998$	$0.998$
$34.0$ ( $47.1\times$ )	$\mathbf{0.998}$	$0.816$	$0.946$	$0.996$
$27.4$ ( $58.4\times$ )	$\mathbf{0.949}$	$0.759$	$0.881$	$0.993$
$24.0$ ( $66.7\times$ )	$\mathbf{0.938}$	$0.706$	$0.845$	$0.994$

Appendix

Table 17: Multi-budget comparison on ToolBench LLaMA-3.1-8B hidden states.

Method	Storage	Recall@1 ( $\uparrow$ )	Coverage ( $\uparrow$ )
Full-RAG	1657.5 KB	$0.849$	10/10
Budget-RAG ( $k{=}2$ bytes)	174.0 KB	$0.273$	$1/10$
Streaming-Proto ( $k{=}2$ bytes)	174.0 KB	$0.806$	10/10
RAG-KMeans ‡ ( $k{=}2$ bytes)	174.0 KB	$0.831$	10/10
RSM-full ( $\tau{=}0.16$ , $k{=}2$ )	174.0 KB	$\mathbf{0.824\pm 0.000}$ (5 seeds)	10/10
RSM-full ( $k{=}4$ )	277.5 KB	$0.824$	10/10

Appendix

Table 18: Subject Recall@1 under a sequential topic stream on MMLU. Mean

\pm

std is over 5 seeds for RSM-full at

k{=}2

. ‡ Offline.

	$\sigma=0.0$	$\sigma=1.0$	$\sigma=2.0$	$\sigma=3.0$
Mean judge quality (0/0.5/1 scale):
RSM-full ( $k{=}8$ )	$0.360$	$0.365$	$\mathbf{0.377}$	$0.373$
Streaming-Proto	$0.355$	$0.341$	$0.348$	$0.350$
Budget-RAG	$0.355$	$0.344$	$0.347$	$0.345$
Retention (% of own $\sigma{=}0$ baseline):
RSM-full	$100.0$	$101.4$	$\mathbf{104.7}$	$103.6$

Appendix

Table 19: AMA-Bench noise-robustness results.

Figure 2: Quality retention under chunk-side Gaussian noise (

n{=}50

episodes, 2 seeds; error bars

\pm 1

se).

Method	bt=2048	bt=4096	bt=8192	Seeds
RSM-full	$0.279\pm\mathbf{0.005}$	$0.314\pm\mathbf{0.005}$	$0.358\pm\mathbf{0.004}$	$\{43,44,45\}$
Online K-Means	$0.242\pm 0.003$	$0.281\pm 0.034$	$0.299\pm 0.044$	$\{42,43,44,45\}$
Oja ( $k{=}4$ )	$0.232\pm 0.004$	$0.248\pm 0.033$	$0.265\pm 0.042$	$\{42,43,44,45\}$
Relative std (coefficient of variation, %):
RSM-full	$\mathbf{1.8}$	$\mathbf{1.5}$	$\mathbf{1.1}$
Online K-Means	$1.2$	$11.9$	$14.8$

Appendix

Table 20: Stream-order invariance on AMA-Bench. Lower values indicate greater stability.

	$K{=}0$	$K{=}5$	$K{=}15$
Mean judge quality:
RSM-full ( $k{=}8$ )	$\mathbf{0.404}$	$\mathbf{0.382}$	$\mathbf{0.359}$
Streaming-Proto	$0.358$	$0.331$	$0.307$
Budget-RAG	$0.392$	$0.368$	$0.345$
Retention (% of own $K{=}0$ baseline):
RSM-full	$100.0$	$94.5$	$88.9$

Appendix

Table 21: AMA-Bench max-cosine adversarial injection results in the moderate-adversarial regime.

Figure 3: Quality retention under query-aware max-cosine adversarial chunk injection (

n{=}50

episodes, 2 seeds).

$N$	$n_{\text{ep}}$	RSM-full atoms	RSM-cent.-merge atoms	Atom $\Delta\%$ (paired $t$ , $p$ )	Quality $\Delta$ (pp)
$16$ k	$75$	$73.16$	$79.63$	$\mathbf{+8.84\%}$ ( $t{=}21.03$ , $p{<}10^{-32}$ )	$-0.11$ ( $p{=}.88$ NS)
$32$ k	$75$	$84.31$	$93.21$	$\mathbf{+10.56\%}$ ( $t{=}23.14$ , $p{<}10^{-35}$ )	$+0.56$ ( $p{=}.53$ NS)
Pooled	$150$	—	—	—	$+0.22$ ( $p{=}.70$ NS)

Appendix

Table 22: RSM-cent-merge ablation (write-side: max-member fallback removed) on AMA-Long Track-A.

Figure 4: Atom-count growth on AMA-Long Track-A. Blue: RSM-full. Orange: RSM-cent-merge ablation at

N{=}16

k and

32

k. Gray: raw-chunk linear reference.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 357 papers

MemRefine: LLM-Guided Compression for Long-Term Agent Memory
Jun 11, 2026Minjae Kim, Jinheon Baek, Soyeong Jeong +1Persistent Memory for Language Models LLM Context Compression
Korea University · KAIST
LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
Jul 17, 2026Jing Yu, Yibo Zhao, Jiaming Zhang +1Persistent Memory for Language Models Long-Term Conversational Memory
School of Data Science and Engineering, East China Normal University
Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents
Mar 20, 2026Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall +2Persistent Memory for Language Models LLM Memory
Memori Labs Inc.

Jun 11, 2026J/K move · Enter open · S save

MemRefine: LLM-Guided Compression for Long-Term Agent Memory

Minjae Kim, Jinheon Baek, Soyeong Jeong, Sung Ju Hwang

Korea University · KAIST

Persistent Memory for Language Models LLM Context Compression Long-Term Conversational Memory Agent Memory Management Memory Management for LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly expected to operate over long-term interactions, where information from past dialogues must be preserved and recalled to support future tasks. However, as interactions accumulate, the memory store grows without bound and fills with redundant entries that inflate storage cost and degrade retrieval by crowding out the most useful evidence. Furthermore, this is especially limiting on resource-constrained platforms with hard memory budgets, motivating us to formulate storage-budgeted memory management, the task of keeping an already constructed memory store within a fixed budget while preserving information useful for future interactions. To this end, we then propose MemRefine, an LLM-guided framework that, since surface similarity poorly reflects factual value, uses similarity only to propose candidate pairs and defers delete, merge, and preserve decisions to an LLM judge based on factual content, iterating until the budget is met. Across multiple memory frameworks and long-term conversation benchmarks, MemRefine consistently meets target budgets while preserving downstream performance and outperforming rule-based baselines under tight budgets.

Abstract

LLM Agent Memory

Explore similar work

MemRefine: LLM-Guided Compression for Long-Term Agent Memory

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents