CLAG: Adaptive Memory Organization via Agent-Driven Clustering for Small Language Model Agents

Taeyun Roh; Wonjune Jang; Junha Jung; Jaewoo Kang

cs.CLMar 16, 2026

CLAG: Adaptive Memory Organization via Agent-Driven Clustering for Small Language Model Agents

Authors: Taeyun Roh, Wonjune Jang, Junha Jung, Jaewoo Kang

Organizations: Korea University · Myongji University · AIGEN Sciences

Abstract

Large language model agents heavily rely on external memory to support knowledge reuse and complex reasoning tasks. Yet most memory systems store experiences in a single global retrieval pool which can gradually dilute or corrupt stored knowledge. This problem is especially pronounced for small language models (SLMs), which are highly vulnerable to irrelevant context. We introduce CLAG, a CLustering-based AGentic memory framework where an SLM agent actively organizes memory by clustering. CLAG employs an SLM-driven router to assign incoming memories to semantically coherent clusters and autonomously generates cluster-specific profiles, including topic summaries and descriptive tags, to establish each cluster as a self-contained functional unit. By performing localized evolution within these structured neighborhoods, CLAG effectively reduces cross-topic interference and enhances internal memory density. During retrieval, the framework utilizes a two-stage process that first filters relevant clusters via their profiles, thereby excluding distractors and reducing the search space. Experiments on multiple QA datasets with three SLM backbones show that CLAG consistently improves answer quality and robustness over prior memory systems for agents, remaining lightweight and efficient.

Figures & tables

Figure 1: Conceptual comparison between existing global memory systems and CLAG. (Left) Traditional approaches manage memories in a single global pool, where topic-mixed updates and retrieval lead to high interference and noise accumulation. (Right) CLAG employs agent-driven clustering to organize memories into semantically coherent neighborhoods. By confining evolution and retrieval to these local clusters, our framework significantly reduces cross-topic interference and enhances memory stability.

Figure 2: Overview of the proposed CLAG framework. Left: Agentic Routing. An SLM router assigns each incoming memory note

m_{\text{new}}

to the most relevant cluster using semantic metadata, and updates the corresponding cluster profile

\mathcal{P}

. Middle: Localized Evolution. An evolution agent performs consolidation (e.g., linking, rewriting, strengthening) within the routed cluster to maintain topic-consistent neighborhoods and reduce cross-topic interference. Right: Two-Stage Retrieval. Given a query, CLAG first filters clusters using profile-based selection (Stage 1), then retrieves fine-grained memories only inside the selected clusters (Stage 2), reducing the effective search space and suppressing retrieval noise.

Model	Method	LoCoMo		HotpotQA		BioASQ
Model	Method	F1	BLEU-1	F1	BLEU-1	F1	BLEU-1
Qwen3-0.6B	RAG	12.9	10.39	11.75	11.17	2.4	1.71
	A-mem	14.29	11.8	12.04	10.65	3.61	2.83
	GAM	16.05	13.24	7.81	6.69	3.40	3.37
	MemoryOS	4.30	3.24	9.02	7.34	3.12	1.29
	CLAG (Ours)	20.99	17.88	15.50	14.33	22.01	17.23

Table 1: Overall results on LoCoMo, HotpotQA, and BioASQ across three backbone models. Best results are marked in bold , 2nd-best results are marked in underline .

Method	Retrieval (ms)	End-to-End (ms)
RAG	17.80	289.60
A-mem	18.54	408.68
GAM	8303.41	17934.32
MemoryOS	220.79	968.75
CLAG (Ours)	142.43	514.14

Table 2: Latency comparison on Qwen3-0.6B backbone. All measurements were conducted on a single NVIDIA A100 GPU. Values represent the average time in milliseconds (ms).

Model	Method	Multi Hop		Temporal		Open Domain		Single-Hop		Adversarial
Model	Method	F1	BLEU-1	F1	BLEU-1	F1	BLEU-1	F1	BLEU-1	F1	BLEU-1
Qwen3-0.6B	RAG	6.11	6.04	8.34	7.41	9.07	7.92	8.09	5.52	30.35	24.90
	A-mem	6.80	6.34	2.34	2.37	9.97	7.58	8.16	6.00	40.11	33.85
	GAM	8.46	7.08	6.50	6.78	10.92	8.45	9.41	6.92	41.25	34.60
	MemoryOS	2.29	1.99	7.19	5.60	8.93	7.68	3.58	2.53	3.85	2.72
	CLAG	10.08	8.80	11.53	9.39	8.68	6.57	14.10	11.07	50.34	45.00

Table 3: Detailed experimental results on the LoCoMo dataset across five categories. Results are reported in F1 and BLEU-1 (%). Best results are marked in bold , 2nd-best results are marked in underline .

Dataset	Method	E-Prec	E-Recall	E-F1	R@5	R@10	nDCG@10
LoCoMo	RAG	0.66	4.83	1.12	3.12	4.83	3.26
	A-mem	0.68	5.50	1.17	3.86	5.50	3.62
	CLAG (Ours)	1.20	9.74	2.07	7.18	9.74	6.89
HotpotQA	RAG	9.75	14.71	11.44	10.71	14.71	23.71
	A-mem	8.05	12.63	9.60	8.38	12.63	22.56
	CLAG (Ours)	9.86	14.20	11.17	9.94	14.20	23.98

Table 4: Retrieval performance analysis across three datasets (Backbone: Qwen3-0.6B). Metrics include Evidence Precision (E-Prec), Recall (E-Recall), F1 (E-F1), Recall at k (R@k), and nDCG@10. Best results are marked in bold , 2nd-best results are marked in underline .

Clustering Strategy	F1	BLEU-1
Cosine-based Clustering	14.78	12.53
K-Means Clustering	15.64	13.36
CLAG (Ours)	22.01	17.23

Table 5: Impact of different clustering methods on BioASQ (Qwen3-0.6B).

Method	LoCoMo		HotpotQA		BioASQ
Method	F1	BLEU-1	F1	BLEU-1	F1	BLEU-1
RAG	19.06	15.93	23.66	21.33	22.21	20.19
A-mem	21.44	18.35	22.73	20.94	18.59	15.86
GAM	19.55	17.26	22.09	19.30	22.96	19.42
MemoryOS	19.60	15.89	24.64	22.37	28.83	22.48
CLAG (Ours)	21.50	17.94	25.40	23.53	25.40	23.54

Table 6: Results on the stronger backbone Qwen3-8B. CLAG remains competitive as model scale increases, achieving the best results on LoCoMo and HotpotQA and the best BLEU-1 on BioASQ. Best results are marked in bold , and second-best results are marked in underline .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	F1	BLEU-1
CLAG (full)	22.01	17.23
w/ Global Evolution	19.55	17.25
w/ Global Retrieval	20.25	16.56

Appendix

Table 7: Ablation on BioASQ (Qwen3-0.6B). Replacing localized evolution with global evolution drops F1 by 2.46, and replacing two-stage retrieval with global retrieval drops F1 by 1.76.

Hyperparameter	Meaning	Value
$n$	Init buffer size (min memories before initial KMeans)	100
$n_{\text{init}}$	Initial clusters for KMeans initialization	3
$\tau_{\text{split}}$	Cluster split threshold (split if size $>\tau_{\text{split}}$ )	300
$k_{\text{route}}$	Routing candidate clusters shown to SLM	3
$\tau$	New-cluster threshold (min cosine sim to accept a cluster)	0.1
$k_{\text{local}}$	Intra-cluster neighbors for evolution/linking	5

Appendix

Table 8: Hyperparameters used in our CLAG implementation. Unless stated otherwise, all experiments use the values shown above.

Initial Clusters ( $n_{\text{init}}$ )	F1	BLEU-1
3 (Default)	20.99	17.88
5	22.71	19.01
10	21.71	18.50

Appendix

Table 9: Sensitivity analysis of the initial cluster count (

n_{\text{init}}

) on LoCoMo (Qwen3-0.6B). The system maintains high performance across all metrics, regardless of initialization settings.

Threshold ( $\tau$ )	F1	BLEU-1	Avg. # Clusters
0.10 (Default)	20.99	17.88	4.7
0.15	22.41	18.93	6.9
0.20	21.68	18.29	15.6
0.25	21.99	18.35	26.5
0.30	21.94	18.73	47.5

Appendix

Table 10: Sensitivity analysis of the new-cluster threshold (

\tau

) on LoCoMo (Qwen3-0.6B). While the number of clusters increases significantly with higher thresholds, retrieval performance remains robust.

Method	Prediction	Result
CLAG (Ours)	“Harry Potter and Game of Thrones”	Correct ✓
RAG	“Not mentioned in the conversation.”	Wrong ✗
A-mem	“Tim’s favorite two mystery novels are not mentioned.”	Wrong ✗
GAM	“Not mentioned in the conversation.”	Wrong ✗
MemoryOS	“Tim writes about both fantasy and mystery novels.”	Wrong ✗

Appendix

Table 11: Comparison of generated responses For the query “Which two mystery novels does Tim particularly enjoy writing about?” , only CLAG correctly identifies the two titles mentioned in the dialogue.

Cluster ID	Profile	Count	Status
0	Speaker discusses how books create new worlds	119	Selected ✓
1	Impact of basketball on community growth	231	Pruned ✗
2	Experience of meeting teammates after a trip	330	Pruned ✗

Appendix

Table 12: Semantic clustering and pruning for the case study sample. CLAG selects the literature-related cluster and prunes away unrelated clusters, reducing the search space from 680 to 119 notes before fine-grained retrieval.

Category	Description
Entity confusion	Routed to a cluster anchored to the wrong entity.
Temporal-slot ambiguity	Routed to profiles lacking temporal cues.
Stale/drift	Baseline retrieves more temporally relevant evidence.
Compositional/Multi-hop miss	Query requires composition, but too few clusters are selected ( $n\leq 1$ ).
Location ambiguity	Routed to profiles lacking spatial cues.
Numeric-slot ambiguity	Routed to profiles lacking numeric evidence.

Appendix

Table 13: Failure taxonomy for CLAG .

Pattern	Count	%
Entity confusion	110	46.2
Temporal-slot ambiguity	46	19.3
Stale / drift	34	14.3
Compositional / Multi-hop miss	17	7.1
Residual ambiguity	13	5.5
Location ambiguity	9	3.8

Appendix

Table 14: Failure distribution (

N=238

Model	Method	LoCoMo		HotpotQA		BioASQ
Model	Method	BERT-F1	METEOR	BERT-F1	METEOR	BERT-F1	METEOR
Qwen3-0.6B	RAG	85.07	12.78	82.11	7.07	78.89	2.68
	A-mem	85.04	12.85	82.20	6.13	79.04	3.03
	GAM	84.91	14.39	82.57	6.33	77.05	2.66
	MemoryOS	84.15	2.68	84.39	5.02	80.65	1.11
	CLAG (Ours)	85.45	18.57	82.47	10.10	83.56	17.75

Appendix

Table 15: Detailed experimental results on LoCoMo, HotpotQA, and BioASQ: BERT-F1 and METEOR scores. Best results are marked in bold , 2nd-best results are marked in underline .

Dataset	Method	Evidence Quality		Ranking		Chars
Dataset	Method	E-Prec	E-F1	R@5	nDCG@10	Chars
LoCoMo	GAM	1.97	3.01	7.50	6.93	17,198.04
	RAG	0.66	1.12	3.12	3.26	1,389.43
	A-mem	0.68	1.17	3.86	3.62	1,483.91
	MemoryOS	3.24	2.87	2.72	3.35	1,373.48
	CLAG	1.20	2.07	7.18	9.74	1,465.65

Appendix

Table 16: Retrieval-quality comparison with GAM and MemoryOS. Best results are marked in bold , and second-best results are marked in underline .

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 191 papers

Lightweight LLM Agent Memory with Small Language Models
Apr 9, 2026Jiaquan Zhang, Chaoning Zhang, Shuxu Chen +9Memory Management for LLM Agents Persistent Memory for Language Models
University of Electronic Science and Technology of China · Kyung Hee University · City University of Hong Kong +1
MemFlow: Intent-Driven Memory Orchestration for Small Language Model Agents
May 5, 2026Jiayi Chen, Yingcong Li, Guiling WangLLM Agent Workflow Optimization Persistent Memory for Language Models
New Jersey Institute of Technology, Newark, USA
H-Mem: A Novel Memory Mechanism for Evolving and Retrieving Agent Memory via a Hybrid Structure
May 15, 2026Jiawei Yu, Yixiang Fang, Xilin Liu +1Agent Memory LLM Agent Memory
The Chinese University of Hong Kong, Shenzhen · Huawei Cloud Computing Technologies CO., LTD.

Apr 9, 2026J/K move · Enter open · S save

Lightweight LLM Agent Memory with Small Language Models

Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Zhenzhen Huang+8

University of Electronic Science and Technology of China · Kyung Hee University · City University of Hong Kong+1

Memory Management for LLM Agents Persistent Memory for Language Models Agent Memory Management Long-Term Conversational Memory Memory Consolidation for LLM Agents

Open paper AskPDF arXiv

Although LLM agents can leverage tools for complex tasks, they still need memory to maintain cross-turn consistency and accumulate reusable information in long-horizon interactions. However, retrieval-based external memory systems incur low online overhead but suffer from unstable accuracy due to limited query construction and candidate filtering. In contrast, many systems use repeated large-model calls for online memory operations, improving accuracy but accumulating latency over long interactions. We propose LightMem, a lightweight memory system for better agent memory driven by Small Language Models (SLMs). LightMem modularizes memory retrieval, writing, and long-term consolidation, and separates online processing from offline consolidation to enable efficient memory invocation under bounded compute. We organize memory into short-term memory (STM) for immediate conversational context, mid-term memory (MTM) for reusable interaction summaries, and long-term memory (LTM) for consolidated knowledge, and uses user identifiers to support independent retrieval and incremental maintenance in multi-user settings. Online, LightMem operates under a fixed retrieval budget and selects memories via a two-stage procedure: vector-based coarse retrieval followed by semantic consistency re-ranking. Offline, it abstracts reusable interaction evidence and incrementally integrates it into LTM. Experiments show consistent gains across model scales, with an average F1 improvement of about 2.5 over A-MEM on LoCoMo, while achieving higher efficiency and low median latency (83 ms for retrieval and 581 ms end-to-end).

Abstract

Agent Memory

Explore similar work

Lightweight LLM Agent Memory with Small Language Models

MemFlow: Intent-Driven Memory Orchestration for Small Language Model Agents

H-Mem: A Novel Memory Mechanism for Evolving and Retrieving Agent Memory via a Hybrid Structure