Not All Memories Are Equal: Hierarchical Collaborative Memory for Validity-Aware Retrieval in LLM Agents

Yufei Shi; Rujing Yao; Ang Li; Yang Wu; Zhuoren Jiang; Xiaozhong Liu

cs.CLSep 9, 2026

Not All Memories Are Equal: Hierarchical Collaborative Memory for Validity-Aware Retrieval in LLM Agents

Authors: Yufei Shi, Rujing Yao, Ang Li, Yang Wu, Zhuoren Jiang, Xiaozhong Liu

Organizations: Nanyang Technological University, Singapore · University of Macau, China · Worcester Polytechnic Institute, USA · Zhejiang University, China

Abstract

In team collaboration scenarios, memory is heterogeneous and continually evolving. Team memories capture collective decisions, protocols, and current consensus, while individual memories preserve member-specific observations, execution traces, and intermediate progress. Existing memory-augmented systems typically retrieve from all stored memories as a flat pool, ranking them by semantic relevance, importance, or recency without modeling hierarchical structure or evolving validity. As a result, they often surface semantically relevant but outdated or conflicting memories, especially individual memories that no longer align with current team consensus, instead of prioritizing currently valid memories. This is particularly problematic when collaborative LLM agents answer user questions, since their responses should be grounded in valid memories. We propose HiCoMER, a framework for hierarchical collaborative memory management and validity-aware retrieval in LLM agents. HiCoMER first maintains the validity of team and individual memories and then retrieves memories that remain valid, rather than retrieving directly from all stored memories. It consists of three components: a Hierarchical Memory Conflict Updater, a Validity-Aware Memory Retriever, and a Memory-Grounded Answer Generator. To evaluate HiCoMER, we construct two new datasets for memory-grounded question answering in collaborative settings. Experiments on both datasets show that HiCoMER consistently outperforms strong baselines by reducing outdated retrieval, preserving current team consensus, and improving downstream QA quality.

Figures & tables

Figure 1: Illustration comparing existing methods and our proposed HiCoMER.

Figure 2: Overall framework of HiCoMER. The framework consists of three modules: Hierarchical Memory Conflict Updater, Validity-Aware Memory Retriever, and Memory-Grounded Answer Generator.

Statistic	ALI	PROTAC	Total
Candidate papers	30	30	60
Retained papers	27	26	53
Valid groups	908	873	1,781
Memory documents	8,917	8,801	17,718
Injected conflicts	317	304	621
Same-time conflicts	189	181	370

Table 1: Detailed statistics.

Standard & Semantic Retrieval
Method	ORR@5 ( $\downarrow$ )	CRR@5 ( $\uparrow$ )	Decision F1 ( $\uparrow$ )	ROUGE-L ( $\uparrow$ )	NDCG@10 ( $\uparrow$ )
Flat RAG	45.83	51.94	41.67	35.38	42.76
Hybrid RAG	41.92	55.68	44.27	37.14	45.31
Rerank RAG	37.57	61.83	48.88	40.47	51.86
Multi-Agent & Hierarchical Memory
G-Memory	29.74	68.91	53.79	44.82	57.63

Table 2: End-to-end evaluation of the full HiCoMER pipeline on the ALI dataset using Llama-3.1-8B as the backbone for the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while answer-level metrics ROUGE-L and Decision F1 assess how well the pipeline converts maintained and retrieved memories into final memory-grounded responses.

Standard & Semantic Retrieval
Method	ORR@5 ( $\downarrow$ )	CRR@5 ( $\uparrow$ )	Decision F1 ( $\uparrow$ )	ROUGE-L ( $\uparrow$ )	NDCG@10 ( $\uparrow$ )
Flat RAG	46.57	50.86	39.43	33.57	41.93
Hybrid RAG	43.04	54.88	42.16	35.61	44.36
Rerank RAG	38.76	60.43	46.48	38.92	50.37
Multi-Agent & Hierarchical Memory
G-Memory	31.03	67.12	50.84	42.69	55.87

Table 3: End-to-end evaluation of HiCoMER on the ALI dataset using Qwen2.5-3B-Instruct for both the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 assess maintained-state ranking, while answer-level metrics ROUGE-L and Decision F1 measure the quality of memory-grounded responses under a smaller backbone.

Variant	ORR@5	CRR@5	NDCG@10	ROUGE-L	Decision F1
Full HiCoMER (Llama-3.1-8B)	14.18	84.87	68.74	52.93	64.61
w/o Hierarchical Memory Conflict Updater	30.42	71.63	58.91	46.86	55.94
w/o Validity-Aware Memory Retriever	29.16	70.31	57.12	45.97	54.83
w/o Memory-Grounded Answer Generator	14.18	84.87	68.74	47.62	56.48

Table 4: Ablation study on the ALI dataset. ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while ROUGE-L and Decision F1 assess memory-grounded response quality. Each row shows the effect of removing a single module from the full HiCoMER pipeline.

Method	Memory Precision	Intervention Usefulness
Mem0	3.73	3.68
HiCoMER	4.11	4.04

Table 5: Human evaluation results on simulated system utility, based on 170 evaluation instances. Memory Precision measures the relevance and accuracy of retrieved context, and Intervention Usefulness measures the actionability of the generated suggestions.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Team Memory	Conflicting Individual/ Historical Memory	Query
ALI	Week 4 team decision: discontinue Compound X due to elevated toxicity in the pilot cohort.	Week 4 lab log: prepared Compound X for follow-up efficacy testing; Week 2 protocol note: expand Compound X testing if preliminary efficacy remains stable.	Should the team proceed with the planned Compound X follow-up experiment?
PROTAC	Week 6 platform update: replace the original degradation assay with the N-end-rule-based validation workflow for downstream screening.	Week 6 experiment log: continued using the previous degradation readout; Week 4 design note: prioritize the earlier target-screening route.	Is the current validation result directly usable under the latest platform protocol?

Appendix

Table 6: Illustrative examples.

Model	Conflict F1	Maintenance Accuracy
Rule-based Updater	46.13	53.38
Qwen2.5-3B Zero-shot	57.84	60.57
Qwen2.5-3B with SFT	77.18	81.47
Qwen2.5-3B with SFT and GRPO	83.57	87.31
Llama-3.1-8B Zero-shot	61.68	64.93
Llama-3.1-8B with SFT	81.76	85.36

Appendix

Table 7: Component-level evaluation of the Hierarchical Memory Conflict Updater on held-out structured maintenance instances from ALI. Conflict F1 measures the accuracy of predicting conflict relations, while Maintenance Accuracy measures the correctness of structured memory updates.

Standard & Semantic Retrieval
Method	ORR@5 ( $\downarrow$ )	CRR@5 ( $\uparrow$ )	Decision F1 ( $\uparrow$ )	ROUGE-L ( $\uparrow$ )	NDCG@10 ( $\uparrow$ )
Flat RAG	48.12	48.73	38.66	33.81	40.68
Hybrid RAG	44.79	52.14	41.08	35.23	43.37
Rerank RAG	40.66	57.58	45.42	38.17	49.09
Multi-Agent & Hierarchical Memory
G-Memory	31.84	64.77	50.31	42.38	54.34

Appendix

Table 8: End-to-end evaluation of the full HiCoMER pipeline on the PROTAC dataset using Llama-3.1-8B as the backbone for the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while answer-level metrics ROUGE-L and Decision F1 assess how well the pipeline converts maintained and retrieved memories into final memory-grounded responses.

Standard & Semantic Retrieval
Method	ORR@5 ( $\downarrow$ )	CRR@5 ( $\uparrow$ )	Decision F1 ( $\uparrow$ )	ROUGE-L ( $\uparrow$ )	NDCG@10 ( $\uparrow$ )
Flat RAG	48.97	47.63	36.88	31.94	39.47
Hybrid RAG	45.74	50.82	39.42	33.76	42.03
Rerank RAG	41.48	56.07	43.61	36.87	47.58
Multi-Agent & Hierarchical Memory
G-Memory	33.02	63.09	47.86	40.58	52.46

Appendix

Table 9: End-to-end evaluation of HiCoMER on the PROTAC dataset using Qwen2.5-3B-Instruct for both the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 assess maintained-state ranking, while answer-level metrics ROUGE-L and Decision F1 measure the quality of memory-grounded responses under a smaller backbone.

Variant	ORR@5	CRR@5	NDCG@10	ROUGE-L	Decision F1
Full HiCoMER (Llama-3.1-8B)	15.37	82.74	66.81	51.43	62.79
w/o Hierarchical Memory Conflict Updater	31.26	70.94	57.63	45.98	54.86
w/o Validity-Aware Memory Retriever	30.11	69.82	56.34	45.21	53.91
w/o Memory-Grounded Answer Generator	15.37	82.74	66.81	46.58	54.97

Appendix

Table 10: Ablation study on the PROTAC dataset. ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while ROUGE-L and Decision F1 assess memory-grounded response quality. Each row shows the effect of removing a single module from the full HiCoMER pipeline.

Memory-Augmented Language Model Agents

Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.

See all 444 papers

From Relevant Memories to Sufficient Evidence: Rethinking Memory Utilization in LLM Agents
Oct 15, 2026Shixuan Li, Wei Yang, Peiyu Zhang +7LLM Agent Memory LLM Agents
University of Southern California · Georgia Institute of Technology
HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents
Apr 20, 2026Shuqi Cao, Jingyi He, Fei TanPersistent Memory for Language Models Hierarchical Memory
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Remember Before You're Asked: MemDream for Self-Probing Memory Evolution
Sep 28, 2026Mingfei Lu, Mengjia Wu, Runsong Jia +2LLM Agent Memory Graph-Based Agent Memory
Australian Artificial Intelligence Institute (AAII) University of Technology Sydney

Oct 15, 2026J/K move · Enter open · S save

From Relevant Memories to Sufficient Evidence: Rethinking Memory Utilization in LLM Agents

Shixuan Li, Wei Yang, Peiyu Zhang, Anzhe Cheng+6

University of Southern California · Georgia Institute of Technology

LLM Agent Memory LLM Agents Retrieval-Augmented Reasoning

Open paper AskPDF arXiv

Long-term memory enables LLM agents to reuse past experiences, yet retrieving relevant memories does not necessarily provide sufficient evidence for answering a query. Essential information may remain fragmented across memory records, obscured by abstraction, or disconnected from the relations required for reasoning. We identify this relevance-sufficiency gap and propose QEMem, a framework that reformulates memory utilization as the construction of query-specific evidence sets, shifting the focus from individual retrieval relevance to collective evidential support. QEMem integrates three complementary mechanisms: query-guided discovery retrieves evidence across multiple memory granularities; source-grounded reification organizes it into structured ViewCards with explicit evidential roles and source references; and sufficiency-guided refinement selectively recovers missing information. Operating on existing memory stores without modifying memory formation, QEMem achieves 91.43% LLM-judge accuracy on LoCoMo with Claude Opus 4.5, outperforming the strongest evaluated baseline, MemR

^3

, by 7.6 percentage points and NEMORI by 10.7 points under a shared memory store. Further analyses demonstrate improved evidence coverage and strong performance on questions requiring cross-session evidence integration.

Abstract

Memory-Augmented Language Model Agents

Explore similar work

From Relevant Memories to Sufficient Evidence: Rethinking Memory Utilization in LLM Agents

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

Remember Before You're Asked: MemDream for Self-Probing Memory Evolution