Not All Memories Are Equal: Hierarchical Collaborative Memory for Validity-Aware Retrieval in LLM Agents
Authors: Yufei Shi, Rujing Yao, Ang Li, Yang Wu, Zhuoren Jiang, Xiaozhong Liu
Organizations: Nanyang Technological University, Singapore · University of Macau, China · Worcester Polytechnic Institute, USA · Zhejiang University, China
In team collaboration scenarios, memory is heterogeneous and continually evolving. Team memories capture collective decisions, protocols, and current consensus, while individual memories preserve member-specific observations, execution traces, and intermediate progress. Existing memory-augmented systems typically retrieve from all stored memories as a flat pool, ranking them by semantic relevance, importance, or recency without modeling hierarchical structure or evolving validity. As a result, they often surface semantically relevant but outdated or conflicting memories, especially individual memories that no longer align with current team consensus, instead of prioritizing currently valid memories. This is particularly problematic when collaborative LLM agents answer user questions, since their responses should be grounded in valid memories. We propose HiCoMER, a framework for hierarchical collaborative memory management and validity-aware retrieval in LLM agents. HiCoMER first maintains the validity of team and individual memories and then retrieves memories that remain valid, rather than retrieving directly from all stored memories. It consists of three components: a Hierarchical Memory Conflict Updater, a Validity-Aware Memory Retriever, and a Memory-Grounded Answer Generator. To evaluate HiCoMER, we construct two new datasets for memory-grounded question answering in collaborative settings. Experiments on both datasets show that HiCoMER consistently outperforms strong baselines by reducing outdated retrieval, preserving current team consensus, and improving downstream QA quality.
Figures & tables
Figure 1: Illustration comparing existing methods and our proposed HiCoMER.
Figure 2: Overall framework of HiCoMER. The framework consists of three modules: Hierarchical Memory Conflict Updater, Validity-Aware Memory Retriever, and Memory-Grounded Answer Generator.
Statistic
ALI
PROTAC
Total
Candidate papers
30
30
60
Retained papers
27
26
53
Valid groups
908
873
1,781
Memory documents
8,917
8,801
17,718
Injected conflicts
317
304
621
Same-time conflicts
189
181
370
Table 1: Detailed statistics.
Method
ORR@5 ( ↓ )
CRR@5 ( ↑ )
Decision F1 ( ↑ )
ROUGE-L ( ↑ )
NDCG@10 ( ↑ )
Standard & Semantic Retrieval
Flat RAG
45.83
51.94
41.67
35.38
42.76
Hybrid RAG
41.92
55.68
44.27
37.14
45.31
Rerank RAG
37.57
61.83
48.88
40.47
51.86
Multi-Agent & Hierarchical Memory
G-Memory
29.74
68.91
53.79
44.82
57.63
Table 2: End-to-end evaluation of the full HiCoMER pipeline on the ALI dataset using Llama-3.1-8B as the backbone for the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while answer-level metrics ROUGE-L and Decision F1 assess how well the pipeline converts maintained and retrieved memories into final memory-grounded responses.
Method
ORR@5 ( ↓ )
CRR@5 ( ↑ )
Decision F1 ( ↑ )
ROUGE-L ( ↑ )
NDCG@10 ( ↑ )
Standard & Semantic Retrieval
Flat RAG
46.57
50.86
39.43
33.57
41.93
Hybrid RAG
43.04
54.88
42.16
35.61
44.36
Rerank RAG
38.76
60.43
46.48
38.92
50.37
Multi-Agent & Hierarchical Memory
G-Memory
31.03
67.12
50.84
42.69
55.87
Table 3: End-to-end evaluation of HiCoMER on the ALI dataset using Qwen2.5-3B-Instruct for both the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 assess maintained-state ranking, while answer-level metrics ROUGE-L and Decision F1 measure the quality of memory-grounded responses under a smaller backbone.
Variant
ORR@5
CRR@5
NDCG@10
ROUGE-L
Decision F1
Full HiCoMER (Llama-3.1-8B)
14.18
84.87
68.74
52.93
64.61
w/o Hierarchical Memory Conflict Updater
30.42
71.63
58.91
46.86
55.94
w/o Validity-Aware Memory Retriever
29.16
70.31
57.12
45.97
54.83
w/o Memory-Grounded Answer Generator
14.18
84.87
68.74
47.62
56.48
Table 4: Ablation study on the ALI dataset. ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while ROUGE-L and Decision F1 assess memory-grounded response quality. Each row shows the effect of removing a single module from the full HiCoMER pipeline.
Method
Memory Precision
Intervention Usefulness
Mem0
3.73
3.68
HiCoMER
4.11
4.04
Table 5: Human evaluation results on simulated system utility, based on 170 evaluation instances. Memory Precision measures the relevance and accuracy of retrieved context, and Intervention Usefulness measures the actionability of the generated suggestions.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Team Memory
Conflicting Individual/ Historical Memory
Query
ALI
Week 4 team decision: discontinue Compound X due to elevated toxicity in the pilot cohort.
Week 4 lab log: prepared Compound X for follow-up efficacy testing; Week 2 protocol note: expand Compound X testing if preliminary efficacy remains stable.
Should the team proceed with the planned Compound X follow-up experiment?
PROTAC
Week 6 platform update: replace the original degradation assay with the N-end-rule-based validation workflow for downstream screening.
Week 6 experiment log: continued using the previous degradation readout; Week 4 design note: prioritize the earlier target-screening route.
Is the current validation result directly usable under the latest platform protocol?
Appendix
Table 6: Illustrative examples.
Model
Conflict F1
Maintenance Accuracy
Rule-based Updater
46.13
53.38
Qwen2.5-3B Zero-shot
57.84
60.57
Qwen2.5-3B with SFT
77.18
81.47
Qwen2.5-3B with SFT and GRPO
83.57
87.31
Llama-3.1-8B Zero-shot
61.68
64.93
Llama-3.1-8B with SFT
81.76
85.36
Appendix
Table 7: Component-level evaluation of the Hierarchical Memory Conflict Updater on held-out structured maintenance instances from ALI. Conflict F1 measures the accuracy of predicting conflict relations, while Maintenance Accuracy measures the correctness of structured memory updates.
Method
ORR@5 ( ↓ )
CRR@5 ( ↑ )
Decision F1 ( ↑ )
ROUGE-L ( ↑ )
NDCG@10 ( ↑ )
Standard & Semantic Retrieval
Flat RAG
48.12
48.73
38.66
33.81
40.68
Hybrid RAG
44.79
52.14
41.08
35.23
43.37
Rerank RAG
40.66
57.58
45.42
38.17
49.09
Multi-Agent & Hierarchical Memory
G-Memory
31.84
64.77
50.31
42.38
54.34
Appendix
Table 8: End-to-end evaluation of the full HiCoMER pipeline on the PROTAC dataset using Llama-3.1-8B as the backbone for the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while answer-level metrics ROUGE-L and Decision F1 assess how well the pipeline converts maintained and retrieved memories into final memory-grounded responses.
Method
ORR@5 ( ↓ )
CRR@5 ( ↑ )
Decision F1 ( ↑ )
ROUGE-L ( ↑ )
NDCG@10 ( ↑ )
Standard & Semantic Retrieval
Flat RAG
48.97
47.63
36.88
31.94
39.47
Hybrid RAG
45.74
50.82
39.42
33.76
42.03
Rerank RAG
41.48
56.07
43.61
36.87
47.58
Multi-Agent & Hierarchical Memory
G-Memory
33.02
63.09
47.86
40.58
52.46
Appendix
Table 9: End-to-end evaluation of HiCoMER on the PROTAC dataset using Qwen2.5-3B-Instruct for both the Hierarchical Memory Conflict Updater and the Memory-Grounded Answer Generator. Retrieval metrics ORR@5, CRR@5, and NDCG@10 assess maintained-state ranking, while answer-level metrics ROUGE-L and Decision F1 measure the quality of memory-grounded responses under a smaller backbone.
Variant
ORR@5
CRR@5
NDCG@10
ROUGE-L
Decision F1
Full HiCoMER (Llama-3.1-8B)
15.37
82.74
66.81
51.43
62.79
w/o Hierarchical Memory Conflict Updater
31.26
70.94
57.63
45.98
54.86
w/o Validity-Aware Memory Retriever
30.11
69.82
56.34
45.21
53.91
w/o Memory-Grounded Answer Generator
15.37
82.74
66.81
46.58
54.97
Appendix
Table 10: Ablation study on the PROTAC dataset. ORR@5, CRR@5, and NDCG@10 measure maintained-state retrieval quality, while ROUGE-L and Decision F1 assess memory-grounded response quality. Each row shows the effect of removing a single module from the full HiCoMER pipeline.
Long-term memory enables LLM agents to reuse past experiences, yet retrieving relevant memories does not necessarily provide sufficient evidence for answering a query. Essential information may remain fragmented across memory records, obscured by abstraction, or disconnected from the relations required for reasoning. We identify this relevance-sufficiency gap and propose QEMem, a framework that reformulates memory utilization as the construction of query-specific evidence sets, shifting the focus from individual retrieval relevance to collective evidential support. QEMem integrates three complementary mechanisms: query-guided discovery retrieves evidence across multiple memory granularities; source-grounded reification organizes it into structured ViewCards with explicit evidential roles and source references; and sufficiency-guided refinement selectively recovers missing information. Operating on existing memory stores without modifying memory formation, QEMem achieves 91.43% LLM-judge accuracy on LoCoMo with Claude Opus 4.5, outperforming the strongest evaluated baseline, MemR3, by 7.6 percentage points and NEMORI by 10.7 points under a shared memory store. Further analyses demonstrate improved evidence coverage and strong performance on questions requiring cross-session evidence integration.
Shixuan Li, Wei Yang, Peiyu Zhang +7
University of Southern California · Georgia Institute of Technology
Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem.
Shuqi Cao, Jingyi He, Fei Tan
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Memory is essential for enabling LLM-based agents to maintain coherent, personalized behavior over long-horizon interactions. However, existing memory systems share a fundamental limitation: they never proactively test their own memory, repairing it only after real queries expose weaknesses. This reactive paradigm means every retrieval failure corresponds to a real interaction in which the cost has already been paid. We propose MemDream, a framework that enables self-probing memory evolution for LLM agents. Our framework periodically enters offline dream cycles where three specialized agents (Dreamer, Analyst, Consolidator) collaboratively probe, diagnose, and repair the memory graph before failures occur. A policy trained via Group Relative Policy Optimization learns which repair operations produce durable retrieval improvements, while a soft decay mechanism provides reversible forgetting driven by the same anticipatory signal. Experiments on LoCoMo and MemoryAgentBench demonstrate that MemDream improves answer F1 by 4.5 points on LoCoMo and achieves a 9.1-point higher overall score on MAB over the strongest reactive-evolution baselines.
Mingfei Lu, Mengjia Wu, Runsong Jia +2
Australian Artificial Intelligence Institute (AAII) University of Technology Sydney