MemBubble: Benchmarking How Agent Memory Shapes LLM Response Diversity

Jiajun Tan; Fei Sun; Xinyu Ma; Shixuan Zhang; Du Su; Qi Cao; Daiting Shi; Huawei Shen

cs.CLOct 18, 2026

MemBubble: Benchmarking How Agent Memory Shapes LLM Response Diversity

Authors: Jiajun Tan, Fei Sun, Xinyu Ma, Shixuan Zhang, Du Su, Qi Cao, Daiting Shi, Huawei Shen,

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Baidu Inc.

Abstract

Memory serves as the core component of LLM-based agents, enabling personalized responses via extracting and using user preferences from past interactions. However, its use may reduce response diversity, thereby narrowing the range of information available to users. Existing evaluations rely largely on synthetic interactions and cover few memory systems, leaving the effects of memory on response diversity insufficiently evaluated in real-world scenarios. We introduce MemBubble, a benchmark of 1000 instances built on preferences extracted from real user-assistant conversations. To measure the extent to which memory concentrates response distributions, we construct test queries in scenarios distinct from user preferences while preserving topical relevance through shared preference categories. Across multiple LLMs and memory mechanisms, entropy decreases by 34.7% on average compared with directly providing the full interaction history, highlighting the important role of memory processing in diversity loss. We also explore simple user-side interventions and changes in information availability but find that these do not effectively close the diversity gap. Our findings highlight the need to consider preserving diversity and user access to information alongside improvements in personalization when designing and using agent memory systems.

Figures & tables

Figure 1: Illustration of response diversity decrease as memory steers responses toward similar preference-aligned content, potentially reinforcing existing biases.

Figure 2: Construction of MemBubble benchmark. Each instance pairs a multi-session interaction history assembled from a preference dialogue and sampled background dialogues with an open-ended test query.

Panel of Figure 3 — Figure 3: Overview of our proposed MemBubble dataset.

Adapted Distinct- $k$
	No mem.	Full history	NaiveRAG	MemoryBank	A-Mem	mem0	MemoryOS
GPT-6-Sol	3.67	3.54 $-0.12$	2.67 $-1.00$	2.41 $-1.26$	2.28 $-1.39$	2.16 $-1.51$	2.81 $-0.85$
GPT-5.6-Luna	3.56	3.50 $-0.06$	2.72 $-0.85$	2.43 $-1.13$	2.23 $-1.33$	2.21 $-1.35$	2.71 $-0.85$
Claude Haiku 4.5	3.58	2.87 $-0.71$	2.86 $-0.72$	2.34 $-1.24$	2.38 $-1.20$	2.09 $-1.49$	3.04 $-0.54$
Gemini 3.7 Flash	2.30	2.47 $+0.18$	2.13 $-0.17$	1.93 $-0.36$	1.79 $-0.51$	1.98 $-0.32$	2.11 $-0.18$
GLM-5.3-flash	3.05	3.55 $+0.50$	2.63 $-0.43$	2.38 $-0.68$	2.14 $-0.91$	2.19 $-0.86$	2.65 $-0.41$

Table 1: Response diversity results across different memory conditions with

k{=}10

and temperature

1.0

. Higher values indicate greater diversity for both metrics. Annotations to the right of the metric values report signed differences from No mem .

Figure 4: Distribution of distinct-

k

of MemBubble samples across memory conditions. Dashed vertical lines indicate the mean for each condition.

Figure 5: Left: mean distinct-

k

across models and memory conditions. Right: distinct-

k

change from No mem. versus preference hit rate. One point represents a model with a memory condition.

Figure 6: Effect of prompting strategy on mean class entropy of different memory conditions.

Panel of Figure 7 — Figure 7: Effects of generation settings on mean distinct- $k$ (left) and class entropy (right) across memory conditions: (a) sampling temperature for GLM-5.3-flash; (b) reasoning effort for GPT-6-Sol.

# Dialogues	0	5	10	20	50	100
MemoryBank	-1.31	-1.00	-0.99	-0.90	-0.83	-0.67
A-Mem	-1.39	-1.24	-1.15	-0.98	-0.93	-0.54
mem0	-1.24	-1.21	-1.18	-1.27	-1.15	-0.93
MemoryOS	-1.27	-0.82	-0.88	-0.80	-0.49	-0.40

Table 2: Mean Distinct-

k

change from No mem. across different background dialogues on subset from 200 randomly selected samples.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task type

Keywords, patterns, or conditions

Rewriting, translation, polishing, proofreading, or summarization

Trigger keywords: translate into , translate , rewrite , reword , rephrase , paraphrase , summarize , summarise , proofread , correct the grammar , fix the grammar , check the grammar , make it more , make it less , shorten , expand this ; 改写, 翻译成, 翻译, 润色, 校对, 总结, 概括, 精简. Required long text block: At least 400 characters in the message; three consecutive nonempty lines totaling at least 300 characters; or a quoted span of at least 200 characters enclosed by """ … """ , " … " , or 「…」. Any one condition suffices.

Role-playing and jailbreak

Expressions: DAN , AIM , JailBreak , developer mode , do anything now ; you are [now] (a|an|my) , pretend to be , pretend you are , act as (a|an|my) ; role-play , role_play , roleplay ; stay in character , ignore [all] (previous|prior) instructions , without any restrictions ; 你现在是, 扮演, 不受[任何]限制. Persona-field prefixes: name: , age: , gender: , personality: , appearance: , background: , species: . A persona-field line begins with one of these prefixes.

Appendix

Table 4: Keywords and textual conditions used for task-type filtering, grouped by the task types excluded in Section 3 . For readability, we list only English and Chinese keywords here, while the filtering rules also include keywords in other major languages. Square brackets indicate optional text, and a vertical bar separates alternatives. The conversation-level exclusion criteria are given in Appendix A.1 .

Stage	Role	Model	Prompt
§ 3.2 Preference extraction	Extractor	DeepSeek-V4-Flash ( DeepSeek-AI and others, 2026 )	Figure 12
§ 3.2 Consistency check	Verifier	GPT-5.6-Luna ( OpenAI, 2026b )	Figure 13
§ 3.2 Semantic assessment	Judges	GPT-5.5 ( OpenAI, 2026a )	Figure 14
§ 3.2 Semantic assessment	Judges	Claude Opus 4.8 ( Anthropic, 2026 )	Figure 14
§ 3.3 Test query generation	Generator	GPT-5.5 ( OpenAI, 2026a )	Figure 15
§ 3.3 Leakage and relevance check	Judge	GPT-5.5 ( OpenAI, 2026a )	Figure 16

Appendix

Table 5: LLMs and prompts used at each stage of MemBubble construction.

Stage	H1–H2	LLM–H1	LLM–H2
Consistency	93.5%	86.0%	90.5%
Semantic assessment	81.0%	83.5%	78.5%
Leakage and Relevance	90.5%	87.5%	91.0%

Appendix

Table 6: Human validation of dataset construction. Each cell reports agreement rate, and H1 and H2 denote the two independent annotators.

Figure 8: Length distributions across the 1,000 MemBubble instances. Each panel includes all instances; bar heights indicate the percentage in each bin. Bin widths are 500 tokens, 5 messages, 2 words, and 1 word, respectively. Source dialogue lengths are measured before adding background sessions.

Field	Mean	Median	Q1–Q3	95th pct.	Min–Max
Source dialogue (tokens)	3,118.2	2,914	1,633–4,009	7,236	76–12,675
Source dialogue (messages)	17.6	14	12–20	34	10–84
Preference (words)	16.2	16	13–19	25	1–35
Test query (words)	11.8	12	11–13	15	4–22

Appendix

Table 7: Detailed length statistics over all 1,000 instances. Q1 and Q3 are the 25th and 75th percentiles; percentiles are rounded to integers. A message is one user or assistant utterance. Preference and query word counts use whitespace splitting, consistent with the dataset overview.

Figure 9: Distributions of relative evidence positions: (a) the supporting message within its source dialogue; (b) the evidence session within the constructed history. Both histograms use 10%-wide bins over all 1,000 instances and share the same axis scales.

Model	No mem.	Full history	NaiveRAG	MemoryBank	A-Mem	mem0	MemoryOS
GPT-6-Sol	4	6	4	1	3	1	2
GPT-5.6-Luna	4	4	1	1	1	1	2
Claude Haiku 4.5	4	5	3	1	1	1	1
Gemini 3.7 Flash	4	4	1	1	1	1	1
GLM-5.3-flash	1	5	1	1	1	1	2
Llama3.3-70B	7	5	6	1	1	1	2

Appendix

Table 8: Distinct-

k

for the sampled instance across response models and memory conditions, with ten independently sampled responses at temperature 1.0.

Figure 10: Effect of response sample size on mean distinct-

k

(left) and class entropy (right) for GPT-5.6-Luna across memory conditions. Here,

k

is the number of independently sampled responses per query.

Panel of Figure 18 — Figure 18: Distributions of distinct- $k$ across memory conditions for the five response models beyond GPT-5.6-Luna, whose results appear in Figure 4 . Each histogram shows the proportions of MemBubble instances at each $d_{k}$ value with $k=10$ independently sampled responses per query. Dashed vertical lines indicate the mean for each condition.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 357 papers

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
May 14, 2026Jingbo Yang, Kwei-Herng Lai, Xiaowen Wang +3Conversational Memory LLM Agent Evaluation
UC Santa Barbara · Microsoft
Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents
Jun 24, 2026Yuxin Wang, Paul Thomas, Zhiwei Yu +5Conversational Memory LLM Agent Evaluation
1Dartmouth College · 2Microsoft · †Independent researcher.
DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
Jun 22, 2026Wenya Xie, Shengming Zhou, Zelin Li +9Agent Memory LLM Agent Memory
University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2

May 14, 2026J/K move · Enter open · S save

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

Jingbo Yang, Kwei-Herng Lai, Xiaowen Wang, Shiyu Chang+2

UC Santa Barbara · Microsoft

Conversational Memory LLM Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks Long-Term Conversational Memory

Open paper AskPDF arXiv

Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.

Abstract

LLM Agent Memory

Explore similar work

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings