One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough
Organizations: Tencent Yuanbao Team · King’s College London · Qilu University of Technology · Renmin University of China ·
Abstract
The value of memory lies not only in recall but in using it to provide appropriate assistance. Recent benchmarks test memory use in personalisation and agentic tasks, but seldom evaluate assistance without fixed answers or integration across multiple participants. Group chats make both challenges more acute: an assistant must integrate information from multiple members, track revisions and conflicts over time, and produce one response for the group. Yet existing group memory benchmarks rely on synthetic conversations and inherit constrained evaluation from general memory benchmarks, focusing on recall against fixed answers or decisions over a closed set. We introduce GroupAssistBench, the first memory benchmark from real-world group conversations with a deployed group assistant, covering 80,299 messages in 50 groups and costing over $53K. Over the same histories, GroupAssistBench evaluates memory recall (L1) with reference answers and memory use (L2) with service requests without fixed answers across single, distributed, and conflicting memories (M1--M3). Requests are verified to require group history, memories are anchored to their sources, and L2 uses tailored rubrics with penalties derived from real user feedback. Across evaluations of frontier models with full context, retrieval methods, and memory systems, the strongest memory system rivals full context on recall but falls far behind on use, especially for distributed and conflicting memories. Together, two additions that relate retrieved memories and bind remembered facts as response constraints bring consistent gains, showing that group memory systems must improve use, not recall alone.
Figures & tables
| Benchmark | Group | Source | Update or conflict | Open service | Rubric | Feedback | #Eval. |
| Memory for one user and agent tasks | |||||||
| MemoryArena ( He et al., 2026 ) | ✗ | partly synthetic | ✗ | ✗ | ✗ | 766 | |
| AMA-Bench ( Zhao et al., 2026 ) | ✗ | agent trajectories | ✓ | ✗ | ✗ | ✗ | 2,496 |
| PersonaMem-v2 ( Jiang et al., 2025 ) | ✗ | synthetic | ✓ | ✗ | ✗ | 5,000 | |
| AlpsBench ( Xiao et al., 2026 ) | ✗ | real user logs | ✓ | ✗ | ✗ | 2,500 | |
| DynamicMem ( Xie et al., 2026 ) | ✗ | synthetic | ✓ | ✗ | ✗ | 3,634 | |
| M1 | M2 | M3 | Total | |
|---|---|---|---|---|
| L1: memory recall | 776 | 448 | 150 | 1,374 |
| L2: memory use | 221 | 650 | 150 | 1,021 |
| Total | 997 | 1,098 | 300 | 2,395 |
| Method | Memory type | L1-M1 | L1-M2 | L1-M3 | L1 avg. met | L1 rank | L2-M1 | L2-M2 | L2-M3 | L2 avg. | L2 rank | Ctx. tokens | |||
| met | F1 | met | F1 | met | F1 | ||||||||||
| Full Context | |||||||||||||||
| Claude Sonnet 5 | 1D | 0.739 | 0.570 | 0.504 | 0.460 | 0.429 | 0.352 | 0.628 | 2 | 0.473 | 0.426 | 0.371 | 0.428 | 3 | 52,435 |
| DeepSeek-v4-flash | 1D | 0.766 | 0.526 | 0.572 | 0.472 | 0.455 | 0.353 | 0.669 | 1 | 0.510 | 0.478 | 0.374 | 0.469 | 1 | 52,435 |
| Qwen-3.5-9b | 1D | 0.622 | 0.467 | 0.344 | 0.341 | 0.285 | 0.278 | 0.494 | 11 | 0.446 | 0.343 | 0.226 | 0.348 | 6 | 52,435 |
| GPT-5.6-terra | 1D | 0.685 | 0.623 | 0.436 | 0.453 | 0.377 | 0.356 | 0.570 | 4 | 0.505 | 0.435 | 0.303 | 0.431 | 2 | 52,435 |
| Metric | EverMemOS | xMemory | LightMem | ||||
|---|---|---|---|---|---|---|---|
| Baseline | +RER | Baseline | +RER | Baseline | +RER | ||
| L1 met | M1 | 0.679 | 0.693 (+1.45) | 0.620 | 0.641 (+2.11) | 0.290 | 0.380 (+8.98) |
| M2 | 0.540 | 0.544 (+0.39) | 0.455 | 0.494 (+3.86) | 0.133 | 0.216 (+8.13) | |
| M3 | 0.561 | 0.566 (+0.50) | 0.435 | 0.456 (+2.14) | 0.178 | 0.235 (+5.71) | |
| L2 | M1 | 0.137 | 0.160 (+2.28) | 0.133 | 0.157 (+2.32) | 0.192 | 0.215 (+2.32) |
| M2 | 0.284 | 0.294 (+1.08) | 0.223 | 0.245 (+2.22) | 0.131 | 0.207 (+7.52) | |
| L2 metric | EverMemOS | xMemory | LightMem | |||
|---|---|---|---|---|---|---|
| Baseline | +ACB | Baseline | +ACB | Baseline | +ACB | |
| M1 | 0.137 | 0.203 (+6.63) | 0.133 | 0.201 (+6.77) | 0.192 | 0.212 (+1.96) |
| M2 | 0.284 | 0.306 (+2.17) | 0.223 | 0.257 (+3.39) | 0.131 | 0.159 (+2.80) |
| M3 | 0.253 | 0.276 (+2.30) | 0.220 | 0.243 (+2.17) | 0.094 | 0.106 (+1.20) |
| Avg. | 0.247 | 0.279 (+3.16) | 0.203 | 0.242 (+3.94) | 0.139 | 0.163 (+2.38) |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Kind | Criterion | Weight |
|---|---|---|
| Grounding | Core anchor of the instance | |
| Grounding | Other anchors | |
| Service | Mandatory requirement of the request | |
| Service | Optional improvement | |
| Penalty | See Table 7 | to |
| Failure pattern | Trigger | Feedback | Weight |
|---|---|---|---|
| Distorting a recalled fact | An anchor contains a numeric or other exact value that is not already covered by a grounding criterion | 24 | |
| Contradicting a settled plan | The group has settled a plan that is relevant to the request before the cutoff | 17 | |
| Violating an explicit agreement or member constraint | A constraint is stated explicitly in the group, rather than inherited from a profile, and is relevant to the request | 21 | |
| Misreading the current state of a member | A member states a current state within 150 messages before the cutoff | 23 | |
| Relying on a superseded value | The instance contains a verified conflict pair | 12 |
| Valid, per annotator (%) | Valid (%) | ||||||
|---|---|---|---|---|---|---|---|
| Question | 1 | 2 | 3 | Majority | Unanimous | Agreement (%) | |
| Q1 Anchor established by cited message | 120 | 99.2 | 97.5 | 97.5 | 98.3 | 96.7 | 98.3 / 98.3 / 98.3 |
| Q2 Anchor necessary | 120 | 96.7 | 96.7 | 95.8 | 97.5 | 94.2 | 98.3 / 96.7 / 96.7 |
| Q3 Answer key or rubric valid | 120 | 97.5 | 95.8 | 95.0 | 97.5 | 93.3 | 98.3 / 96.7 / 95.8 |
| Q4 Conflict pair valid (M3) | 40 | 97.5 | 95.0 | 97.5 | 100.0 | 90.0 | 92.5 / 95.0 / 92.5 |
| Instance valid (all questions) | 120 | 95.0 | 91.7 | 90.0 | 95.8 | 84.2 | 93.3 / 91.7 / 90.0 |
| Agreement | Judge vs. majority | Disagreement (%) | |||||
|---|---|---|---|---|---|---|---|
| Criterion type | Human–human | Judge–human | Precision | Recall | Miss | Extra | |
| Grounding (G) | 351 | 0.85 | 0.86 | 1.00 | 0.67 | 14.2 | 0.0 |
| Service, required (S) | 300 | 0.88 | 0.89 | 0.99 | 0.80 | 10.7 | 0.3 |
| Service, bonus (S) | 180 | 0.86 | 0.86 | 0.92 | 0.70 | 11.7 | 2.2 |
| Penalty (P) | 321 | 0.99 | 0.95 | 0.48 | 1.00 | 0.0 | 4.7 |
| All | 1,152 | 0.90 | 0.89 | 0.94 | 0.74 | 8.9 | 1.7 |
| Question distribution | ||||
|---|---|---|---|---|
| Level | M1 | M2 | M3 | Total |
| L1 | 776 | 448 | 150 | 1,374 |
| L2 | 221 | 650 | 150 | 1,021 |
| Total | 997 | 1,098 | 300 | 2,395 |
| Corpus scale | ||||
| Statistic | Total | Min. | Mean | Max. |
| Method | History and memory access |
|---|---|
| Full Context | Supplies conversation history before the cutoff directly to the task agent, with context fitting when needed. |
| BM25 | Uses overlapping windows of 10 messages with a stride of 5 messages. Ranks these windows and eligible profile entries with BM25 and retrieves the top 20 units. |
| Dense retrieval | Uses the same overlapping windows of 10 messages with a stride of 5 messages and eligible profile entries. Retrieves the top 20 units using embedding similarity. |
| GraphRAG | Uses local search over entities and relations to retrieve graph evidence and associated source passages. |
| RAPTOR | Builds a hierarchy of messages and summaries and retrieves relevant representations across its levels. |
| Mem0 | Extracts and updates memory entries from the conversation, then retrieves entries relevant to the query. |
| Method | Build LLM | Build embed. | Retrieval | Answer | Judge | Total |
|---|---|---|---|---|---|---|
| Full Context (Claude Sonnet 5) | 0.0 | 0.0 | 0.0 | 249.4 | 3.3 | 252.8 |
| Full Context (Terra) | 0.0 | 0.0 | 0.0 | 183.4 | 3.2 | 186.6 |
| Full Context (DeepSeek) | 0.0 | 0.0 | 0.0 | 156.0 | 3.4 | 159.4 |
| Full Context (Qwen) | 0.0 | 0.0 | 0.0 | 164.6 | 3.7 | 168.2 |
| BM25 | 0.0 | 0.0 | 0.0 | 37.8 | 3.2 | 41.0 |
| Dense retrieval | 0.0 | 9.7 | 0.2 | 33.8 | 3.2 | 46.8 |
| Method | Construction cost (USD) | Cost/query (USD) | (USD) | |
|---|---|---|---|---|
| Full Context (Terra) | 0.0 | 0.1088 | — | 260.4 |
| BM25 | 0.0 | 0.0253 | 0 | 60.6 |
| Dense retrieval | 0.2 | 0.0229 | 3 | 55.0 |
| GraphRAG | 264.6 | 0.0906 | 14,574 | 481.5 |
| RAPTOR | 127.6 | 0.0321 | 1,665 | 204.5 |
| Mem0 | 232.9 | 0.0062 | 2,270 | 247.7 |
| System | Level | RER | ACB | RER + ACB | Best condition |
|---|---|---|---|---|---|
| xMemory | L1 met | RER + ACB | |||
| L2 | RER + ACB | ||||
| EverMemOS | L1 met | RER + ACB | |||
| L2 | RER + ACB | ||||
| LightMem | L1 met | RER + ACB | |||
| L2 | RER + ACB |
| System | Level | Condition | M1 | M2 | M3 |
|---|---|---|---|---|---|
| xMemory | L1 | Baseline | 0.760 | 0.641 | 0.677 |
| RER | 0.760 | 0.641 | 0.677 | ||
| ACB | 0.760 | 0.639 | 0.674 | ||
| RER + ACB | 0.762 | 0.640 | 0.682 | ||
| L2 | Baseline | 0.482 | 0.444 | 0.515 | |
| RER | 0.482 | 0.444 | 0.515 |
| Computational costs | |||
|---|---|---|---|
| Component | Input tokens | Output tokens | Cost (USD) |
| Benchmark construction | 1,701.3M | 62.1M | 10,370 |
| Preliminary benchmark iteration | 315.8M | 8.4M | 1,832 |
| Agent-assisted benchmark engineering | 2,583.4M | 11.1M | 13,249 |
| M-axis, L-axis, and combined analysis experiments | 706.5M | 31.2M | 1,871 |
| Baseline adaptation and reruns (estimated) | 961.1M | 103.9M | 3,076 |
Memory Agent Benchmarks
Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.
See all 183 papers