Who Remembers What? Multi-Granularity Memory Extraction for Multi-User, Multi-Agent Interaction
Organizations: AWS Agentic AI · University of Southern California
Abstract
Self-evolving agents improve their own skills, memory, and harness code from experience without retraining the base LLM, and group-evolving agents go further by sharing these lessons across agents. Yet most such systems overlook that sharing across users and heterogeneous agents requires deciding who can benefit from each lesson and what information must remain private. We present MixUA, a memory system that distills independent user-agent trajectories into lessons and assigns each lesson to a user-agent pair, user, agent, or global scope according to whom it benefits and what it may disclose. For evaluation, we introduce CrossLearnBench, a mixed-user-agent benchmark that pairs established agent tasks with user preferences and privacy probes that test whether private facts leak into memory other users can read. With five users and seven agents, MixUA raises pass rates by up to 16 points over no memory and 5 over a single global memory, even on tasks previously tried only by other users on other agents. It retains up to 90% of previously stated preferences, and at most 0.02% of privacy probes find a planted private fact in a forbidden scope. We hope MixUA offers a scalable path toward a global system where users and agents co-evolve.
Figures & tables
| Utility | Preference | Privacy | Resource | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Condition | Pass R3 | Evol. | Held-out | Pref. leak | Exposed | Turns | Memory | Cost | ||
| No memory | 72.0 1.2 | – | – | 11 | – | n/a | 8 | 14.8 | 0 | 0.30 |
| Pair only | 77.6 1.5 | +4.3 0.8 ∗∗ | +1.6 | 80 | 79 | n/a | 20 | 14.8 | 23.0 | 0.60 |
| User only | 80.4 1.4 | +8.8 0.8 ∗∗ | +3.3 | 37 | 20 | n/a | 12 | 13.8 | 39.6 | 1.24 |
| Agent only | 78.6 0.4 | +5.1 1.5 ∗∗ | +3.3 | 12 | 10 | 1.2 ‡ | 9 | 13.8 | 39.7 | 0.66 |
| + protected stream | 80.7 0.1 | +7.3 0.3 ∗∗ | +5.4 ∗ | 33 | 18 | 26.3 | 10 | 13.4 | 39.0 | 1.05 |
| Evolution | Held-out | ||||
| Budget | Context | Pass | Pref. | Cost | |
| 1k | 5.7 | +8.4 | 71.2 | 60.3 | 54 |
| 5k | 13.2 | +10.4 | 70.3 | 66.1 | 54 |
| 10k | 22.8 | +13.1 | 71.6 | 78.0 | 60 |
| 20k | 37.8 | +14.0 | 75.4 | 87.3 | 65 |
| 40k | 61.4 | +16.3 | 75.4 | 90.9 | 81 |
| Benchmark | Scope probe | |||||
|---|---|---|---|---|---|---|
| Removed | Pref. | Metric | With | W/o | ||
| None | 15.3 | 13.0 | 90 | |||
| Global | 9.0 | 4.6 | 90 | |||
| User | 10.5 | 8.8 | 81 | |||
| Agent | 14.8 | 12.5 | 91 | Crowd pass | 89 | 40 |
| Spillover | 44 | 67 | ||||
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Same pair | Shared user/agent | Neither shared | Unseen |
|---|---|---|---|---|
| =195 | =389 | =308 | =52 | |
| Pair only | +12.3 ∗∗ | +3.9 ∗ | +1.6 | -5.6 |
| User only | +8.2 ∗∗ | +14.4 ∗∗ | +3.3 | +1.8 |
| Agent only | +9.7 ∗∗ | +5.7 ∗∗ | +3.3 | -5.5 |
| Global only | +12.3 ∗∗ | +14.7 ∗∗ | +8.9 ∗∗ | +0.0 |
| MixUA (LLM) | +13.3 ∗∗ | +19.3 ∗∗ | +13.0 ∗∗ | +5.8 |
| Agent | None (%) | Pair | User | Agent | Global | MixUA |
|---|---|---|---|---|---|---|
| OpenClaw-Sonnet-5 | 45.0 | +2.5 | +13.3 ∗∗ | +5.0 | +15.4 ∗∗ | +15.8 ∗∗ |
| Codex-Luna | 62.0 | +10.1 ∗ | +16.4 ∗∗ | +10.9 ∗∗ | +21.3 ∗∗ | +22.7 ∗∗ |
| Claude Code-Sonnet-5 | 68.2 | +7.9 ∗ | +6.3 ∗ | +8.5 ∗ | +9.3 ∗∗ | +12.9 ∗∗ |
| Codex-Terra | 69.8 | -4.2 | +1.0 | +4.2 | +10.4 ∗∗ | +10.7 ∗∗ |
| Hermes-Sonnet-5 | 72.0 | +6.7 ∗ | +9.6 ∗∗ | +4.3 | +6.0 ∗ | +19.2 ∗∗ |
| Codex-Sol | 80.1 | +4.5 | +10.3 ∗∗ | +5.1 | +16.0 ∗∗ | +16.8 ∗∗ |
| Run 1 | Run 2 | Run 3 | Pooled | |||||
| Store | Visible | Pass | Pref. | Pass | Pref. | Pass | Pref. | (gain/loss) |
| MixUA (LLM) | Empty | 66.2 | 9 | 65.5 | 9 | 65.2 | 9 | – |
| GLOBAL | 69.1 | 10 | 75.4 | 9 | 76.3 | 10 | +7.8 ∗∗ ∗∗ (42/10) | |
| USER | 73.4 | 76 | 74.8 | 89 | 72.3 | 91 | +8.2 ∗∗ ∗∗ (53/19) | |
| AGENT | 65.5 | 8 | 67.1 | 11 | 70.5 | 9 | +2.2 ∗∗ (21/12) | |
| PAIR | 66.1 ×2 | 46 | 68.3 | 48 | 68.6 | 54 | +1.2 ∗∗ (29/24) | |
| In-domain ( ) | Out-of-domain ( ) | |||||||
|---|---|---|---|---|---|---|---|---|
| Store | Empty | All | Global | Restated | Empty | All | Global | Restated |
| MixUA (LLM) | 62.9 | +9.3 ∗∗ | +8.9 ∗∗ | +7.2 ∗∗ | 69.6 | +8.6 ∗ | +6.2 | +2.5 |
| MixUA (LLM) + review | 62.9 | +6.6 ∗ | +7.9 ∗∗ | +6.1 ∗∗ | 71.4 | +6.9 ∗ | +6.9 | +3.1 |
| MixUA (agentic) | 64.9 | +3.0 | +0.8 | +4.7 | 67.3 | +8.5 ∗ | +7.9 ∗ | +4.9 |
| MixUA (agentic) + review | 61.8 | +9.2 ∗∗ | +5.9 ∗∗ | +6.3 ∗ | 70.8 | +5.4 | +3.0 | -1.2 |
| Pair only | 62.8 | +8.5 ∗∗ | – | – | 70.8 | +3.0 | – | – |
| Context | Ungraded | USD per run | R3 vs. R1, always-pass tasks (%) | ||||
|---|---|---|---|---|---|---|---|
| Condition | (k chars) | (%) | Agent | Memory | Turns | Time | Cost |
| No memory | 0 | 1.7 | 154 | 0 | +2 | -18 | -4 |
| Pair only | 23.0 | 1.4 | 175 | 101 | +1 | +16 | +55 |
| User only | 39.6 | 1.8 | 180 | 346 | -13 | -5 | +53 |
| Agent only | 39.7 | 1.6 | 180 | 118 | -18 | -15 | +43 |
| Global only | 39.7 | 1.4 | 156 | 580 | -17 | -23 | +36 |
| Re-evolved store | Fixed store, held-out | |||||||
|---|---|---|---|---|---|---|---|---|
| Budget | Runs | Context | Pass R3 | Pref. R3 | Pass | Pref. | Cost | |
| 1k | 2 | 5.7 | +8.4 | 83.3 | 59.5 | 71.2 0.8 | 60.3 0.4 | 54 2 |
| 5k | 2 | 13.2 | +10.4 | 87.2 | 73.7 | 70.3 1.1 | 66.1 0.1 | 54 1 |
| 10k | 2 | 22.8 | +13.1 | 85.3 | 87.5 | 71.6 2.4 | 78.0 0.5 | 60 3 |
| 20k | 2 | 37.8 | +14.0 | 85.1 | 86.9 | 75.4 1.3 | 87.3 0.3 | 65 2 |
| 40k | 3 | 61.4 | +16.3 | 89.8 | 89.9 | 75.4 1.6 | 90.9 0.6 | 81 2 |
| Gain over no memory (pp) | Preference (%) | Privacy (%) | Resource | |||||||||
| Condition | Pass | All | First | Shared | Neither | R2-8 | R8 | Pref. leak | Opened | Turns | Memory | Cost |
| =821 | =526 | =112 | =179 | |||||||||
| No memory | 70.5 | – | – | – | – | 11 | 12 | n/a | 10 | 18.2 | 0 | 0.41 |
| Pair only | 73.0 | +2.4 | +2.5 | +3.5 | +1.7 | 66 | 44 | n/a | 15 | 18.7 | 15.6 | 0.70 |
| Global only | 79.1 | +8.5 ∗∗ | +5.5 ∗∗ | +15.9 ∗∗ | +13.0 ∗∗ | 11 | 12 | 0.2 | 9 | 14.8 | 19.5 | 1.94 |
| MixUA (LLM) + review | 82.3 | +11.4 ∗∗ | +10.1 ∗∗ | +12.5 ∗∗ | +15.1 ∗∗ | 81 | 76 | 5.8 | 27 | 16.9 | 38.7 | 3.77 |
| First attempt | Attempted before | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | None | Pair | Global | MixUA | None | Pair | Global | MixUA | ||
| AutoCodeBench | 77 | 61 | +0.0 | +0.0 | +2.6 | 30 | 63 | +0.0 | +23.3 ∗ | +20.0 ∗ |
| FeatureBench | 85 | 61 | +10.5 ∗ | +23.3 ∗∗ | +22.4 ∗∗ | 42 | 60 | -4.8 | +28.6 ∗∗ | +23.8 ∗∗ |
| SpreadsheetBench | 70 | 83 | -5.8 | -4.3 | +4.3 | 47 | 77 | +0.0 | +6.5 | +6.4 |
| SWT-Bench | 83 | 63 | +7.2 | +2.4 | +16.9 ∗∗ | 48 | 54 | +6.4 | +19.1 ∗ | +16.7 |
| Terminal-Bench 2.0 | 55 | 67 | +3.7 | +7.4 | +7.3 | 21 | 67 | +0.0 | +4.5 | +4.8 |
| Gain (pp) | Preference (%) | Privacy | Resource | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Users | Condition | Pass | All | First | Told | All | Pref. leak (%) | Private facts | Memory | User | Cost |
| 5 | No memory | 71.3 | – | – | 10 | 10 | n/a | n/a | 0 | – | 0.35 |
| Pair only | 72.3 | +1.0 | +2.2 | 68 | 68 | n/a | n/a | 15.5 | – | 0.65 | |
| Global only | 79.0 | +7.8 ∗∗ | +5.8 ∗ | 11 | 11 | 0.2 | 0/5685 | 19.7 | – | 1.89 | |
| MixUA (LLM) + review | 80.3 | +8.9 ∗∗ | +8.0 ∗∗ | 89 | 89 | 3.8 | 2/9753 | 39.1 | 15.5 | 3.84 | |
| 20 | No memory | 70.8 | – | – | 11 | 11 | n/a | n/a | 0 | – | 0.38 |
| Utility | Preference | Privacy | Resource | |||||||
| Scopes | Pass R3 | Evol. | Held-out | Pref. leak | Exposed | Turns | Memory | Cost | ||
| MixUA (LLM) | 88.1 1.7 | +15.3 1.8 ∗∗ | +13.0 ∗∗ | 90 | 90 | 4.8 | 26 | 13.2 | 60.2 | 3.35 |
| Global + Pair | 87.3 2.2 | +12.8 1.4 ∗∗ | +11.2 ∗∗ | 81 | 84 | 1.3 | 17 | 12.3 | 43.6 | 3.55 |
| User + Agent + Pair | 81.6 0.4 | +9.0 1.8 ∗∗ | +4.6 | 90 | 91 | 4.4 | 31 | 16.3 | 56.7 | 1.26 |
| Global + Agent + Pair | 84.8 2.2 | +10.5 1.3 ∗∗ | +8.8 ∗∗ | 81 | 81 | 0.8 | 12 | 12.4 | 45.6 | 3.37 |
| Global + User + Pair | 86.8 0.2 | +14.8 0.6 ∗∗ | +12.5 ∗∗ | 91 | 89 | 3.1 | 25 | 13.6 | 57.3 | 3.69 |
| Setting | Metric | With | Without |
|---|---|---|---|
| Pair -Work | Leak | 22 | 95 |
| Recall | 85 | 80 | |
| Agent -Crowd | Pass | 89 | 40 |
| Misapplied | 0 | 14 | |
| Agent -Norm | Spillover | 44 | 67 |
| Pass | 93 | 85 |
| Suite | Topic | Example instruction (paraphrased) |
| TB2 | Set up servers, packages, repos | Package a Python library and serve it from a local PyPI server. |
| Data, ML and science workflows | Rebuild a PyTorch model from its weights; retrain only the output layer. | |
| Recover or compute an answer | Break a FEAL-like cipher with a known-plaintext attack and decrypt the ciphertexts. | |
| Build, port, reverse-engineer code | Port a COBOL program to Python with byte-identical output files. | |
| VMax | Markdown, MDX and template parsing | Fix remark’s inline-code output when the code itself contains backticks. |
| Vue SSR, compiler and DOM rendering | Fix Vue server-side rendering dropping teleported content from the HTML. |
| Suite | Evol. | Held-out | Evol. att. | Att./arm |
|---|---|---|---|---|
| AutoCodeBench | 21 | 4 | 79 | 28 |
| FeatureBench | 20 | 4 | 76 | 28 |
| SpreadsheetBench | 23 | 4 | 95 | 28 |
| SWT-Bench | 22 | 4 | 87 | 28 |
| Terminal-Bench 2.0 | 39 | 0 | 39 | 0 |
| USACO | 23 | 4 | 107 | 28 |
| Group | Tasks | R1 | R2 | R3 | Attempts |
| Domain coverage | 60 | 72 | 90 | 90 | 252 |
| Repeated pair | 10 | 20 | 20 | 20 | 60 |
| Cross-agent | 14 | 14 | 14 | 14 | 42 |
| Fresh-task control | 39 | 25 | 7 | 7 | 39 |
| Task family | 30 | 10 | 10 | 10 | 30 |
| Preference pair | 5 | 10 | 10 | 10 | 30 |
| Condition | Writable scopes | Writer | Review |
|---|---|---|---|
| No memory | none | n/a | n/a |
| Pair / User / Agent / Global -only | one | LLM | off |
| MixUA (LLM) | all four | LLM | off |
| MixUA (LLM) + review | all four | LLM | LLM |
| MixUA (agentic) | all four | agentic | off |
| MixUA (agentic) + review | all four | agentic | agentic |
| Role | Harness (version) | Model id | Turn cap |
|---|---|---|---|
| Task agent | Claude Code (2.1.263) | claude-opus-5 , claude-sonnet-5 | 150 |
| Task agent | Hermes (0.19.0) | claude-sonnet-5 | 150 |
| Task agent | OpenClaw (2026.7.1) | claude-sonnet-5 | none |
| Task agent | Codex CLI (0.153.4) | gpt-5.6-sol , gpt-5.6-luna , gpt-5.6-terra | none |
| Splitter, writer, review (LLM) | – | claude-opus-5 | – |
| Agentic writer and review | Codex CLI (0.139.0) | gpt-5.6-sol | none |
| Registry dataset | Repository | Commit | Evol. | Held-out |
|---|---|---|---|---|
| autocodebench@lite200 | laude-institute/harbor-datasets | d3ac43a32042f566b117630adfdc9e72c96893cd | 21 | 4 |
| [email protected] | laude-institute/harbor-datasets | 37db108843a49bb31a592e37a75e2c40dc3f9749 | 20 | 4 |
| [email protected] | laude-institute/harbor-datasets | 0ad7209b09780e78c6c80f194e49e353191be1cc | 23 | 4 |
| [email protected] | laude-institute/harbor-datasets | e39578f9f952caa51cd6c0de04f125254102994a | 22 | 4 |
| [email protected] | laude-institute/terminal-bench-2 | 69671fbaac6d67a7ef0dfec016cc38a64ef7a77c | 39 | 0 |
| [email protected] | laude-institute/harbor-datasets | 56fac05ddbf784bfaa640f4919bceef50433fbed | 23 | 4 |
| Split | Task ids |
|---|---|
| AutoCodeBench ( autocodebench@lite200 ) | |
| Evolution (21) | autocodebench-cpp-001 , autocodebench-cpp-006 , autocodebench-csharp-001 , autocodebench-csharp-003 , autocodebench-csharp-009 , autocodebench-dart-004 , autocodebench-dart-008 , autocodebench-dart-009 , autocodebench-elixir-003 , autocodebench-elixir-008 , autocodebench-go-004 , autocodebench-go-009 , autocodebench-go-010 , autocodebench-java-001 , autocodebench-java-004 , autocodebench-javascript-001 , autocodebench-javascript-005 , autocodebench-julia-005 , autocodebench-kotlin-001 , autocodebench-kotlin-005 , autocodebench-kotlin-008 |
| Held-out (4) | autocodebench-dart-001 , autocodebench-javascript-008 , autocodebench-javascript-009 , autocodebench-javascript-010 |
| FeatureBench ( [email protected] ) | |
| Evolution (20) | astropy__astropy.b0db0daa.test_basic_rgb.067e927c.lv2 , astropy__astropy.b0db0daa.test_compressed.8daeb7d6.lv1 , astropy__astropy.b0db0daa.test_compression_failures.48dc420b.lv1 , astropy__astropy.b0db0daa.test_containers.6079987d.lv1 , astropy__astropy.b0db0daa.test_converter.2ef0539f.lv1 , astropy__astropy.b0db0daa.test_tilde_path.383d7c0c.lv1 , astropy__astropy.b0db0daa.test_tiled_compression.a4fe298a.lv1 , lightning-ai__pytorch-lightning.126fa6f1.test_connector.825ef3b1.lv1 , lightning-ai__pytorch-lightning.126fa6f1.test_hooks.34620588.lv1 , lightning-ai__pytorch-lightning.126fa6f1.test_wrappers.d88ce6f6.lv1 , lightning-ai__pytorch-lightning.126fa6f1.test_xla.7c94e7d2.lv1 , matplotlib__matplotlib.86a476d2.test_angle_helper.48e96b5a.lv2 , matplotlib__matplotlib.86a476d2.test_backend_registry.872ba384.lv1 , mlflow__mlflow.93dab383.test_ai_command_utils.85dcb487.lv2 , mlflow__mlflow.93dab383.test_bedrock_autolog.f008b521.lv1 , mlflow__mlflow.93dab383.test_client.42de781c.lv1 , mlflow__mlflow.93dab383.test_client_webhooks.75e39b52.lv1 , mlflow__mlflow.93dab383.test_dependencies_schema.7019486f.lv1 , mlflow__mlflow.93dab383.test_file_store_logged_model.a9596c54.lv1 , mlflow__mlflow.93dab383.test_jsonpath_utils.eb851c42.lv1 |
| Held-out (4) | fastapi__fastapi.02e108d1.test_compat.71e8518f.lv2 , lightning-ai__pytorch-lightning.126fa6f1.test_data.12056068.lv1 , lightning-ai__pytorch-lightning.126fa6f1.test_seed.1eaa1d46.lv2 , pandas-dev__pandas.82fa2715.test_iceberg.85771c70.lv2 |
| Item | Category | Q8 wording (condensed) | Used as |
|---|---|---|---|
| Q8A | Purchase habits | Basic purchasing habits, such as foods, clothes, and stores | survey fact |
| Q8B | Friends | Who your friends are and what they are like | survey fact |
| Q8C | Location history | Physical location over time, from cell phone GPS | survey fact |
| Q8E | Health and medications | State of your health and the medicines you take | survey fact |
| Q8F | Political views | Political views and the candidates you support | survey fact |
| Q8I | Media tastes | Tastes in music, movies, books, websites, magazines | survey fact |
| Preference | Options | Check on the agent’s document |
|---|---|---|
| Filename (16) | handoff-note.md , run-record.md , completion-note.md , delivery-note.md , work-log.md , outcome-memo.md , activity-log.md , progress-digest.md , session-digest.md , wrap-up.md , resolution-brief.md , task-recap.md , task-writeup.md , debrief.md , field-notes.md , closing-brief.md | a file the agent created has exactly this name |
| Sign-off date (8) | YY:MM:DD , YYYY-MM-DD , DD/MM/YYYY , MM/DD/YY , YYYY.MM.DD , DD-MMM-YYYY , YYYYMMDD , MMM DD, YYYY | the last content line holds a date in this format (format, not value) |
| Sections (10) | the ten 3-subsets of {Summary, Approach, Changes, Testing, Next Steps}, each with a fixed order; each section appears twice in each position | the three headings appear, first occurrences in this order |
| Banned markup (3) | em dashes; bold or italic emphasis; bullet or numbered lists | absent from the prose, ignoring code and URLs |
| Pew response | Sign-off date per harness | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| User | Case | Private | Filename | Sections (in order) | Banned | Claude Code | Codex | Hermes | OpenClaw |
| u0 | 457 | 1/14 | work-log.md | Testing, Summary, Next Steps | lists | YYYY.MM.DD | DD/MM/YYYY | DD-MMM-YYYY | YYYYMMDD |
| u1 | 110 | 6/14 | session-digest.md | Next Steps, Summary, Approach | emphasis | MMM DD, YYYY | MM/DD/YY | DD/MM/YYYY | YYYY.MM.DD |
| u2 | 107 | 10/14 | completion-note.md | Summary, Testing, Changes | em dash | MM/DD/YY | DD-MMM-YYYY | YYYY.MM.DD | YY:MM:DD |
| u3 | 382 | 12/14 | resolution-brief.md | Approach, Changes, Testing | lists | DD/MM/YYYY | MMM DD, YYYY | YYYYMMDD | YYYY-MM-DD |
| u4 | 133 | 9/14 | task-writeup.md | Approach, Testing, Summary | emphasis | YYYY-MM-DD | YYYY.MM.DD | MM/DD/YY | DD-MMM-YYYY |
| Other | ||||||||
|---|---|---|---|---|---|---|---|---|
| Item | Global | User | Agent | Pair | users | agents | pairs | Released |
| Issued identifier | N | N | N | N | N | N | N | N |
| Protected fact | N | F | N | F | N | N | N | N |
| Shareable fact | F | F | F | F | N | F | N | F |
| Base preference | N | M | N | F | N | N | N | F |
| Harness override | N | N | N | M | N | N | N | F |
LLM Agent Memory
Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.
See all 399 papers