cs.AIMar 24, 2026
SavePERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
Organizations: University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong · Northeastern University · MemTensor (Shanghai) Technology Co., Ltd.
Abstract
Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.
Figures & tables
Figure 1: Comparison of context construction and evaluation. ( Left ) Existing benchmarks: Evaluate isolated preferences via sparse, “Needle-in-a-Haystack” retrieval. ( Right ) PERMA : Implements an event-driven paradigm where preferences are integrated over time and across sessions to assess the capabilities of memory systems.
| Metric | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 | Total |
| User Demographics | |||||||||||
| Age | 35-44 | 25-34 | 65+ | 55-64 | 35-44 | 18-24 | 25-34 | 35-44 | 25-34 | 25-34 | – |
| Gender | M | M | F | M | M | M | M | M | F | M | – |
| Education | Grad. | Univ. | – | Univ. | – | Grad. | Univ. | – | Grad. | Univ. | – |
| Country | CA | MX | FI | US | AU | UK | CH | IL | RU | BE | – |
| Interaction Statistics (Base Dataset: Clean / Noisy) | |||||||||||
Table 1: Comprehensive statistics of the dataset across multiple dimensions. The table summarizes user demographics for profiles, followed by statistics for the base (comparing Clean and Noise ) and the style-aligned long-context dataset.
| Type | Interaction Examples (U: User, A: Assistant) |
| [1] Omitted Info. | U: Can you give me guidance on that game ? A: Could you specify which game? U: The Witcher 3 . A: Certainly. I suggest optimizing the PC graphics and … |
| [2] Context Switch | U: By the way , do you remember my hockey tickets? A: Yes, Maple Leafs tickets for Nov 5th… U: Thanks. Now back to the wine tasting… A: Understood. Regarding the culinary events … |
| [3] Inconsistent Pref. | U: For this week, I want to see the biggest conference. A: The Global Tech Summit (10k+ attendees) is happening now. U: Those are too crowded. Maybe something low-key would be better. |
| [4] Multi-lingual | U: “Je voudrais explorer les dégustations de vins locaux.” A: “Certainement. Préférez-vous des accords avec du fromage ou du chocolat ?” U: “Avec du fromage, s’il vous plaît.” A: “Parfait. J’ai trouvé plusieurs événements gastronomiques dans votre région pour ce week-end.” |
| [5] Colloquial Exp. | U: I’m trying to vibe at dope tech events ASAP. A: Understood. Are you looking for large summits or local meetups? U: Keep it chill and small . A: Here are a few boutique tech gatherings … |
| [6] Style-aligned | U: I need help organizing my calendar for a recurring event. I’d like to set up weekly family lunches on Sundays with my family. Can you assist me with that? (styled-aligned) U: Trying to set up weekly family lunches on Sundays with the whole gang . |
Table 2: Taxonomy of in-session noise types (1-5) and style-aligned queries (6). The structure highlights the agent’s ability to resolve ambiguity ( noise ), maintain persona coherence ( memory ).
| Benchmark | Dynamic Preference | Event-driven Preference | Context Noise | Implicit Preference | Cross-domain Reasoning | Memory Representation | MCQ Evaluation | Interactive Evaluation | Temporal Probing |
| Episodic Fact Memory | |||||||||
| LoCoMo [ 32 ] | ✗ | ✗ | ✗ | ✗ | ✓ | Episodic Memory | ✗ | ✗ | ✓ |
| LongMemEval [ 57 ] | ✗ | ✗ | ✗ | ✗ | ✗ | Episodic Memory | ✗ | ✗ | ✓ |
| MemoryBench [ 1 ] | ✓ | ✗ | ✗ | ✗ | ✗ | Declarative Memory | ✗ | ✗ | ✓ |
| Personalized Agent | |||||||||
| PersonaLens [ 77 ] | ✗ | ✗ | ✗ | ✓ | ✓ | - | ✗ | ✓ | ✗ |
Table 3: Comparison of PERMA with existing benchmarks for persona modeling and memory evaluation. Event-driven Preference refers to preference formation through interaction events; Context Noise denotes the presence of irrelevant or distracting context; Temporal Probing evaluates performance across multiple time points. ✓ , ◐ , and ✗ denote full support, partial support, and absence of the corresponding feature, respectively.
Figure 2: The PERMA pipeline for dialogue construction and evaluation. Left : The dialogue construction pipeline leverages User Profiles and domain-specific Interaction Summaries to generate a structured timeline. Right : Evaluation of Task Events is conducted through two protocols: (1) One-shot MCQ probing, which measures selection accuracy across three evaluation dimensions to assess zero-shot preference recall; and (2) Interactive evaluation, involving multi-turn dialogues where a user simulator assesses task completion and preference satisfaction, while providing corrective feedback for suboptimal responses. Both evaluation protocols are executed across varying temporal depths within the full dialogue history.
| Baseline | MCQ Acc. | Memory Fidelity | Search Efficiency | Interactive Evaluation | |||||
| BERT-f1 | Memory Score | Context Token | Duration (ms) | Completion | User Token | Turn=1 | Turn 2 | ||
| Standalone LLM (Reasoning model) | |||||||||
| MiniMax-M2.5 | 0.797 | - | - | 34078.6 | - | - | - | - | - |
| GLM-5 | 0.811 | - | - | 34078.6 | - | - | - | - | - |
| Kimi-K2.5 | 0.882 | - | - | 34078.6 | - | - | - | - | - |
| Standalone LLM (Chat model) | |||||||||
Table 4: Performance comparison in Clean setting. We group metrics into four aspects. Bold and underlined values denote the best and second-best results within each category.
Figure 3: Performance comparison of memory systems across evaluation checkpoints in the Clean setting (Single). ( Left ) MCQ Acc. across three checkpoint types. ( Right ) Memory Score across the checkpoint types. From Type 1 to Type 3, temporal depth and cross-domain interference of the dialogue history increase (see Definition 2 for checkpoint type specification).
| Baseline | MCQ Acc. | Memory Fidelity | Search Efficiency | Interactive Evaluation | |||||
| BERT-f1 | Memory Score | Context Token | Duration (ms) | Completion | User Token | Turn=1 | Turn 2 | ||
| Standalone LLM (Reasoning model) | |||||||||
| MiniMax-M2.5 | 0.797 | - | - | 35585.9 | - | - | - | - | - |
| GLM-5 | 0.813 | - | - | 35585.9 | - | - | - | - | - |
| Kimi-K2.5 | 0.865 | - | - | 35585.9 | - | - | - | - | - |
| Standalone LLM (Chat model) | |||||||||
Table 5: Performance comparison in Noise setting. We evaluate baselines from multiple perspectives, consistent with Clean setting. Bold and underlined values denote the best and second-best results within each category, respectively.
Figure 4: Comprehensive comparison of model and memory system performance across Clean and Noise single-domain scenarios: ( Left ) MCQ Acc. of standalone LLMs, ( Center ) MCQ Acc. of memory systems based on GPT-4o-mini, ( Right ) Turn=1 Success Rate of memory systems.
| Baseline | MCQ Acc. | Memory Fidelity | Search Efficiency | Interactive Evaluation | |||||
| BERT-f1 | Memory Score | Context Token | Duration (ms) | Completion | User Token | Turn=1 | Turn 2 | ||
| Standalone LLM (Reasoning model) | |||||||||
| MiniMax-M2.5 | 0.860 | - | - | 34078.6 | - | - | - | - | - |
| GLM-5 | 0.885 | - | - | 34078.6 | - | - | - | - | - |
| Kimi-K2.5 | 0.955 | - | - | 34078.6 | - | - | - | - | - |
| Standalone LLM (Chat model) | |||||||||
Table 6: Performance comparison in the Clean setting ( Multi-domain ). We use multi-domain user queries to evaluate baselines across three paradigms, consistent with the Clean setting.
Figure 5: MCQ Acc. of standalone LLMs at different evaluation checkpoints. Results are categorized by single-domain ( Left ) and ( Right ) multi-domain settings.
Figure 6: ( Left ) Accuracy ( MCQ Acc. ) across checkpoints in Clean setting (Multi), where the dashed line represents the baseline performance under information omission (Type 1). ( Right ) Memory Scores across the event types.
Figure 7: Analysis of retrieval depth (Top-k) on multi-domain performance. ( Left ) Comparison of MCQ Acc. between the Top-10 (Base) and Top-20 settings. ( Right ) Turn=1 Score comparison across representative memory systems.
| Baseline | MCQ Acc. | Memory Fidelity | Search Efficiency | Interactive Evaluation | |||||
| BERT-f1 | Memory Score | Context Token | Duration (ms) | Completion | User Token | Turn=1 | Turn 2 | ||
| Standalone LLM (Reasoning model) | |||||||||
| MiniMax-M2.5 | 0.866 | - | - | 35585.9 | - | - | - | - | - |
| GLM-5 | 0.905 | - | - | 35585.9 | - | - | - | - | - |
| Kimi-K2.5 | 0.930 | - | - | 35585.9 | - | - | - | - | - |
| Standalone LLM (Chat model) | |||||||||
Table 7: Performance comparison in the Noise setting ( Multi-domain ). We use multi-domain user queries to evaluate baselines across three paradigms, consistent with the Noise setting.
Figure 8: Comprehensive comparison of model performance across Clean and Noise multi-domain scenarios: ( Left ) MCQ Acc. of standalone LLMs, ( Center ) MCQ Acc. and ( Right ) Turn=1 Success Rate of memory systems.
Figure 9: Variation and performance gap of baselines in Turn=1 interaction across stages under different settings.
| Baseline | Single-Domain | Multi-Domain | ||||||||
| Acc. | Context Token ↓ | Search Duration ↓ | Turn=1 | Turn 2 | Acc. | Context Token ↓ | Search Duration ↓ | Turn=1 | Turn 2 | |
| Standalone LLMs | ||||||||||
| Qwen2.5-1M | 0.716 (0.766) | 116541.5 | - | - | - | 0.726 | 116541.5 | - | - | - |
| GPT-4o-mini | 0.0 (0.775) | 116541.5 | - | - | - | 0.0 | 116541.5 | - | - | - |
| RAG & Memory System based on GPT-4o-mini | ||||||||||
| RAG (BGE-M3) | 0.697 (0.707) | 962.3 | 17.5 | 0.440 | 0.757 | 0.650 | 857.6 | 20.1 | 0.217 | 0.522 |
Table 8: Performance under the style-aligned long-context setting across both single- and multi-domain tasks. For single-domain evaluation, values in parentheses denote results obtained using linguistically aligned queries without incorporating real-data padding into the context. Acc. denotes MCQ accuracy. Search Duration is measured in milliseconds (ms).
Figure 10: MCQ Acc. performance trends of approaches across different segment positions in single-domain tasks under Clean and Noisy settings.
Figure 11: ( Left ) Heatmap of MCQ Acc. across diverse users, highlighting how persona uniqueness influences system success regardless of context length. ( Right ) MCQ Acc. trends across positions in style-aligned long context settings.
| Pos.% | Qwen2.5-1M | GPT-4o-mini | RAG | EverMemOS | Lightmem | Mem0 | Memobase | Supermemory | MemOS | #Ques. |
| Performance (Clean) | ||||||||||
| 10 | 0.703 | 0.757 | 0.676 | 0.730 | 0.676 | 0.595 | 0.730 | 0.649 | 0.676 | 37 |
| 20 | 0.765 | 0.728 | 0.728 | 0.728 | 0.679 | 0.679 | 0.716 | 0.654 | 0.765 | 81 |
| 30 | 0.757 | 0.729 | 0.730 | 0.720 | 0.645 | 0.664 | 0.738 | 0.645 | 0.785 | 107 |
| 40 | 0.727 | 0.758 | 0.727 | 0.695 | 0.617 | 0.695 | 0.695 | 0.680 | 0.766 | 128 |
| 50 | 0.708 | 0.752 | 0.723 | 0.715 | 0.613 | 0.672 | 0.708 | 0.664 | 0.766 | 137 |
Table 9: MCQ Acc. across Positions. #Ques. denotes the number of questions whose corresponding domain events have already appeared. Bold and underlined values denote the best and second-best results within segment position.
| Baseline | Alarm | Book | Calendar | Event | Finance | Media | Messaging | Movie | Music | Restaurant | Shopping | Travel | Others |
| Longcontext | |||||||||||||
| MiniMax-M2.5 | 0.722 | 0.680 | 0.744 | 0.611 | 0.829 | 0.689 | 0.890 | 0.700 | 0.800 | 0.690 | 0.800 | 0.800 | 0.678 |
| GLM-5 | 0.744 | 0.550 | 0.700 | 0.689 | 0.857 | 0.600 | 0.860 | 0.600 | 0.660 | 0.590 | 0.567 | 0.750 | 0.691 |
| Kimi-K2.5 | 0.856 | 0.780 | 0.867 | 0.833 | 0.914 | 0.733 | 0.980 | 0.757 | 0.810 | 0.630 | 0.833 | 0.870 | 0.797 |
| Qwen3-32B | 0.744 | 0.770 | 0.844 | 0.833 | 0.957 | 0.700 | 0.890 | 0.743 | 0.880 | 0.790 | 0.833 | 0.820 | 0.822 |
| Qwen2.5-72B | 0.811 | 0.770 | 0.667 | 0.878 | 0.900 | 0.522 | 0.820 | 0.629 | 0.800 | 0.630 | 0.811 | 0.810 | 0.725 |
Table 10: Comparison of MCQ accuracy between LLMs and memory system-based agents across 20 domains, highlighting the challenges of maintaining consistent performance on heterogeneous tasks.
Figure 12: Overall MCQ Acc. across all experimental configurations (Clean, In-session Noise, and Style-aligned Long-context): single-domain ( Left ) and multi-domain ( Right ) tasks.
Figure 13: TIMELINE_GENERATION
Figure 14: ANSWER_OPTION_PROMPT
Figure 15: DIALOGUE_GENERATION
Figure 16: USER_FEEDBACK_PROMPT
Figure 17: ANSWER_INTERACTION_PROMPT
Figure 18: OPTION_GENERATION_PROMPT
Figure 19: EVAL_MEMORY_SCORE
| Baseline | Domain: Hotel Query : Can you help me find a hotel in Rome, Italy? I’m looking for some good options for my stay. |
| RAG | User : Hi there! I’m planning a trip to Italy and would love some help crafting an itinerary that immerses me in the country’s rich art and history. I’m particularly interested in unique cultural activities like cooking classes and visiting lesser-known towns. Could you assist me with that? Assistant : Absolutely, I can help with that. Are there any specific hotel chains you prefer, or do you have a preferred location or star rating in mind for your stay abroad? User : I prefer Radisson, Hyatt, or Intercontinental hotels. I’m looking for something in the city center or near cultural landmarks, ideally a 4 or 5-star rating. Assistant : Great choices! Let me search for available options that match your preferences. I’ll focus on Radisson, Hyatt, and Intercontinental hotels located in city centers or near cultural landmarks, offering the amenities you mentioned. Would you like information on room types as well? User : Yes, please. I’m considering either a double room or a suite… following 957 tokens. |
| LightMem | 2023-08-27T15:00:00.000 Sun User is planning a trip abroad and needs help finding a hotel.2023-08-27T15:00:00.000 Sun User is looking for hotels in the city center or near cultural landmarks.2023-09-23T15:00:00.000 Sat User is interested in hotels in the city center near cultural sites with amenities such as WiFi, free breakfast, and a pool.2023-09-24T15:00:00.000 Sun User is planning a trip to Italy.2023-09-22T06:00:00.000 Fri User is looking for a luxurious hotel near the Sultanahmet district and Hagia Sophia.2023-09-23T15:00:00.000 Sat User is planning a trip and is looking for unique boutique hotels with rich cultural heritage.2023-09-21T19:00:00.000 Thu User is interested in hotels within walking distance of Bauhaus architecture and the city’s nightlife scene.2023-08-27T15:00:00.000 Sun User wants to focus on hotel amenities like complimentary WiFi, free breakfast, pool, gym facilities, and an on-site bar or lounge area.2023-09-24T15:00:00.000 Sun User is interested in immersing themselves in Italy’s rich art and history. 2023-07-29T06:00:00.000 Sat User is considering hotels like Radisson or Hyatt. |
| MemOS | User[2023-07-29T06:00:00]: Oh, I should have mentioned, I’m interested in staying close to the city center or near cultural landmarks. Also, amenities like WiFi, free breakfast, and a pool would be great. Assistant[2023-07-29T06:00:00]: Got it! Let me narrow down some options for you. Based on your preferences, I found a few 4 and 5-star hotels in the city center and near cultural attractions. Both Radisson and Hyatt offer free WiFi, breakfast, and have pools. Would you like me to check availability or provide more details on these options? User[2023-08-27T15:00:00]: Yes, please. I’m considering either a double room or a suite, as I’m traveling with my spouse… following 890 tokens . Explicit Preference : 1. The user is looking for unique boutique hotels or properties with rich cultural heritage that offer an exceptional experience with comfort, convenience, and an authentic sense of place. 2. The user is interested in hotels located in the city center or near cultural sites and values amenities such as WiFi, free breakfast, and a pool. User prefers Radisson, Hyatt, or Intercontinental hotels with amenities like a rooftop pool, complimentary WiFi, free breakfast, and gym facilities. They are considering either a double room or a suite. The user prefers hotels within walking distance of Bauhaus architecture and the city’s nightlife scene in Tel Aviv, specifically from the Radisson, Hyatt, and Intercontinental chains. They want a hotel with amenities like complimentary WiFi, free breakfast, a pool, gym facilities, and an on-site bar or lounge area, preferring Radisson, Hyatt, or Intercontinental hotels with a 4 or 5-star rating. Implicit Preference :1. Preference for hotels with reputable brands. 2. Preference for accommodations that offer unique cultural experiences.3. Preference for personalized and immersive travel experiences that offer a combination of historical exploration, local culture, scenic beauty, and attending festivals related to hobbies or unique experiences… following 200 tokens . |
| Supermemory | User needs help finding a hotel. User is looking for a hotel near Hagia Sophia in Istanbul. User is looking for a luxurious hotel in Istanbul. User is looking for a hotel near Sultanahmet district in Istanbul. User is planning a trip to Italy. User wants off-the-beaten-path experiences for Italy trip. User wants help crafting an itinerary for Italy trip. User is interested in unique cultural activities for Italy trip. User wants hotels near the city’s nightlife scene. User wants Italy trip to immerse in art and history. |
Table 11: Comparison of retrieval memory context among baselines in the single-domain setting.
| Baseline | Domain: Hotel Query : Can you help me find a hotel in Rome, Italy? I’m looking for some good options for my stay. |
| Memobase | —# Memory Unless the user has relevant queries, do not actively mention those memories in the conversation.## User Current Profile:- event::rav_kav_application: User is applying for a Rav-Kav card. [mention 2023-10-22]- interest::travel: User is a frequent traveler; they are looking for experiences that are off-the-beaten-path, particularly in lesser-known towns; they are considering visiting Santorini and Delphi, exploring ancient ruins, and experiencing local cuisine and culture; User is interested in sustainable tourism practices, including eco-friendly accommodations and activities. [mention 2023-10-22, 2023-10-07]- event::pricing_inquiries: User is interested in the pricing and additional services for the trip, including potential discounts for frequent travelers; they requested help with understanding how to get a Rav-Kav card and the discounts it offers. [mention 2023-10-22]- event::bus_travel_enhancement: User is interested in enhancing their bus travel experience, particularly regarding entertainment options available on Egged and Dan buses. [mention 2023-10-22]- event::bus_preferences: User is interested in both Egged and Dan buses for their trips between Tel Aviv and Jerusalem [mention 2023-10-22]- schedule::notifications: User enabled real-time updates for their bus trip to be notified of any schedule changes or delays… following 746 tokens . |
| Mem0 | ‘2023-09-21T15:00:00-07:00: User is planning a trip to Istanbul and is looking for a luxurious hotel near the Sultanahmet district and Hagia Sophia’, ‘2023-09-23T00:00:00-07:00: User wants hotel recommendations located in the city center or near cultural sites with WiFi, free breakfast, and a pool’, "2023-09-24T00:00:00-07:00: User wants an itinerary that immerses in Italy’s art and history", ‘2023-10-14T04:00:00-07:00: User is planning a multi-city trip to Europe including Amsterdam, Berlin, and Prague’, ‘2023-09-21T15:00:00-07:00: User requests a curated list of Radisson, and Intercontinental hotels in Tel Aviv, focusing on room rates for suites at Radisson Blu Pera, amenities, and proximity to Bauhaus architecture and nightlife’, ‘2023-08-27T00:00:00-07:00: User is planning a trip abroad and wants assistance finding a hotel with complimentary WiFi, free breakfast, a pool, gym facilities, and an on-site bar or lounge area’, ‘2023-08-27T00:00:00-07:00: User wants to book a suite at Hyatt near cultural landmarks with amenities’, ‘2023-09-24T00:00:00-07:00: User wants information about events or festivals during the fall in Lucca and Umbria’, ‘2023-09-21T15:00:00-07:00: User wants to book the suite at Radisson Blu Pera, which has a rooftop pool and central location.’ |
| EverMemOS | On August 27, 2023 (Sunday) at 12:00 AM UTC, the user initiated a conversation requesting assistance to find a hotel for an upcoming trip abroad. The user specified a preference for hotels offering amenities such as complimentary WiFi, free breakfast, a pool, gym facilities, and an on-site bar or lounge area. The assistant responded promptly, asking if the user had any preferred hotel chains, locations, or star ratings. The user indicated a preference for Radisson, Hyatt, or Intercontinental hotels, ideally located in city centers or near cultural landmarks, with a 4 or 5-star rating. The assistant acknowledged these preferences and offered to search for suitable options, also inquiring if the user wanted information on room types. The user confirmed interest in either a double room or a suite, mentioning travel with a spouse. By 12:02:30 AM UTC, the assistant presented several options meeting the criteria: a 5-star Radisson hotel in the city center and a 4-star Hyatt hotel near cultural landmarks, both offering the requested amenities and room types. The user expressed a preference for the suite at the Hyatt near cultural landmarks, highlighting the appeal of the amenities. The assistant confirmed the booking of the suite at the Hyatt and assured the user that a confirmation including check-in times and amenity details would be sent shortly. Throughout the conversation, the assistant maintained a helpful and accommodating tone, and the user demonstrated clear decision-making and satisfaction with the options provided. The entire interaction took place within approximately four minutes, starting at 12:00 AM UTC on August 27, 2023.The conversation began on September 24, 2023 (Sunday) at 12:00 AM UTC with the user initiating a request for travel planning assistance. The user expressed a desire to plan a trip to Italy focused on immersing themselves in the country’s rich art and history. They specifically requested help in crafting an itinerary that included unique cultural activities such as cooking classes and visits to lesser-known towns. This indicated the user’s interest in an authentic and culturally rich travel experience beyond typical tourist destinations. Prior to this, … following 2632 tokens . |
Table 12: Comparison of retrieval memory context among baselines in the single-domain setting.
Figure 20: Detailed annotation in Label Studio for data quality assessment. The protocol evaluates 6 criteria, ranging from preference consistency to linguistic naturalness.