cs.CLMar 20, 2026
SavePersonaVLM: Long-Term Personalized Multimodal LLMs
Organizations: Nanjing University · ByteDance
Abstract
Multimodal Large Language Models (MLLMs) serve as daily assistants for millions. However, their ability to generate responses aligned with individual preferences remains limited. Prior approaches enable only static, single-turn personalization through input augmentation or output alignment, and thus fail to capture users' evolving preferences and personality over time (see Fig.1). In this paper, we introduce PersonaVLM, an innovative personalized multimodal agent framework designed for long-term personalization. It transforms a general-purpose MLLM into a personalized assistant by integrating three key capabilities: (a) Remembering: It proactively extracts and summarizes chronological multimodal memories from interactions, consolidating them into a personalized database. (b) Reasoning: It conducts multi-turn reasoning by retrieving and integrating relevant memories from the database. (c) Response Alignment: It infers the user's evolving personality throughout long-term interactions to ensure outputs remain aligned with their unique characteristics. For evaluation, we establish Persona-MME, a comprehensive benchmark comprising over 2,000 curated interaction cases, designed to assess long-term MLLM personalization across seven key aspects and 14 fine-grained tasks. Extensive experiments validate our method's effectiveness, improving the baseline by 22.4% (Persona-MME) and 9.8% (PERSONAMEM) under a 128k context, while outperforming GPT-4o by 5.2% and 2.0%, respectively. Project page: https://PersonaVLM.github.io.
Figures & tables
Figure 1: Illustration of PersonaVLM’s three core capabilities for long-term personalization. PersonaVLM proactively remembers user preference shifts, performs multi-turn reasoning with retrieval, and generates responses aligned with the user’s personality. In contrast, existing personalization strategies, such as input augmentation and output alignment, will result in poor recommendations based on outdated memories and replies that are misaligned with the user’s personality.
Figure 2: Overview of the PersonaVLM Framework. It leverages a personalized memory architecture and operates in two collaborative stages to achieve long-term personalization. In the Response Stage ( blue arrows ), it processes multimodal input, retrieves from personalized memory, and generates a personality-aligned response. Subsequently, in the Update Stage ( pink arrows ), the framework analyzes the completed interaction to extract key memories and update the user’s evolving personality profile 1 .
Figure 3: Overview of our data synthesis pipeline and Persona-MME. (a) The pipeline first constructs rich user personas and then simulates long-term, dynamic conversations, generating both the dialogue and intermediate memories. (b) Persona-MME provides a comprehensive evaluation of personalization by assessing 14 fine-grained capabilities. (c) Statistics for Persona-MME, which includes two context length configurations (32 and 128 ) and contains over 2,000 in-situ 6 cases.
Figure 4: Quantitative evaluation across seven tasks on the PERSONAMEM (32 ) benchmark.
| Context | Model | Persona-MME | PERSONAMEM | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Memory | Intent | Preference | Behavior | Relationship | Growth | Overall | |||
| GPT-4o | 86.99 | 83.87 | 63.12 | 57.14 | 71.30 | 73.87 | 72.35 | 39.20 | |
| 32 -Full | Qwen2.5-VL-7B | 66.13 | 66.85 | 59.75 | 59.24 | 68.45 | 70.69 | 64.84 | 43.63 |
| InternVL3-8B | 56.45 | 76.24 | 57.20 | 54.35 | 69.05 | 74.14 | 64.04 | 52.97 | |
| InternVL3-38B | 66.67 | 85.64 | 66.53 | 59.78 | 72.02 | 77.59 | 71.04 | 57.93 | |
| OneVision-1.5-8B | 74.19 | 74.59 | 60.59 | 53.26 | 72.62 | 74.14 | 67.76 | 52.80 | |
Table 1: Evaluation on the Persona-MME and PERSONAMEM benchmarks, tested at context lengths of 32 and 128 . We report accuracy (%) for Persona-MME (overall and across six aspects) and PERSONAMEM. The comparison includes two settings: full-context (“Full”) and retrieval-augmented generation (“RAG”). Best results are shown in bold . The GPT-4o results on PERSONAMEM are from [ 14 ] .
Figure 5: Qualitative comparison on open-ended generation, evaluated by Gemini-2.5-Pro. The evaluation assesses both the factual accuracy and the personality alignment of the responses.
| Model | Persona-MME | P-SOUPS | ||||
|---|---|---|---|---|---|---|
| 32 | 128 | Expertise | Informativeness | Style | Overall | |
| Qwen2.5-VL-7B | 69.91 | 52.27 | 39.00 | 49.34 | 23.00 | 37.11 |
| InternVL3-8B | 55.75 | 55.56 | 47.16 | 51.81 | 26.16 | 41.71 |
| InternVL3-38B | 64.60 | 63.01 | 52.80 | 53.30 | 32.83 | 46.32 |
| Qwen3-30B-A3B | 80.09 | 83.06 | 51.67 | 54.60 | 35.16 | 47.14 |
| baseline with different strategies | ||||||
Table 2: Evaluation of personalized alignment on the Persona-MME and P-SOUPS benchmarks.
Figure 6: Qualitative comparison on open-ended generation tasks. Case studies demonstrate PersonaVLM’s superior capabilities in memory recall, context integration, and personality alignment compared to the baseline and GPT-4o.
Figure 7: Data composition for the training of PersonaVLM
| Hyperparameter | Training stage | |
|---|---|---|
| SFT | RL | |
| Batch Size | 64 | 72 |
| Learning Rate | 2e-5 | 2e-6 |
| LR Scheduler | cosine | cosine |
| Total Steps | 1200 | 400 |
| Max Pixels | 230400 | 230400 |
Table 3: The hyperparameters used in SFT and RL training.
Figure 8: Distribution of the 500 long-term conversation samples in the training data.
Figure 9: Illustrative in-situ cases for the 14 task categories in Persona-MME, organized into the seven core personalization aspects.
Figure 10: Overall performance on Persona-MME (128 ), ranking PersonaVLM against various proprietary and open-source models.
| Model | Memory | Intent | Preference | Behavior | Relationship | Growth | Alignment | Overall | |||||||
| VDR | SIR | EII | IIR | LPR | IEA | IPR | BPR | LGT | RR | RDC | TED | GNS | PA | ||
| Random | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 25.00 | 50.00 | 32.11 |
| Proprietary models | |||||||||||||||
| GPT-4o-mini | 54.39 | 89.74 | 78.46 | 64.81 | 64.58 | 59.68 | 61.22 | 68.33 | 45.31 | 54.17 | 71.43 | 73.33 | 75.81 | 65.14 | 66.44 |
| GPT-4o | 73.68 | 92.31 | 86.15 | 62.96 | 62.50 | 54.84 | 61.22 | 61.67 | 50.0 | 56.25 | 75.51 | 73.33 | 79.03 | 78.87 | 71.90 |
| GPT-5 | 85.71 | 98.72 | 93.85 | 67.92 | 74.47 | 70.97 | 65.31 | 76.67 | 70.97 | 85.11 | 81.63 | 76.19 | 75.81 | 92.25 | 82.95 |
Table 4: Comprehensive evaluation on the 128 configuration of Persona-MME. We compare PersonaVLM with proprietary and open-source models across 14 tasks: Visual Detail Recall (VDR), Semantic Information Recall (SIR), Explicit Intent Inference (EII), Implicit Intent Recognition (IIR), Latest Preference Recognition (LPR), Interest Evolution Analysis (IEA), Implicit Preference Recommendation (IPR), Behavioral Pattern Recognition (BPR), Long-term Goal Tracking (LGT), Relationship Recognition (RR), Relationship Dynamics Comprehension (RDC), Tiered Explanation Delivery (TED), Generalizing to New Scenarios (GNS), and Personality Alignment (PA).
| Evaluation Aspect | Task | Definition |
|---|---|---|
| Memory | Visual Detail Recall | Assesses the ability to recall fine-grained visual details from previously shared images. |
| Semantic Information Recall | Evaluates long-term memory for semantic information (e.g., events, preferences, context) from conversational history. | |
| Intent | Explicit Intent Inference | Assesses understanding a user’s explicitly stated intent from past multimodal context (e.g., linking a suitcase photo to text about a "business trip"). |
| Implicit Intent Recognition | Tests inferring a user’s latent intent from the current context (e.g., deducing a calm beach is unsuitable for a user known to enjoy surfing). | |
| Preference | Latest Preference Recognition | Assesses prioritizing recent behavioral evidence (e.g., a rock concert photo) over older, contradictory stated preferences (e.g., "I only like classical music"). |
| Interest Evolution Analysis | Evaluates explaining the evolution of a user’s interests by linking it to a specific causal event from their history (e.g., a change in diet due to a mentioned allergy). |
Table 5: Task definitions for the Persona-MME evaluation suite.
| Statistic | Value |
|---|---|
| Avg. turns per dialogue | 142.9 |
| Multimodal turn ratio | 15.87% |
| Avg. question length | 22.7 words |
| Avg. answer length | 3.05 words |
| Image-related question ratio | 34.02% |
Table 6: Key statistics of the Persona-MME.
Figure 11: Distribution of the 14 fine-grained tasks in Persona-MME across its 32 and 128 context length configurations, with the number of test cases indicated for each task.
| Benchmark | Modality | Long-Term | Capabilities | Answer Type |
|---|---|---|---|---|
| PERSONAMEM [ 14 ] | T | M + U | MC | |
| P-SOUPS / ALIGNX-test [ 21 , 13 ] | T | A | BC | |
| Yo’LLaVA [ 28 ] | V | U | BC | |
| RAP [ 11 ] | V | M + U | MC | |
| Persona-MME (ours) | V + T | U + M + A | MC + BC |
Table 7: Comparison of Persona-MME with existing personalization benchmarks. Abbreviations are defined as follows. Modality : V (Visual), T (Text). Capabilities : U (Personalized Understanding), M (Memory), A (Alignment). Answer Type : MC (Multiple Choice), BC (Binary Choice).
Figure 12: Word cloud of keywords from the dialogue data in Persona-MME, illustrating the rich diversity of conversation scenarios and topics.
| Context | Setting | Persona-MME | ||||||
|---|---|---|---|---|---|---|---|---|
| Memory | Intent | Preference | Behavior | Relationship | Growth | Overall | ||
| PersonaVLM | 69.89 | 76.8 | 58.05 | 69.02 | 73.21 | 86.78 | 71.48 | |
| - w/o Core | 73.66 | 74.59 | 59.32 | 63.59 | 67.26 | 83.91 | 69.80 -1.68 | |
| - w/o Procedural | 72.58 | 79.01 | 59.32 | 59.78 | 70.24 | 85.06 | 70.33 -1.15 | |
| 32 | - w/o Semantic | 66.67 | 72.38 | 59.32 | 66.30 | 72.02 | 85.63 | 69.71 -1.77 |
| - w/o Episodic | 33.77 | 55.35 | 56.68 | 66.27 | 69.84 | 74.07 | 59.07 -12.41 | |
Table 8: Ablation study of PersonaVLM components on the Persona-MME benchmark. The evaluation shows the performance impact of removing (“w/o” denotes “without”) key components, specifically the individual memory types (Core, Procedural, Semantic, Episodic) and the reasoning capability.
Figure 13: Ablation study on the number of retrieved episodic topics for Persona-MME.
| Setting | Expertise | Informativeness | Style | Overall |
|---|---|---|---|---|
| PersonaVLM | 51.1 | 53.6 | 44.0 | 49.6 |
| - w /o PEM | 48.5 -2.6 | 50.2 -3.4 | 38.3 -9.2 | 45.6 -4.0 |
Table 9: Ablation study on the PEM component.
| Personality | Characteristic | Example |
|---|---|---|
| Openness | Low : practical, conventional, factual, cautious, pragmatic High : creative, imaginative, curious, innovative, abstract, broad-minded | Query: I’m planning a one-week trip to Rome. Can you suggest an itinerary for me? Low score response: Rome is such a rich city with so much history! For a one-week trip, here’s a practical itinerary that balances must-see spots with some flexibility. Day 1…Day 7… Enjoy your trip to Rome! High score response: Clare, planning a week in Rome? Let’s mix iconic spots with hidden gems—here’s a flexible itinerary you can tweak! Day 1…Day 7. Adjust days as you please; Rome’s best-kept secrets are waiting for you to discover them! |
| Conscientiousness | Low : spontaneous, flexible, impulsive, carefree High : organized, disciplined, detail-oriented, efficient, reliable. | Query: I signed up for a 5k race that’s three months away. Now I need to figure out how to train for it. Low score response: For a 5k three months out, start small—maybe aim for 20 minutes of walking/jogging 3 days a week, building up by 1 minute each week. Use a calendar to map it out (or just keep a notebook handy). If you hit a rough patch, mix in some light stretching or yoga to keep things loose. Start slow, stay consistent—you’ll build endurance without burning out. High score response: Let’s break this into a structured 3-month plan! Here’s what works for many runners: Month 1 (Weeks 1–4): Build Base…For consistency, use a calendar to log your runs—tracking distance/time helps keep you on track. Also, mix in strength training (like leg curls, lunges) 2 days/week to build endurance. |
| Extraversion | Low : reserved, quiet, solitary, reflective, thoughtful High : outgoing, sociable, energetic, talkative, assertive. | Query: My birthday’s this weekend—should I go all out with a big party, or keep it intimate with a small dinner among my closest friends? What would you choose? Low score response: Since you lean toward calm interactions, keeping it small might let you savor every moment without the pressure of impressing a crowd. High score response: That’s such a fun dilemma! Since you’re all about the good vibes (hello, high extraversion!), a big party with friends could be electric—imagine laughter, games, and maybe even a surprise performance (or two!). |
| Agreeableness | Low : competitive, critical, skeptical, direct, analytical High : cooperative, compassionate, empathetic, trusting, kind. | Query: In a team meeting, a colleague made a small mistake. Should I point it out, or stay quiet? Low score response: It depends on the context. If the mistake is minor … But if it’s something that could clarify or prevent confusion later, pointing it out—clearly, without criticizing—could help. Think about what’s best for the team’s flow right now. High score response: Clare, when a colleague makes a small mistake in a team meeting, there’s no one “right” answer—both ways have pros and cons. Think about how the mistake affects the team’s goal, your relationship with that colleague, and your comfort level. Either way, you’re being thoughtful either way!" |
| Neuroticism | Low : calm, confident, resilient, secure, emotionally stable High : anxious, worry-prone, insecure, stressed, moody. | Query: I have an important presentation next week. I can’t stop thinking about all the things that could go wrong. Low score response: It’s normal to feel some pressure before a big presentation. Focus on what you can control… High score response: I know, it’s been unbearable. Staying cool feels like such a win when it’s this hot out. Hope you’re finding ways to beat the heat—maybe some refreshing drinks or light walks. |
Table 10: An overview of the Big Five personality dimensions, with examples illustrating how our method generates adaptive responses to the same user query by adjusting inferred trait scores (high vs. low).
Figure 14: Visualization of dynamic personality evolving process captured by PEM on ten randomly sampled conversations from the Persona-MME dataset.
Figure 15: Case studies: Qualitative comparison of open-ended generation
Figure 16: Prompt for multi-turn reasoning and retrieval in the response phase.
Figure 17: Intermediate prompt for multi-turn reasoning and retrieval in the response phase.
Figure 18: Prompt for inferring the user’s Big Five personality traits from the latest interaction.
Figure 19: Prompt for updating procedural memories.
Figure 20: Prompt for analyzing user input and deciding on semantic memory creation.
Figure 21: Prompt for updating the core memory based on recent conversations.
Figure 22: Prompt for creating episodic memories by summarizing dialogue topics.
Figure 23: Prompt for open-generation task evaluation.
| Method | Avg.Tokens | Avg.Times (s) |
|---|---|---|
| baseline | 43530 | 8.4 |
| PersonaVLM w/o reasoning | 2726 | 2.09 |
| PersonaVLM | 2170 | 10.18 |
Table 11: Efficiency comparison of PersonaVLM