Personalizing Multimodal Large Language Models (MLLMs) as intelligent assistants has recently emerged as an active research topic, where an MLLM is expected to go beyond generic, one-size-fits-all replies, and instead generate responses grounded in user-specific objects, entities, and concepts encountered during interactions. Nevertheless, existing studies mainly focus on personalizing MLLMs on a static, predefined set of personalized concepts and require users to manually and frequently extend this set to accommodate newly emerging concepts, which is infeasible in dynamic-world scenarios. To address this limitation, we recast conventional MLLM personalization as an Open-Set problem, and propose TAME-O, the first long-context open-set personalized MLLM assistant. Specifically, TAME-O introduces a training-free, plug-and-play Concept Reconciliation Skill (CR Skill). It enables the assistant to automatically unlock novel concepts during prolonged human-MLLM interactions while preventing confusion with previously unlocked ones, allowing the MLLM to continually grasp more and more concepts and progressively refine the modeling of known ones. Extensive experiments under the open-set personalization setting underscore the efficacy of our method.
Figures & tables
Figure 1: Paradigm comparison between (a) Closed-Set Personalization, where the MLLM passively acquires novel concepts from user provision, and (b) Open-Set Personalization, where the MLLM actively grasps new concepts from everyday interactions.
Figure 2: Paradigm of TAME-O. When receiving a new dialogue, the CR skill first retrieves candidate personalized concepts from the TAD Memory in a temporally weighted manner. The retrieved candidates are then provided to the MLLM, together with the user query and recent dialogue context, to determine whether the referenced concept is existing or novel, achieving open-set personalization.
Figure 3: A concept fission example for concept Cat 1 .
Methods
MLLM Backbone
Training Free
State Aware
Open Set
LCMP-H
LCMP-O
ACC-C
ACC-F
SPR-L
SPR-S
CRA
ACC-C
ACC-F
SPR-L
SPR-S
CRA
Backbone
InternVL3.5-8B
✓
✗
✗
40.00
20.00
12.78
26.11
—
32.00
18.67
16.44
7.11
—
Qwen3.5-9B
36.11
20.56
17.22
32.78
—
38.67
22.22
29.78
14.67
—
MyVLM
LLaVA-v1.6-7B
✗
✗
✗
40.00
13.33
8.89
12.22
—
46.67
21.33
18.22
6.67
—
Yo’LLaVA
LLaVA-v1.5-13B
✗
✗
✗
38.89
9.44
8.89
17.22
—
56.44
6.67
20.89
9.78
—
RAP
LLaVA-v1.5-13B
✗
✗
✗
41.11
4.44
33.33
23.89
—
34.22
20.89
19.11
17.33
—
Table 1: Performance comparison on the LCMP-H and LCMP-O datasets. The best results are in black bold . Higher values of all metrics indicate better performance. TAME † is the modified TAME.
Figure 4: Performance comparison between TAME † and TAME-O with respect to a continually increasing number of concepts. ■ denotes TAME † , while ▲ denotes TAME-O.
LCMP-H
LCMP-O
Module
Variant
ACC-C
ACC-F
SPR-L
SPR-S
CRA
ACC-C
ACC-F
SPR-L
SPR-S
CRA
CR Skill
w/o TAD Memory
77.78
57.22
60.56
68.89
69.67
70.67
48.44
68.44
53.33
66.59
w/o MLLM Decision
72.78
48.89
40.00
52.22
51.13
46.22
37.78
52.44
29.33
46.33
w/o CR Skill
66.11
38.33
24.44
45.56
36.84
52.44
34.67
50.67
29.33
33.41
PCF Strategy
w/o PCF Strategy
78.89
59.44
61.11
67.22
69.92
73.78
47.56
70.22
54.67
68.15
TAME-O
ALL
82.22
61.67
64.44
70.00
72.18
75.11
52.89
72.00
59.11
71.49
Table 2: Ablation study on core components in TAME-O with Qwen3.5-9B.
Figure 5: Comparison of CFR among TAME † , TAME-O w/o PCF Strategy, and TAME-O.
Figure 6: Performance on LCMP-O with concepts stratified by confusability. ACC and SPR average ACC-C/ACC-F and SPR-L/SPR-S, respectively. ■ denotes TAME † , while ▲ denotes TAME-O.
Figure 7: Performance (mean ± std) across five concept appearance orders. Numbers above the bars are the gains over TAME † . denotes TAME † , while denotes TAME-O.
Figure 8: Case study on resolving an existing concept (left) and a novel concept (right).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Cases of personalized concepts from LCMP benchmark.
Parameter λ
Parameter K
LCMP-H
0
0.1
1
2
5
1
2
3
4
5
ACC-C (%)
78.33
80.56
82.22
82.44
78.67
74.44
82.78
82.22
81.11
80.00
Parameter λ
Parameter K
LCMP-O
0
0.1
1
2
5
1
2
3
4
5
ACC-C (%)
72.89
75.89
75.11
75.44
76.11
71.44
74.17
75.11
75.67
75.18
Appendix
Table 3: Performance of TAME-O under varying values of hyper-parameters.
LCMP-H
LCMP-O
Time Step (t)
50
100
200
300
400
50
100
200
300
400
CFA (%)
100.00
100.00
100.00
100.00
100.00
100.00
100.00
94.74
95.65
96.67
Appendix
Table 4: CFA of TAME-O over successive time steps on different datasets.
Methods
Open-Set Method
LCMP-H
LCMP-O
ACC-C
ACC-F
SPR-L
SPR-S
CRA
ACC-C
ACC-F
SPR-L
SPR-S
CRA
PeKit
TAME †
41.67
26.67
18.33
23.33
24.56
38.22
22.22
16.00
10.22
19.82
TAME-O
48.33
45.56
28.89
37.22
49.62
48.00
26.22
20.00
12.44
48.33
R2P
TAME †
36.11
26.11
26.11
23.33
25.56
32.44
21.33
16.00
28.44
23.16
TAME-O
42.22
31.67
36.67
35.00
54.14
50.22
34.22
17.33
30.22
52.34
Appendix
Table 5: Performance comparison on the LCMP-H and LCMP-O datasets. Higher values indicate better performance.
Figure 10: Case study: the procedure of TAME-O in resolving an existing old concept (left) and a novel concept (right).
Figure 11: Case study: two concepts that have similar attributes are selected to be fused into one concept.
Figure 12: Case study: incorrect concept identification leads to the fission of an already grasped concept.
Figure 18Figure 19
Figure 13: Cases of personalized concepts from LCMP-O benchmark.
LLM personalization adapts language models to users’ preferences, histories, expertise, and changing goals while balancing usefulness, privacy, and control.
Multimodal Large Language Models (MLLMs) serve as daily assistants for millions. However, their ability to generate responses aligned with individual preferences remains limited. Prior approaches enable only static, single-turn personalization through input augmentation or output alignment, and thus fail to capture users' evolving preferences and personality over time (see Fig.1). In this paper, we introduce PersonaVLM, an innovative personalized multimodal agent framework designed for long-term personalization. It transforms a general-purpose MLLM into a personalized assistant by integrating three key capabilities: (a) Remembering: It proactively extracts and summarizes chronological multimodal memories from interactions, consolidating them into a personalized database. (b) Reasoning: It conducts multi-turn reasoning by retrieving and integrating relevant memories from the database. (c) Response Alignment: It infers the user's evolving personality throughout long-term interactions to ensure outputs remain aligned with their unique characteristics. For evaluation, we establish Persona-MME, a comprehensive benchmark comprising over 2,000 curated interaction cases, designed to assess long-term MLLM personalization across seven key aspects and 14 fine-grained tasks. Extensive experiments validate our method's effectiveness, improving the baseline by 22.4% (Persona-MME) and 9.8% (PERSONAMEM) under a 128k context, while outperforming GPT-4o by 5.2% and 2.0%, respectively. Project page: https://PersonaVLM.github.io.
Personalizing Multimodal Large Language Models (MLLMs) aims to recognize users' unique concepts from visual data and provide personalized responses. Although prior work has shown the benefit of concept descriptions and reasoning for this task, MLLM descriptions often include information, such as state and context, that does not help and may in fact hinder the unique identification of the target concept among other visually similar items. Effective descriptions of personal concepts should instead be accurate, discriminative, and free of distracting details. To achieve such descriptions, we introduce Reinforced Reference Game (RRG), a learning framework that promotes discriminative descriptions through a novel reinforced multimodal reference game. The MLLM plays both the roles of speaker and listener in a contrastive game setting, whose goal is to effectively communicate discriminative information about a target concept. Our approach formulates a verifiable contrastive reward over hard positives (dissimilar views of the same concept) and hard negatives (visually similar but different concepts). Empirically, RRG achieves state-of-the-art across multiple tasks on three personalization benchmarks. RRG generalizes to unseen domains and outperforms existing methods based on concept descriptions and personalization-specific RL frameworks. We will release code and models in the project page.
Deepayan Das, Davide Talon, Yiming Wang +2
University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy
Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.
Yizhuo Chen, Xin Liu, Ruijie Wang +7
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame