TAMEing the Open-World Personalization: Towards Open-Set Personalized MLLM Assistant
Organizations: University of Electronic Science and Technology of China · Intelligent Digital Media Technology Key Laboratory of Sichuan Province
Abstract
Personalizing Multimodal Large Language Models (MLLMs) as intelligent assistants has recently emerged as an active research topic, where an MLLM is expected to go beyond generic, one-size-fits-all replies, and instead generate responses grounded in user-specific objects, entities, and concepts encountered during interactions. Nevertheless, existing studies mainly focus on personalizing MLLMs on a static, predefined set of personalized concepts and require users to manually and frequently extend this set to accommodate newly emerging concepts, which is infeasible in dynamic-world scenarios. To address this limitation, we recast conventional MLLM personalization as an Open-Set problem, and propose TAME-O, the first long-context open-set personalized MLLM assistant. Specifically, TAME-O introduces a training-free, plug-and-play Concept Reconciliation Skill (CR Skill). It enables the assistant to automatically unlock novel concepts during prolonged human-MLLM interactions while preventing confusion with previously unlocked ones, allowing the MLLM to continually grasp more and more concepts and progressively refine the modeling of known ones. Extensive experiments under the open-set personalization setting underscore the efficacy of our method.
Figures & tables
| Methods | MLLM Backbone | Training Free | State Aware | Open Set | LCMP-H | LCMP-O | ||||||||
| ACC-C | ACC-F | SPR-L | SPR-S | CRA | ACC-C | ACC-F | SPR-L | SPR-S | CRA | |||||
| Backbone | InternVL3.5-8B | ✓ | ✗ | ✗ | 40.00 | 20.00 | 12.78 | 26.11 | — | 32.00 | 18.67 | 16.44 | 7.11 | — |
| Qwen3.5-9B | 36.11 | 20.56 | 17.22 | 32.78 | — | 38.67 | 22.22 | 29.78 | 14.67 | — | ||||
| MyVLM | LLaVA-v1.6-7B | ✗ | ✗ | ✗ | 40.00 | 13.33 | 8.89 | 12.22 | — | 46.67 | 21.33 | 18.22 | 6.67 | — |
| Yo’LLaVA | LLaVA-v1.5-13B | ✗ | ✗ | ✗ | 38.89 | 9.44 | 8.89 | 17.22 | — | 56.44 | 6.67 | 20.89 | 9.78 | — |
| RAP | LLaVA-v1.5-13B | ✗ | ✗ | ✗ | 41.11 | 4.44 | 33.33 | 23.89 | — | 34.22 | 20.89 | 19.11 | 17.33 | — |
| LCMP-H | LCMP-O | ||||||||||
| Module | Variant | ACC-C | ACC-F | SPR-L | SPR-S | CRA | ACC-C | ACC-F | SPR-L | SPR-S | CRA |
| CR Skill | w/o TAD Memory | 77.78 | 57.22 | 60.56 | 68.89 | 69.67 | 70.67 | 48.44 | 68.44 | 53.33 | 66.59 |
| w/o MLLM Decision | 72.78 | 48.89 | 40.00 | 52.22 | 51.13 | 46.22 | 37.78 | 52.44 | 29.33 | 46.33 | |
| w/o CR Skill | 66.11 | 38.33 | 24.44 | 45.56 | 36.84 | 52.44 | 34.67 | 50.67 | 29.33 | 33.41 | |
| PCF Strategy | w/o PCF Strategy | 78.89 | 59.44 | 61.11 | 67.22 | 69.92 | 73.78 | 47.56 | 70.22 | 54.67 | 68.15 |
| TAME-O | ALL | 82.22 | 61.67 | 64.44 | 70.00 | 72.18 | 75.11 | 52.89 | 72.00 | 59.11 | 71.49 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Parameter | |||||||||
| LCMP-H | 0 | 0.1 | 1 | 2 | 5 | 1 | 2 | 3 | 4 | 5 |
| ACC-C (%) | 78.33 | 80.56 | 82.22 | 82.44 | 78.67 | 74.44 | 82.78 | 82.22 | 81.11 | 80.00 |
| Parameter | Parameter | |||||||||
| LCMP-O | 0 | 0.1 | 1 | 2 | 5 | 1 | 2 | 3 | 4 | 5 |
| ACC-C (%) | 72.89 | 75.89 | 75.11 | 75.44 | 76.11 | 71.44 | 74.17 | 75.11 | 75.67 | 75.18 |
| LCMP-H | LCMP-O | |||||||||
| Time Step (t) | 50 | 100 | 200 | 300 | 400 | 50 | 100 | 200 | 300 | 400 |
| CFA (%) | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 94.74 | 95.65 | 96.67 |
| Methods | Open-Set Method | LCMP-H | LCMP-O | ||||||||
| ACC-C | ACC-F | SPR-L | SPR-S | CRA | ACC-C | ACC-F | SPR-L | SPR-S | CRA | ||
| PeKit | TAME † | 41.67 | 26.67 | 18.33 | 23.33 | 24.56 | 38.22 | 22.22 | 16.00 | 10.22 | 19.82 |
| TAME-O | 48.33 | 45.56 | 28.89 | 37.22 | 49.62 | 48.00 | 26.22 | 20.00 | 12.44 | 48.33 | |
| R2P | TAME † | 36.11 | 26.11 | 26.11 | 23.33 | 25.56 | 32.44 | 21.33 | 16.00 | 28.44 | 23.16 |
| TAME-O | 42.22 | 31.67 | 36.67 | 35.00 | 54.14 | 50.22 | 34.22 | 17.33 | 30.22 | 52.34 | |
LLM Personalization
LLM personalization adapts language models to users’ preferences, histories, expertise, and changing goals while balancing usefulness, privacy, and control.
See all 135 papers