People often make decisions by holistically reasoning over heterogeneous forms of personal information. In this paper, we study the relatively underexplored capability of multimodal personalized reasoning in multimodal large language models (MLLMs). To this end, we introduce MPRBench, a novel and comprehensive benchmark specifically designed to evaluate personalized reasoning across a wide range of tasks and real-world challenges. MPRBench consists of 12 sub-tasks and additionally supports interpretable analysis of representative error cases. Through extensive experiments, we find that current personalized MLLMs still struggle with personalized reasoning due to several key challenges, including inaccurate recognition of personalized concepts and sensitivity to irrelevant personal information. We hope that MPRBench will stimulate further research on personalized reasoning in MLLMs. Code and data are available at https://github.com/kkdh15/MPRBench
Figures & tables
Figure 1: Conceptual Overview and Problem Setup for Multimodal Personalized Reasoning.
Figure 2: Illustrative Examples and Statistics of MPRBench.
Figure 3: Overview of the Data Construction Pipeline for MPRBench.
Vision-specific Reasoning
Text-specific Reasoning
Multimodal
Method
predict
spatial
numeric
commsense
change
V-Avg
predict
numeric
multi-hop
T-Avg
predict
sentiment
numeric
history
M-Avg
Avg
Random Choice
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
Proprietary+Prompting
GPT-4o [ 1 ] +P
91.6
64.3
63.5
71.6
65.0
70.6
97.5
70.1
40.3
62.5
88.1
79.4
50.0
49.4
73.7
69.4
GPT-5-mini [ 42 ] +P
90.3
57.4
83.0
66.7
91.9
79.9
95.8
90.6
48.4
70.7
77.8
69.0
58.0
92.4
74.4
75.9
GPT-5 [ 42 ] +P
88.4
66.1
81.2
72.6
87.8
80.3
98.3
95.8
77.4
87.2
88.1
79.4
64.0
94.9
84.1
83.1
Table 1: Evaluation on proposed MPRBench.
Figure 4: Diagnostic Analysis of Multimodal Reasoning Paths
Figure 5: Distractor Injection Points in the Input Structure.
Figure 6: Analysis on Failure Cases.
Figure 7: Analysis on Cross-modal Distractor.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Source
# Concepts
Person
Yo’LLaVA [ 30 ]
7
MC-LLaVA [ 3 ]
9
MMPB [ 19 ]
6
CustomConcept101 [ 22 ]
3
SoccerAgent [ 37 ]
22
Web
22
Appendix
Table 2: Distribution of personalized concepts in MPRBench across concept categories and source datasets.
Task Category
Sub-task
# Questions
Visual-Specific
Prediction
155
Spatial
115
Numerical
271
Commonsense
102
Visual Appearance Change
123
Sub-total (Visual)
766
Appendix
Table 3: Detailed statistics of MPRBench across different task categories.
Category
Attribute
Definitions
Visually-grounded
gender
Gender of Person Concept which is strongly correlated with the concept’s appearance in reference images.
height
Height of Person Concept which inferred from visual context and diversity-oriented rules.
Diversity-oriented
name
Name of Person Concept by using unique name list to avoid duplication.
birthday
Birthday of Person Concept randomly generated based on estimated age to ensure temporal diversity.
family
Family Information about Person Concept which includes Family Relation with its birthday each. Randomly generated based on Person Concept’s birthday and Replationship.
Context-aware
personality
Personality of Person Concept generated based on visual queries and fixed attributes.
Appendix
Table 4: Full List and Definitions of Attributes in the Textual Reference for Person Concepts.
Category
Attribute
Definitions
Identification
name
Name of Object Concept assigned by a unique name list to avoid duplication.
Diversity-oriented
acquisition_date
Acquisition date of Object Concept.
manufacturing_date
Manufacturing date of Object Concept which is always before acquisition date.
owner
Name of Object Concept’s Owner which assigned Person Concept Name for further extensibility .
Context-aware
object_type
Type of Object Concept.
acquisition_process
Acquisition process of Object Concept.
Appendix
Table 5: Full List and Definitions of Attributes in the Textual Reference for Object Concepts.
Attribute
Validation criterion
N
Agr.
Gender
match w/ annotators
50
100.0
Age
within ± 10-year range
50
96.0
Preference
free of contradiction
150
99.3
Event
consistent & grounded
150
94.0
Appendix
Table 6: Validation of the generated textual references. N denotes the number of inspected items, and Agr. denotes the agreement rate (%) between two annotators.
Stage
Inspected
Retained
Rate
VCR query selection
4,000
120
3.0%
Type-3 image filtering
3,117
579
18.6%
Appendix
Table 7: Retention rates of the visual query filtering stages in MPRBench .
Method
Natural
Synthetic
Δ
GPT-4o+P
70.4
79.3
+8.9
GPT-5-mini+P
67.9
77.2
+9.3
GPT-5+P
76.5
81.2
+4.7
Gemini-2.5 Pro+P
73.5
76.2
+2.7
LLaVA+P
46.9
50.7
+ 3.8
Qwen2.5-VL-7B+P
60.5
54.3
− 6.2
Appendix
Table 8: Accuracy on natural versus synthetically generated visual queries, averaged over the sub-tasks that contain both image sources. Δ denotes synthetic minus natural.
Method
w/ distractors
w/o distractors
Δ
GPT-4o+P
92.2
93.7
+1.5
GPT-5-mini+P
89.3
97.9
+8.6
GPT-5+P
88.4
95.8
+7.4
Gemini-2.5 Pro+P
83.5
96.8
+13.3
LLaVA+P
68.9
82.1
+13.2
Qwen2.5-VL-7B+P
75.7
90.5
+14.8
Appendix
Table 9: Effect of removing visually similar distractors from the query images of the predictive reasoning sub-task. Δ denotes w/o distractors minus w distractors.
Figure 8: Analysis on Text-Specific Distractor
Method
w/ distractors
w/o distractors
Δ
GPT-4o+P
60.5
69.4
+8.9
GPT-5-mini+P
71.9
75.9
+4
GPT-5+P
79.9
83.1
+3.2
Gemini-2.5 Pro+P
78.5
83.1
+4.6
Qwen2.5-VL-7B+P
45.4
51.0
+5.6
RAP
33.0
33.4
+0.4
Appendix
Table 10: Effect of inter-concept distractors. Scores are the overall Avg. on MPRBench. Δ denotes w/o distractors minus w distractors.
Figure 9: Comparison between General Multi-modal Capabilities and Personalized Reasoning Performance
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Personalized Large Language Models (PLLMs) aim to tailor responses to individual users, where a central challenge is preference reasoning: inferring query-relevant preferences from user-related history. Existing benchmarks, however, largely assume that such preference can be retrieved from semantically related history. We study an underexplored but practically important regime, profile-preference conceptual misalignment (PRCM), where observable profile cues and query-specific preferences lie in different concept spaces, making semantic retrieval inconsistent for personalization. We introduce VIBE-Bench, a benchmark with two psychology-grounded tasks, 3,504 personas and 12,239 dialogues, including a manually verified gold test set, and requires cross-concept preference reasoning beyond surface semantic overlap. Experiments with several personalization methods show that current PLLMs largely rely on shallow semantic correlations and fail to acquire robust cross-concept mappings. These findings establish PRCM as a distinct failure regime in PLLMs and position VIBE-Bench as a focused testbed for advancing preference reasoning beyond semantic matching.
Yiwen Jiang, Yang Deng, Stephanie Fong +9
1Monash University · 2Singapore Management University · University of Liverpool +1
Personalizing Multimodal Large Language Models (MLLMs) aims to recognize users' unique concepts from visual data and provide personalized responses. Although prior work has shown the benefit of concept descriptions and reasoning for this task, MLLM descriptions often include information, such as state and context, that does not help and may in fact hinder the unique identification of the target concept among other visually similar items. Effective descriptions of personal concepts should instead be accurate, discriminative, and free of distracting details. To achieve such descriptions, we introduce Reinforced Reference Game (RRG), a learning framework that promotes discriminative descriptions through a novel reinforced multimodal reference game. The MLLM plays both the roles of speaker and listener in a contrastive game setting, whose goal is to effectively communicate discriminative information about a target concept. Our approach formulates a verifiable contrastive reward over hard positives (dissimilar views of the same concept) and hard negatives (visually similar but different concepts). Empirically, RRG achieves state-of-the-art across multiple tasks on three personalization benchmarks. RRG generalizes to unseen domains and outperforms existing methods based on concept descriptions and personalization-specific RL frameworks. We will release code and models in the project page.
Deepayan Das, Davide Talon, Yiming Wang +2
University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy