Vision-language models can be personalized in a training-free manner by directly providing user profiles, preferences, or visual references at inference time, without updating model parameters. However, direct personalized prompting does not guarantee that the model will reliably exploit such evidence. The predictive distribution under the positive user profile often mixes two sources: personalized signals genuinely supported by the current profile, and the model's generic visual or linguistic priors. As a result, from the positive-profile response alone, it is difficult to determine whether a high-confidence answer is supported by the user profile or merely reflects the model's default preference. To address this problem, we propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive profile , a counterfactual profile , and an empty profile . Our method keeps the prediction under as the anchored base, and explicitly estimates the marginal contribution of personalization from score differences across the three conditions. We further introduce normalized-entropy-based uncertainty calibration, allowing the strength of personalized enhancement to adapt to the reliability of the residual signal. Experiments on MMPB, YoLLaVA, and MyVLM show that the proposed method improves personalized multimodal understanding without fine-tuning, with consistent gains on identity-sensitive visual personalization tasks. Additional analysis shows that entropy calibration stabilizes residual decoding when the contrastive personalization signal is uncertain.
Figures & tables
Figure 1: Ambiguity of positive-profile prompting. The output distribution under the positive profile E+ mixes profile-supported signals with the VLM’s generic prior. From E+ alone, a high-scoring response may be difficult to attribute to either source.
Figure 2: Our decoding framework estimates the marginal contribution of personalization via three evidence conditions ( E+ , E− , E0 ), with entropy-based calibration controlling the residual strength.
Subset
#Samples
E+ Err.
Prior-Aligned
Pref. Incon.
1250
25.92
78.09
Rec. Aware
1598
22.65
89.78
Combined
2848
24.09
84.26
Table 1: Prior-aligned errors under positive-profile prompting on MMPB. All values except #Samples are percentages. Prior-Aligned denotes the share of E+ errors where y^+=y^0=y∗ .
Model
Method
MMPB (Macro)
YoLLaVA
MyVLM
Avg.
Qwen2.5-VL-3B
No Profile
49.82
42.05
50.00
47.29
E+ Prompting
68.98
86.67
95.86
83.84
E0 Contrast
69.68 +0.70
89.23 +2.56
96.13 +0.27
85.01 +1.17
E− Contrast
70.77 +1.79
95.90 +9.23
98.62 +2.76
88.43 +4.59
Ours
70.76 +1.78
97.44 +10.77
99.38 +3.52
89.19 +5.35
Qwen2.5-VL-7B
No Profile
48.80
52.31
50.69
50.60
Table 2: Overall performance across models and personalized multimodal benchmarks. “No Profile” denotes the no-personalization lower bound using the empty-profile condition. Green subscripts indicate absolute gains over E+ Prompting. “ E0 Contrast” and “ E− Contrast” are simple single-reference residual baselines without entropy calibration. We report macro accuracy on MMPB and weighted accuracy on YoLLaVA and MyVLM. Avg. denotes the arithmetic mean over the three benchmarks.
Method
P.A.
P.I.
R.A.
R.I.
E+ Prompting
32.85
73.90
76.80
81.95
E0 Contrast
40.90
83.50
81.08
79.40
E− Contrast
39.65
88.00
94.29
80.90
Ours
43.00
90.00
94.22
79.55
Table 3: Fine-grained results on representative MMPB sub-tasks. Pref.Aware(P.A.) and Pref.Incon.(P.I.) evaluate preference-oriented personalization, while Rec.Aware(R.A.) and Rec.Incon.(R.I.) evaluate personalized visual recognition under user-specific context.
Method
Accuracy
Δ vs Base
Base
84.84
0.00
w/o Entropy
75.23
-9.61
Ours
81.59
-3.25
Table 4: Robustness on the public MME benchmark under irrelevant textual personalization context. Base denotes the original model without any user profile. Ours w/o Entropy removes entropy-based confidence calibration, while Ours uses the full calibrated residual decoding rule.
Method
MMPB (Macro)
YoLLaVA
E+ Prompting
68.98
86.67
w/o Entropy
70.35
95.39
Ours
70.76
97.44
Table 5: Ablation of entropy-based confidence calibration on personalized benchmarks. Ours w/o Entropy removes confidence calibration and directly applies residual personalization, while Ours uses the full entropy-calibrated decoding rule.
Figure 3: Residual evidence analysis. Each point corresponds to an error made by direct E+ prompting; blue points are corrected by our method, while red points remain incorrect. Panel (a) compares the residual margins provided by the empty-profile reference E0 and the counterfactual reference E− , where larger values indicate stronger support for the gold answer over the original E+ prediction. Panel (b) relates entropy confidence γ to the residual margin for the gold answer, showing whether the residual is both reliable and directed toward the correct response. Point size in panel (a) indicates γ .
Subset
Win
Tie
Avg. Score ( ↑ )
Overall
63.0
18.0
3.54 → 4.70
Text-profile QA
45.0
30.0
3.80 → 4.38
Image-profile QA
75.0
10.0
3.37 → 4.92
Table 6: LLM-judge evaluation on the 100-example open-ended personalized QA benchmark using Qwen2.5-VL-3B. We compare standard positive prompting against our token-level calibrated residual decoding. Win/Tie denotes the percentage of cases where the judge prefers our method, considers the two responses tied. Avg. Score reports the mean overall judge score on a 0–6 scale.
Figure 4: Qualitative examples of token-level personalized decoding. The calibrated residual shifts the first-token decision toward the profile-supported answer and assigns higher confidence to identity-specific tokens during full-answer generation.
Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing interest in LVLM personalization, which aims to enable models to quickly and effectively learn out-of-distribution multimodal concepts to meet user-specific needs. However, many existing methods rely on inference-time training, which reduces efficiency. They also struggle to maintain accuracy in complex multi-image, multi-concept settings. These limitations restrict the broader deployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweight projection module capable of operating in complex scenarios to extract fine-grained visual semantics from multiple reference images, seamlessly transforming these features alongside identity-label mappings into continuous prompts. To maximize computational efficiency, this module adaptively determines the prompt length based on the intrinsic visual complexity of each concept. Crucially, to overcome the environmental biases and cross-concept interference prevalent in real-world applications, we introduce two novel geometric regularizations. These constraints refine prompt representations by decoupling key identities from transient environmental states and separating concepts to avoid semantic confusion. Extensive experiments show that ICPT achieves state-of-the-art personalization accuracy across diverse tasks and LVLM backbones.
Yanshu Li, Jiaqian Li, Kuai Yu +4
Brown University · Columbia University · University of Alabama at Birmingham +2
This paper tackles compositional personalization of vision-language models (VLMs). In this problem, multiple user-defined concepts must be recognized or described jointly at test time. We introduce Gate-and-Merge, a zero-shot framework that enables compositional personalization without the need for co-occurrence training. During personalization, each concept is learned independently as a lightweight LoRA adapter, paired with a concept token. The base model remains unchanged and concepts are kept disentangled. At inference, we enable composition by merging concept-specific LoRA updates directly in weight space. To suppress irrelevant activations and prevent interference, a gating mechanism is employed to estimate textual and visual cues and select only the modules that contribute to the prediction. We further stabilize composition by combining only the most meaningful and mutually consistent updates, helping preserve each concept's identity. Our quantitative and qualitative analyses show consistent gains in performance across multiple personalization tasks in both single-concept and compositional settings.
AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.
Soroush Seifi, Simon Gardier, Vaggelis Dorovatas +2