Training-Free VLM Personalization via Calibrated Residual Decoding

Jiaao Yu; Yujian Ma; Xianming Hu; Pengran Wang; Ang Li

cs.CVAug 23, 2026

Training-Free VLM Personalization via Calibrated Residual Decoding

Authors: Jiaao Yu, Yujian Ma, Xianming Hu, Pengran Wang, Ang Li

Organizations: East China Normal University, Shanghai, China · Chang’an University, Shanxi, China

Abstract

Vision-language models can be personalized in a training-free manner by directly providing user profiles, preferences, or visual references at inference time, without updating model parameters. However, direct personalized prompting does not guarantee that the model will reliably exploit such evidence. The predictive distribution under the positive user profile often mixes two sources: personalized signals genuinely supported by the current profile, and the model's generic visual or linguistic priors. As a result, from the positive-profile response alone, it is difficult to determine whether a high-confidence answer is supported by the user profile or merely reflects the model's default preference. To address this problem, we propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive profile , a counterfactual profile , and an empty profile . Our method keeps the prediction under as the anchored base, and explicitly estimates the marginal contribution of personalization from score differences across the three conditions. We further introduce normalized-entropy-based uncertainty calibration, allowing the strength of personalized enhancement to adapt to the reliability of the residual signal. Experiments on MMPB, YoLLaVA, and MyVLM show that the proposed method improves personalized multimodal understanding without fine-tuning, with consistent gains on identity-sensitive visual personalization tasks. Additional analysis shows that entropy calibration stabilizes residual decoding when the contrastive personalization signal is uncertain.

Figures & tables

Figure 1: Ambiguity of positive-profile prompting. The output distribution under the positive profile

E^{+}

mixes profile-supported signals with the VLM’s generic prior. From

E^{+}

alone, a high-scoring response may be difficult to attribute to either source.

Figure 2: Our decoding framework estimates the marginal contribution of personalization via three evidence conditions ( $E^{+}$ , $E^{-}$ , $E^{0}$ ), with entropy-based calibration controlling the residual strength.

Subset	#Samples	$E^{+}$ Err.	Prior-Aligned
Pref. Incon.	1250	25.92	78.09
Rec. Aware	1598	22.65	89.78
Combined	2848	24.09	84.26

Table 1: Prior-aligned errors under positive-profile prompting on MMPB. All values except #Samples are percentages. Prior-Aligned denotes the share of

E^{+}

errors where

\hat{y}^{+}=\hat{y}^{0}\neq y^{*}

Model	Method	MMPB (Macro)	YoLLaVA	MyVLM	Avg.
Qwen2.5-VL-3B	No Profile	49.82	42.05	50.00	47.29
	$E^{+}$ Prompting	68.98	86.67	95.86	83.84
	$E^{0}$ Contrast	69.68 +0.70	89.23 +2.56	96.13 +0.27	85.01 +1.17
	$E^{-}$ Contrast	70.77 +1.79	95.90 +9.23	98.62 +2.76	88.43 +4.59
	Ours	70.76 +1.78	97.44 +10.77	99.38 +3.52	89.19 +5.35
Qwen2.5-VL-7B	No Profile	48.80	52.31	50.69	50.60

Table 2: Overall performance across models and personalized multimodal benchmarks. “No Profile” denotes the no-personalization lower bound using the empty-profile condition. Green subscripts indicate absolute gains over

E^{+}

Prompting. “

E^{0}

Contrast” and “

E^{-}

Contrast” are simple single-reference residual baselines without entropy calibration. We report macro accuracy on MMPB and weighted accuracy on YoLLaVA and MyVLM. Avg. denotes the arithmetic mean over the three benchmarks.

Method	P.A.	P.I.	R.A.	R.I.
$E^{+}$ Prompting	32.85	73.90	76.80	81.95
$E^{0}$ Contrast	40.90	83.50	81.08	79.40
$E^{-}$ Contrast	39.65	88.00	94.29	80.90
Ours	43.00	90.00	94.22	79.55

Table 3: Fine-grained results on representative MMPB sub-tasks. Pref.Aware(P.A.) and Pref.Incon.(P.I.) evaluate preference-oriented personalization, while Rec.Aware(R.A.) and Rec.Incon.(R.I.) evaluate personalized visual recognition under user-specific context.

Method	Accuracy	$\Delta$ vs Base
Base	84.84	0.00
w/o Entropy	75.23	-9.61
Ours	81.59	-3.25

Table 4: Robustness on the public MME benchmark under irrelevant textual personalization context. Base denotes the original model without any user profile. Ours w/o Entropy removes entropy-based confidence calibration, while Ours uses the full calibrated residual decoding rule.

Method	MMPB (Macro)	YoLLaVA
$E^{+}$ Prompting	68.98	86.67
w/o Entropy	70.35	95.39
Ours	70.76	97.44

Table 5: Ablation of entropy-based confidence calibration on personalized benchmarks. Ours w/o Entropy removes confidence calibration and directly applies residual personalization, while Ours uses the full entropy-calibrated decoding rule.

Figure 3: Residual evidence analysis. Each point corresponds to an error made by direct

E^{+}

prompting; blue points are corrected by our method, while red points remain incorrect. Panel (a) compares the residual margins provided by the empty-profile reference

E^{0}

and the counterfactual reference

E^{-}

, where larger values indicate stronger support for the gold answer over the original

E^{+}

prediction. Panel (b) relates entropy confidence

\gamma

to the residual margin for the gold answer, showing whether the residual is both reliable and directed toward the correct response. Point size in panel (a) indicates

\gamma

Subset	Win	Tie	Avg. Score ( $\uparrow$ )
Overall	63.0	18.0	3.54 $\rightarrow$ 4.70
Text-profile QA	45.0	30.0	3.80 $\rightarrow$ 4.38
Image-profile QA	75.0	10.0	3.37 $\rightarrow$ 4.92

Table 6: LLM-judge evaluation on the 100-example open-ended personalized QA benchmark using Qwen2.5-VL-3B. We compare standard positive prompting against our token-level calibrated residual decoding. Win/Tie denotes the percentage of cases where the judge prefers our method, considers the two responses tied. Avg. Score reports the mean overall judge score on a 0–6 scale.

Figure 4: Qualitative examples of token-level personalized decoding. The calibrated residual shifts the first-token decision toward the profile-supported answer and assigns higher confidence to identity-specific tokens during full-answer generation.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 734 papers

Personalize Your Large Vision-language Models With In-context Prompt Tuning
May 29, 2026Yanshu Li, Jiaqian Li, Kuai Yu +4Vision-Language Models Large Vision-Language Models
Brown University · Columbia University · University of Alabama at Birmingham +2
Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
May 9, 2026Guodong Ding, Angela YaoVision-Language Models VLM Adaptation
National University of Singapore
Ego: Embedding-Guided Personalization of Vision-Language Models
Mar 10, 2026Soroush Seifi, Simon Gardier, Vaggelis Dorovatas +2VLM Adaptation Memory-Augmented VLMs
Toyota Motor Europe

May 29, 2026J/K move · Enter open · S save

Personalize Your Large Vision-language Models With In-context Prompt Tuning

Yanshu Li, Jiaqian Li, Kuai Yu, Xi Xiao+3

Brown University · Columbia University · University of Alabama at Birmingham+2

Vision-Language Models Large Vision-Language Models Visual Prompt Learning

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing interest in LVLM personalization, which aims to enable models to quickly and effectively learn out-of-distribution multimodal concepts to meet user-specific needs. However, many existing methods rely on inference-time training, which reduces efficiency. They also struggle to maintain accuracy in complex multi-image, multi-concept settings. These limitations restrict the broader deployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweight projection module capable of operating in complex scenarios to extract fine-grained visual semantics from multiple reference images, seamlessly transforming these features alongside identity-label mappings into continuous prompts. To maximize computational efficiency, this module adaptively determines the prompt length based on the intrinsic visual complexity of each concept. Crucially, to overcome the environmental biases and cross-concept interference prevalent in real-world applications, we introduce two novel geometric regularizations. These constraints refine prompt representations by decoupling key identities from transient environmental states and separating concepts to avoid semantic confusion. Extensive experiments show that ICPT achieves state-of-the-art personalization accuracy across diverse tasks and LVLM backbones.

Abstract

Vision-Language Models

Explore similar work

Personalize Your Large Vision-language Models With In-context Prompt Tuning

Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models

Ego: Embedding-Guided Personalization of Vision-Language Models