Ego: Embedding-Guided Personalization of Vision-Language Models

Soroush Seifi; Simon Gardier; Vaggelis Dorovatas; Daniel Olmeda Reino; Rahaf Aljundi

cs.CVMar 10, 2026

Ego: Embedding-Guided Personalization of Vision-Language Models

Authors: Soroush Seifi, Simon Gardier, Vaggelis Dorovatas, Daniel Olmeda Reino, Rahaf Aljundi

Organizations: Toyota Motor Europe

Abstract

AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.

Figures & tables

Figure 1: Personalization approaches vs Ego . Existing methods typically require test-time or LVLM fine-tuning, or depend on external vision modules, and often fail to support multi-concept or video-level personalization. In contrast, Ego is training-free, LVLM-agnostic, requires no external modules, and efficiently enables single-concept, multi-concept, and video-native personalization within a unified framework.

Figure 2: Our proposed method Ego . Personalized Concept Introduction: The LVLM is tasked to estimate the subject area in the reference image and generate keywords describing its main characteristics. Ego identifies the most representative visual tokens via keywords cross cross-attention and creates a concept memory. Inference: Given a test image, the LVLM in Ego accesses internal concept memories in context to recall and reason about known subjects in the image. Ego requires neither additional training nor external modules.

			Single Concept									Multi Concept
Method	Model	Training Time $\downarrow$	MyVLM [ 1 ]			Yo’LLaVA [ 24 ]			This-is-my (Single) [ 32 ]			This-is-my (Multi)			RAP (Multi) [ 11 ]
			Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
1 Reference View
RAP [ 11 ]	Intern	24hrs	63.4	98.2	77.0	47.8	93.7	63.3	83.4	91.3	87.1	100.0	62.0	76.5	90.7	100.0	95.1
R2P [ 7 ]	Intern	5.98s	54.1	93.4	68.5	53.1	85.6	65.5	61.0	76.3	67.7	–	–	–	–	–	–
Ego (Ours)	Intern	1.40s	86.0	94.8	90.2	77.2	83.4	80.2	81.3	77.0	79.1	93.9	78.2	88.6	100.0	96.9	98.4

Table 1: Recognition. Performance comparison across methods and settings using InternVL3 (14B) [ 39 ] and Qwen2.5-VL (7B) [ 2 ] . Best results are shown in bold , and second-best results are underlined . For training-free methods, the Training Time column reflects the average time required for concept introduction. Ego attains state-of-the-art recognition accuracy across datasets while introducing concepts with minimal overhead. Note that RAP dataset is limited to a single-reference training set, and R2P [ 7 ] does not support multi-concept tasks.

		Single Concept			Multi Concept		Video
Method	Sample Runtime (s $\downarrow$ )	VQA (Acc. $\uparrow$ )		Captioning (Recall $\uparrow$ )	VQA (Acc. $\uparrow$ )	Captioning (Recall $\uparrow$ )	VQA (Acc. $\uparrow$ )
		Yo’LLaVA	This-is-my	MyVLM	This-is-my	This-is-my	This-is-my
RAP [ 11 ]	7.8	97.6	90.0	65.6	53.7	43.6	–
R2P [ 7 ]	7.0	94.0	82.0	77.5	–	–	–
PeKit	5.8	94.6	92.0	81.1	51.8	35.2	59.9
Ego (Ours)	6.0	92.3	88.0	91.3	72.2	70.9	70.0

Table 2: VQA Acc. and Captioning Recall. Performance comparison ( 1 ref. view and InternVL3). Ego attains competitive accuracy on single-concept VQA while achieving SOTA performance in both multi-concept and video settings. Ego delivers superior captioning-recall. R2P only supports single-concept personalization and RAP does not support video personalization.

Figure 3: Qualitative results. Top row: keywords and highlighted patches of selected visual tokens ( Ego concept memory) for various concepts, illustrating their representativeness and adaptability to object size. Bottom row: Ego demonstrates Video QA capability.

Method	% of Visual Tokens	Word Injection	F1 Score $\uparrow$
Keywords	✗	✓	71.3
Full Visual	100%	✗	84.1
Full Visual + Keywords	100%	✓	82.5
Uniform	20%	✗	77.7
Ego (1-view)	20%	✗	80.4
Ego (5-view)	20%	✗	85.7

Table 3: Ablation . Ego outperforms uniform visual token selection and Full Visual under the same In Context tokens budget.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Prec.	Rec.	F1
	All concepts
$K_{c}\leq 50$	81.3	77.0	79.1
$K=50$	82.4	74.3	78.1
	Zak’s Dog Coffee 4
$K_{c}\leq 50$	81.8	34.6	48.6
$K=50$	100.0	3.85	7.41

Appendix

Table 4:

K

K_{c}

: Ego adapts to the size of the target concept in the reference views and boosts the performance compared to a fixed memory size selection strategy.

Figure 4: Extracted visual tokens for Zak’s Dog Coffee . Left: Fixed $K=50$ Right : Dynamic $K_{c}=25$ . Ego removes 25 background patches by adapting to the concept’s size.

Experiment	Prec.	Rec.	F1
Ego (29, 30, 35, 36, 39)	81.3	77.0	79.1
Manual selection (20-24)	81.7	73.5	77.4
Uniform layer selection	78.7	73.1	75.8

Appendix

Table 5: LVLM Layer selection: Parentheses indicate the selected layers for InternVL3-14B. Ego outperforms the baselines by emphasizing more discriminative features while suppressing background noise.

Experiment	Prec.	Rec.	F1
	1 Ref. View
Keywords	81.3	77.0	79.1
Caption	80.4	77.6	79.0
	5 Ref. views
Keywords	86.1	68.7	76.5
Caption	87.2	65.9	75.1

Appendix

Table 6: Full description vs Keywords cross-attention on This-is-my single concept recognition task

Experiment	Prec.	Rec.	F1
	1 Ref. View
No sampling	81.3	77.0	79.1
$N=5$	81.2	75.9	78.5
	5 Ref. views
No sampling	87.2	65.9	75.1
$N=5$	86.8	63.5	73.4

Appendix

Table 7: Visual token selection over 5 samples vs no sampling on This-is-my single concept recognition task.

Method	Single VQA	Multi VQA	Video VQA	Single Recognition			Multi Recognition
Method	Single VQA	Multi VQA	Video VQA	Prec.	Rec.	F1	Prec.	Rec.	F1
Ego (Ours)	88.0	72.2	70.0	81.3	77.0	79.1	93.9	78.2	88.6
Full Ref. View	86.0	66.7	68.2	84.3	67.7	75.1	100.0	65.4	79.1
Base VLM	86.0	55.7	55.1	50.3	32.3	39.3	81.4	16.4	27.2

Appendix

Table 8: VQA Accuracy and Captioning Recall. Full Ref. View vs Ego vs Base VLM on all This-is-my tasks.

InternVL Size	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
2B (tiny)	48.2	91.5	63.1
8B (small)	88.4	81.1	84.6
14B (medium)	85.0	86.4	85.7
14B & G-SAM	83.2	94.3	88.4

Appendix

Table 9: Ego ’s performance with various model sizes and reference objects segmented with G-SAM on Yo’LLaVA dataset.

Figure 5: Ego ’s generated keywords, estimated concept sizes, and selected patches are shown using examples from various datasets. Ego demonstrates the ability to accurately estimate concept sizes and extract informative patches for each object while minimizing background interference.

Figure 6: Qualitative results on multi-concept VQA in the This-is-My dataset. Ego maintains accurate responses, while RAP [ 11 ] modifies the model’s behavior and PeKit [ 32 ] fails on fine-grained distinctions.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Personalize Your Large Vision-language Models With In-context Prompt Tuning
May 29, 2026Yanshu Li, Jiaqian Li, Kuai Yu +4Vision-Language Models Large Vision-Language Models
Brown University · Columbia University · University of Alabama at Birmingham +2
Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
May 9, 2026Guodong Ding, Angela YaoVision-Language Models VLM Adaptation
National University of Singapore
Training-Free VLM Personalization via Calibrated Residual Decoding
Aug 23, 2026Jiaao Yu, Yujian Ma, Xianming Hu +2Vision-Language Models VLM Adaptation
East China Normal University, Shanghai, China · Chang’an University, Shanxi, China

May 29, 2026J/K move · Enter open · S save

Personalize Your Large Vision-language Models With In-context Prompt Tuning

Yanshu Li, Jiaqian Li, Kuai Yu, Xi Xiao+3

Brown University · Columbia University · University of Alabama at Birmingham+2

Vision-Language Models Large Vision-Language Models Visual Prompt Learning

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing interest in LVLM personalization, which aims to enable models to quickly and effectively learn out-of-distribution multimodal concepts to meet user-specific needs. However, many existing methods rely on inference-time training, which reduces efficiency. They also struggle to maintain accuracy in complex multi-image, multi-concept settings. These limitations restrict the broader deployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweight projection module capable of operating in complex scenarios to extract fine-grained visual semantics from multiple reference images, seamlessly transforming these features alongside identity-label mappings into continuous prompts. To maximize computational efficiency, this module adaptively determines the prompt length based on the intrinsic visual complexity of each concept. Crucially, to overcome the environmental biases and cross-concept interference prevalent in real-world applications, we introduce two novel geometric regularizations. These constraints refine prompt representations by decoupling key identities from transient environmental states and separating concepts to avoid semantic confusion. Extensive experiments show that ICPT achieves state-of-the-art personalization accuracy across diverse tasks and LVLM backbones.

Abstract

VLM Adaptation

Explore similar work

Personalize Your Large Vision-language Models With In-context Prompt Tuning

Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models

Training-Free VLM Personalization via Calibrated Residual Decoding