Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

Tianci Luo; Haohao Pan; Jinpeng Wang; Niu Lian; Xinrui Chen; Bin Chen; Shu-Tao Xia; Chun Yuan

cs.CVApr 4, 2026

Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

Authors: Tianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian, Xinrui Chen, Bin Chen, Shu-Tao Xia, Chun Yuan

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Computer Science and Engineering, Northeastern University · Harbin Institute of Technology, Shenzhen

Abstract

Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR (Label-aware Prompt Retrieval), which highlights the role of labels in prompt selection. Our framework first designs an image-label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR.

Figures & tables

Figure 1: We randomly sample

100

query-prompt training pairs constructed by SupPR [ 46 ] . We compute the label matching consistency and VICL performance to investigate their correlation.

Figure 2: Prompt retrieval paradigms. (a) Label-agnostic pipelines rely on image similarity and may yield label-inconsistent prompts and disturb inference. (b) LaPR considers both image similarity and label consistency for prompt retrieval, retrieving more relevant prompts.

Figure 3: Overview of LaPR . ( a ) LaPR architecture. Prompt labels are injected to form joint embeddings. On both sides, experts produce mode specific features and a query conditioned router picks the matching mode and extracts its information, resulting in label-aware query embeddings and query-relevant prompt embeddings. ( b ) Training framework. Each mini-batch alternates an expert step (performance-guided contrastive learning, router fixed) and a router step (label-guided contrastive learning with load balancing, experts frozen).

		Seg. (mIoU) $\uparrow$
Prompt Selection Method	Ref.	Fold-0	Fold-1	Fold-2	Fold-3	AVG	Det. (mIoU) $\uparrow$	Col. (MSE) $\downarrow$
Random [ 3 ]	NIPS 2022	28.66	30.21	27.81	23.55	27.56	25.45	0.67
UnsupPR [ 46 ]	NIPS 2023	34.75	35.92	32.41	31.16	33.56	26.84	0.63
SupPR [ 46 ]	NIPS 2023	37.08	38.43	34.40	32.32	35.56	28.22	0.63
Zhu et al. [48]	AAAI 2025	36.86	42.22	37.11	30.84	36.76	28.25	0.62
Partial2Global [ 42 ]	NIPS 2024	38.81	41.54	37.25	36.01	38.40	30.66	0.58

Table 1: Comparison of LaPR with existing methods across three VICL tasks, including foreground segmentation, single-object detection, and colorization. The best performance values are boldfaced , and the second-best ones are italicized for clarity.

		Target
Method	Source	Fold-0	Fold-1	Fold-2	Fold-3	AVG
SupPR	Fold-0	—	35.46	32.44	30.95	32.95
	Fold-1	34.92	—	32.96	31.03	32.97
	Fold-2	34.71	36.48	—	30.08	33.76
	Fold-3	34.01	35.83	32.15	—	34.00
Partial2Global	Fold-0	—	36.38	32.63	30.90	33.30

Table 2: Transferability across folds for LaPR , SupPR, and Partial2Global on the segmentation benchmark. Each method is trained on one fold and evaluated on the remaining folds.

			Seg. (mIoU $\uparrow$ )
Type	ID	Method	Fold-0	Fold-1	Fold-2	Fold-3	AVG	Det. (mIoU $\uparrow$ )
Full Model	(0)	LaPR (Ours)	41.92	46.27	39.63	37.63	41.36	32.01
	(1)	w/o Router	38.27	43.31	36.23	34.99	38.20	29.69
Architectural Components	(2)	w/o Prompt Label	39.14	44.24	37.82	35.56	39.19	30.94
Feature Extractor Substitution	(3)	CLIP $\rightarrow$ DINOv2	41.51	46.08	39.93	38.05	41.39	32.06
Optimization Strategy	(4)	Single-Stage Training	40.16	44.70	38.34	36.22	39.86	31.21

Table 3: Ablation study on segmentation and detection across different configurations.

Figure 4: Qualitative visualization comparing LaPR (label-aware prompt retrieval) with SupPR (label-agnostic prompt retrieval). LaPR consistently retrieves label compatible prompts and yields visibly more accurate VICL predictions.

Figure 5: Proportion of each mode-specific expert selected under each category, visualized as a heatmap.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment
Mar 19, 2026Tianci Luo, Jinpeng Wang, Shiyu Qin +5In-Context Learning Feature Fusion
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen · Meituan, Beijing
Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts
Oct 22, 2026Nimrod Shabtay, Asaf Yehudai, Avihu Dekel +2Multimodal ICL In-Context Learning
IBM Research · Tel-Aviv University
T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
Nov 20, 2025Shao-Jun Xia, Huixin Zhang, Zhengzhong TuVision-Language Models In-Context Learning
Duke University · Texas A&M University

Mar 19, 2026J/K move · Enter open · S save

PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment

Tianci Luo, Jinpeng Wang, Shiyu Qin, Niu Lian+4

Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen · Meituan, Beijing

In-Context Learning Feature Fusion

Open paper AskPDF arXiv

Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmentation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes a reliable locality-aware paradigm for prompt fusion, moving beyond prior patch-wise approaches. Code is available at https://github.com/luotc-why/ICLR26-PromptHub.

Abstract

In-Context Learning

Explore similar work

PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs