Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning
Authors: Tianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian, Xinrui Chen, Bin Chen, Shu-Tao Xia, Chun Yuan
Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Computer Science and Engineering, Northeastern University · Harbin Institute of Technology, Shenzhen
Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR (Label-aware Prompt Retrieval), which highlights the role of labels in prompt selection. Our framework first designs an image-label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR.
Figures & tables
Figure 1: We randomly sample 100 query-prompt training pairs constructed by SupPR [ 46 ] . We compute the label matching consistency and VICL performance to investigate their correlation.
Figure 2: Prompt retrieval paradigms. (a) Label-agnostic pipelines rely on image similarity and may yield label-inconsistent prompts and disturb inference. (b) LaPR considers both image similarity and label consistency for prompt retrieval, retrieving more relevant prompts.
Figure 3: Overview of LaPR . ( a ) LaPR architecture. Prompt labels are injected to form joint embeddings. On both sides, experts produce mode specific features and a query conditioned router picks the matching mode and extracts its information, resulting in label-aware query embeddings and query-relevant prompt embeddings. ( b ) Training framework. Each mini-batch alternates an expert step (performance-guided contrastive learning, router fixed) and a router step (label-guided contrastive learning with load balancing, experts frozen).
Seg. (mIoU) ↑
Prompt Selection Method
Ref.
Fold-0
Fold-1
Fold-2
Fold-3
AVG
Det. (mIoU) ↑
Col. (MSE) ↓
Random [ 3 ]
NIPS 2022
28.66
30.21
27.81
23.55
27.56
25.45
0.67
UnsupPR [ 46 ]
NIPS 2023
34.75
35.92
32.41
31.16
33.56
26.84
0.63
SupPR [ 46 ]
NIPS 2023
37.08
38.43
34.40
32.32
35.56
28.22
0.63
Zhu et al. [48]
AAAI 2025
36.86
42.22
37.11
30.84
36.76
28.25
0.62
Partial2Global [ 42 ]
NIPS 2024
38.81
41.54
37.25
36.01
38.40
30.66
0.58
Table 1: Comparison of LaPR with existing methods across three VICL tasks, including foreground segmentation, single-object detection, and colorization. The best performance values are boldfaced , and the second-best ones are italicized for clarity.
Target
Method
Source
Fold-0
Fold-1
Fold-2
Fold-3
AVG
SupPR
Fold-0
—
35.46
32.44
30.95
32.95
Fold-1
34.92
—
32.96
31.03
32.97
Fold-2
34.71
36.48
—
30.08
33.76
Fold-3
34.01
35.83
32.15
—
34.00
Partial2Global
Fold-0
—
36.38
32.63
30.90
33.30
Table 2: Transferability across folds for LaPR , SupPR, and Partial2Global on the segmentation benchmark. Each method is trained on one fold and evaluated on the remaining folds.
Seg. (mIoU ↑ )
Type
ID
Method
Fold-0
Fold-1
Fold-2
Fold-3
AVG
Det. (mIoU ↑ )
Full Model
(0)
LaPR (Ours)
41.92
46.27
39.63
37.63
41.36
32.01
(1)
w/o Router
38.27
43.31
36.23
34.99
38.20
29.69
Architectural Components
(2)
w/o Prompt Label
39.14
44.24
37.82
35.56
39.19
30.94
Feature Extractor Substitution
(3)
CLIP → DINOv2
41.51
46.08
39.93
38.05
41.39
32.06
Optimization Strategy
(4)
Single-Stage Training
40.16
44.70
38.34
36.22
39.86
31.21
Table 3: Ablation study on segmentation and detection across different configurations.
Figure 4: Qualitative visualization comparing LaPR (label-aware prompt retrieval) with SupPR (label-agnostic prompt retrieval). LaPR consistently retrieves label compatible prompts and yields visibly more accurate VICL predictions.
Figure 5: Proportion of each mode-specific expert selected under each category, visualized as a heatmap.
Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmentation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes a reliable locality-aware paradigm for prompt fusion, moving beyond prior patch-wise approaches. Code is available at https://github.com/luotc-why/ICLR26-PromptHub.
Tianci Luo, Jinpeng Wang, Shiyu Qin +5
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen · Meituan, Beijing
Vision-language models (VLMs) struggle with in context learning compared to their textual counterparts. Our preliminary study shows that VLMs rely on textual shortcuts and underutilize the visual demonstrations. We therefore hypothesize that VLMs are insufficiently trained to perform visual in-context learning (vICL). To this end, we construct MIME, a task-agnostic vICL training set without textual shortcuts. We ensure the instruction does not state the rule linking the demonstration images to their answers, so the model must infer it from the visual demonstrations. Fine-tuning on MIME improves vICL accuracy by 8.1 to 18.0 points across three VLMs, preserves single-image performance, and improves multi-image and text only in-context learning. In contrast, stating the rule in the training data sharply reduces this gain. We further propose DTAD (Demonstration Text Attention Dropout), which randomly withholds demonstration labels during training so that the VLM must infer the rule from the demonstration images. While achieving comparable vICL accuracy to fine-tuning on MIME, DTAD yields more consistent attention to the demonstration images. On TrueMICL, Qwen3-VL-8B trained on MIME matches frontier models such as Claude Opus 5, and surpasses them with DTAD. Finally, adding MIME to the instruction-tuning mix of LLaVA 1.5 13B more than doubles its vICL accuracy without hurting single-image abilities, whereas a same-size multi-image set improves it only slightly. These results suggest that vICL data should be a standard component of VLM training.
Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training. Recent advances in large vision-language models (VLMs) have shown promising VICL capability when the demonstration pair and the query belong to the same vision task, but real use cases often provide mismatched examples, making it unclear whether a VLM should imitate the demonstrated transformation or infer a new one from the query. This raises a fundamental question: Can VLMs perform cross-task VICL where demonstration and query differ? In the paper, we study this cross-task VICL setting and propose T2T-VICL, a collaborative prompt-transfer framework, which converts mismatched visual demonstrations into implicit textual guidance without explicitly naming the tasks. To do so, a large teacher VLM first generates structured descriptions of visual changes and task differences between task pairs, from which we construct a dataset of diverse implicit cross-task relations. We then distill this capability into a lightweight student VLM that produces content-dependent prompts from a task-A demonstration pair and a task-B query. The generated prompt is used to guide a frozen image-editing VLM, and a score-based inference strategy is introduced to rank multiple candidates. Experiments on 12 low-level vision tasks and over 20 evaluated cross-task pairs show that T2T-VICL consistently improves task-aware alignment over fixed prompting and often also improves image fidelity, revealing both the potential and limits of cross-task VICL. Our code is available on GitHub.