PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment

Tianci Luo; Jinpeng Wang; Shiyu Qin; Niu Lian; Yan Feng; Bin Chen; Chun Yuan; Shu-Tao Xia

cs.CVMar 19, 2026

PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment

Authors: Tianci Luo, Jinpeng Wang, Shiyu Qin, Niu Lian, Yan Feng, Bin Chen, Chun Yuan, Shu-Tao Xia

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen · Meituan, Beijing

Abstract

Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmentation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes a reliable locality-aware paradigm for prompt fusion, moving beyond prior patch-wise approaches. Code is available at https://github.com/luotc-why/ICLR26-PromptHub.

Figures & tables

Figure 1: (a) Condenser performs patch-wise fusion to fuse composite prompt, while leveraging model-agnostic supervision signals at the input level. (b) PromptHub transcends Condenser by enforcing a locality-aware chain that unifies fusion-utilization-prediction. It aligns spatial priors into coherent prompt representations, reinforces the backbone’s concentration on fused cues, and integrates label prediction to maintain the integrity of VICL pipeline. (c) Comparison of Condenser and PromptHub across three tasks under both single-prompting and multi-prompting configurations.

Figure 2: The training and inference framework of PromptHub based on MAE-VQGAN.

	Seg. (mIoU ↑)
Model	Fold-0	Fold-1	Fold-2	Fold-3	Mean	Det. (mIoU ↑)	Col. (MSE ↓)
Zero-Shot
Random ( Bar et al., 2022 )	28.66	30.21	27.81	23.55	27.56	25.45	0.67
UnsupPR ( Zhang et al., 2023 )	34.75	35.92	32.41	31.16	33.56	26.84	0.63
Prompt-SelF ( Sun et al., 2025 )	35.69	38.25	35.86	33.37	35.79	28.08	0.63
Retriever Training

Table 1: PromptHub performance is compared with different baselines in three downstream tasks foreground segmentation ( Seg .), single-object detection ( Det .), and image colorization ( Col .). The results for

{N}=1,16

, representing the cases with 1 and 16 prompts respectively, are listed separately. The highest results are denoted in bold , while the suboptimal results are indicated in italics .

Model	Seg. (mIoU ↑)
Model	Fold-0	Fold-1	Fold-2	Fold-3	Mean
Prompt-SelF	40.13	42.14	37.84	38.52	39.66
InMeMo	38.74	43.82	40.45	37.12	40.03
Condenser N=1	40.39	44.54	40.23	36.33	40.37
Condenser N=16	40.37	44.85	41.03	35.84	40.52
PromptHub N=1	40.36	45.24	40.43	37.94	41.00

Table 2: Transferability evaluation. We train models on COCO-5 i and test on Pascal-5 i .

Figure 5: Performance comparison with baselines in multi-prompt VICL scenario.

		Seg. (mIoU ↑)
#	Model	Fold-0	Fold-1	Fold-2	Fold-3	Mean	Det. (mIoU ↑)
(0)	PromptHub N=1	43.26	50.75	43.83	42.82	45.17	44.51
(1)	PromptHub N=16	45.93	53.12	45.44	46.74	47.81	45.59
Effectiveness of Learning Objectives
(2)	w/o $\mathcal{L}_{u}$ N=1	42.71	51.14	42.78	42.41	44.76	43.45
(3)	w/o $\mathcal{L}_{u}$ N=16	45.54	52.25	44.59	44.47	46.71	44.83

Table 3: Ablation study of PromptHub. The best are marked in bold and second-best in italic .

Figure 6: The visualization of the fused prompt pair after passing through the VQGAN decoder.

Figure 7: Comparison of PromptHub and Condenser across different retrieval.

	Det. (mIoU ↑)
Method	Fold-0	Fold-1	Fold-2	Fold-3	Mean
Condenser N=1	38.15	35.70	35.49	30.31	34.91
Condenser N=16	41.25	36.66	37.86	39.02	38.70
PromptHub N=1	41.59	36.25	37.71	32.15	36.93
PromptHub N=16	43.40	38.55	39.52	40.66	40.53

Table 4: Transferability experiment (unseen task) where both Condenser and PromptHub are trained on segmentation and evaluated on detection.

Method	Standard mIoU	Perturbed mIoU	Performance Drop
Condenser N=1	44.14	42.36	-1.78
Condenser N=16	46.63	45.24	-1.39
PromptHub N=1	45.17	44.01	-1.16
PromptHub N=16	47.81	47.15	-0.66

Table 5: Comparison of standard and perturbed mIoU under spatial misalignment, along with the corresponding performance drops.

	Seg. (mIoU ↑)
Method	Fold-0	Fold-1	Fold-2	Fold-3	Mean
Condenser N=1	18.93	29.73	24.26	27.94	25.22
Condenser N=16	14.27	24.56	18.85	20.56	19.56
PromptHub N=1	21.25	37.61	35.01	29.79	30.92
PromptHub N=16	25.22	43.22	36.07	30.41	33.73

Table 6: Comparison the mIoU between the fused prompt labels and the query labels across methods to evaluate semantic alignment.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Introduction to MAE-VQGAN ( Bar et al., 2022 ) : (a) In the pre-training stage, MAE ( He et al., 2022 ) is trained to enhance its inference capability through a masked reconstruction task on CVF dataset. (b) In the inference stage, the prompt pair is placed above, with the query positioned below, and both are fed into the model for generative processing.

Method	Inference Time	GPU Cost
Method	(ms/query)	(MB/query)
MAE-VQGAN	51.26	416.14
InMeMo	54.28	497.50
Prompt-SelF N=16	984.62	441.75
Condenser N=1	59.17	565.42
Condenser N=16	66.61	1021.86

Appendix

Table 7: Comparison of the inference time and GPU overhead between PromptHub and baselines.

Figure 9: Evaluation of PromptHub’s performance on three tasks across varying values of

\sigma

Method	Fold-0	Fold-1	Fold-2	Fold-3	Mean	Det
PromptHub N=1 (query-adaptive sigma)	43.79	51.93	44.56	43.18	45.86	44.25
PromptHub N=16 (query-adaptive sigma)	46.44	52.97	45.66	46.89	47.99	45.41
PromptHub N=1 (hyperparameter sigma)	43.26	50.75	43.83	42.82	45.17	44.51
PromptHub N=16 (hyperparameter sigma)	45.93	53.12	45.44	46.74	47.81	45.59

Appendix

Table 8: Comparison of results between query-conditioned sigma and hyperparameter sigma.

Method	Fold-0	Fold-1	Fold-2	Fold-3	Mean
SupPR	26.85	32.73	33.48	28.40	30.37
InMeMo	28.13	38.31	37.94	33.08	34.37
PromptHub N=16	38.56	46.54	45.34	39.23	42.41

Appendix

Table 9: Results of Multi-Objective Segmentation Experiments.

Panel of Figure 10 — Figure 10: Visualization of attention map for $N=2$ .

Figure 11: Comparative visualization of our method against the existing state-of-the-art method for Foreground Segmentation and Single-Object Detection and Colorization tasks.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning
Apr 4, 2026Tianci Luo, Haohao Pan, Jinpeng Wang +5In-Context Learning Retrieval-Augmented ICL
Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Computer Science and Engineering, Northeastern University · Harbin Institute of Technology, Shenzhen
T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
Nov 20, 2025Shao-Jun Xia, Huixin Zhang, Zhengzhong TuVision-Language Models In-Context Learning
Duke University · Texas A&M University
Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts
Oct 22, 2026Nimrod Shabtay, Asaf Yehudai, Avihu Dekel +2Multimodal ICL In-Context Learning
IBM Research · Tel-Aviv University

Apr 4, 2026J/K move · Enter open · S save

Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

Tianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian+4

Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Computer Science and Engineering, Northeastern University · Harbin Institute of Technology, Shenzhen

In-Context Learning Retrieval-Augmented ICL Visual Prompt Learning

Open paper AskPDF arXiv

Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR (Label-aware Prompt Retrieval), which highlights the role of labels in prompt selection. Our framework first designs an image-label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR.

Abstract

In-Context Learning

Explore similar work

Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts