pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

Shentong Mo; Xufang Luo; Dongsheng Li

cs.CVFeb 26, 2026

pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

Authors: Shentong Mo, Xufang Luo, Dongsheng Li

Organizations: Carnegie Mellon University · Microsoft Research

Abstract

Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and segmentation. Typically, prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific prompt tokens and utilizes a dynamic token dispatching mechanism at various prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.

Figures & tables

Figure 1: Illustration of the proposed pMoE framework. Here, we demonstrate how the dispatcher handles tokens and produces integrated prompt tokens for Expert

k

, with the same method applied to other experts as well. The dynamic dispatching method takes expert prompt tokens from all experts and the state of the current expert as inputs, and outputs dispatching weights for controlling portions to integrate prompt tokens for the next layer. Different colors represent distinct weight groups, applied to corresponding expert prompt tokens, yielding different integrated prompt tokens. This dynamic dispatching mechanism ensures communication and interaction among diverse experts, making the model contribute the most relevant knowledge to the final output.

Method	Natural (7)	Specialized (4)	Structured (8)	Average
VPT ( Jia et al., 2022 )	78.48	82.43	54.98	69.42
EXPRES ( Das et al., 2023 )	79.70	84.00	55.00	70.21
SNF ( Wang et al., 2023 )	83.79	86.13	59.61	74.10
Bi-AdaptFormer ( Jie et al., 2023 )	82.11	86.40	62.43	74.73
LSPT ( Mo et al., 2024b )	85.26	88.57	66.25	77.95
LSPT + pMoE (ours)	87.18	90.25	69.32	80.31

Table 1: Comparison results of visual prompt tuning of supervised ImageNet-21K ViT-B/16 weights on VTAB-1K benchmarks. Numbers in (

\cdot

) denote the number of downstream datasets.

Method	Total Params	CUB	Flowers	Cars	Dogs	NABirds	Nature	Specialized	Structured
VPT ( Jia et al., 2022 )	1.02X	82.67	94.41	79.18	83.33	75.99	70.27	83.04	42.38
VPT ( Jia et al., 2022 )	1.04X	82.95	94.65	79.37	83.62	76.21	70.55	83.42	42.69
VPT + pMoE (ours)	1.04X	83.38	94.85	79.72	83.95	76.73	71.03	83.85	43.11
GaPT ( Yoo et al., 2023 )	1.02X	82.86	93.71	79.02	83.37	76.02	74.84	83.38	49.10
GaPT ( Yoo et al., 2023 )	1.04X	83.05	93.98	79.25	83.68	76.28	75.13	83.67	49.46
GaPT + pMoE (ours)	1.04X	83.52	94.62	79.69	84.06	76.85	75.68	84.15	50.02

Table 2: Comparison results of DINO v2 pre-trained vision transformers on FGVC and VTAB-1k datasets. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.

Method	Total	Hyper	MESAD	AMLC	APTOS	ISIC	Kvasir	LCBC	MLLB	EyePACS
Method	Params	Polyp	Prosta	Cell	Eye	Skin	Polyp	Cell	Cell	Eye
VPT ( Jia et al., 2022 )	1.02X	62.89	43.78	35.75	57.52	50.89	66.53	42.87	35.37	48.75
VPT ( Jia et al., 2022 )	1.04X	63.15	43.97	35.98	57.83	51.06	66.92	43.11	35.76	48.97
VPT + pMoE (ours)	1.04X	65.23	45.08	37.53	59.62	52.75	68.73	45.68	37.89	50.67
GaPT ( Yoo et al., 2023 )	1.02X	65.18	45.79	37.26	59.37	51.58	67.13	45.16	36.85	51.57
GaPT ( Yoo et al., 2023 )	1.04X	65.57	46.08	37.53	59.68	51.82	67.46	45.52	37.21	51.92

Table 3: Comparison results of visual prompt tuning of DINO v2 pre-trained vision transformers on color images. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.

Method	Total	Vindr	CBIS	COVIDx	SYMH	RSNAB	CheXpert	RSNA
Method	Params	Lung	Breast	Lung	Shoulder	Bone	Chest	Lung
VPT ( Jia et al., 2022 )	1.02X	65.73	74.61	76.18	76.86	51.72	70.85	69.25
VPT ( Mo et al., 2024b )	1.04X	66.02	74.89	76.43	77.12	51.98	71.06	69.52
VPT + pMoE (ours)	1.04X	68.31	76.75	78.56	79.38	54.23	73.58	72.81
GaPT ( Yoo et al., 2023 )	1.02X	66.92	75.15	77.25	77.25	52.83	71.37	70.29
GaPT ( Yoo et al., 2023 )	1.04X	67.37	75.52	77.83	77.79	53.46	71.85	70.78

Table 4: Comparison results of visual prompt tuning of DINO v2 pre-trained vision transformers on X-ray images. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.

Method	Total	Heide	CC-CCII	Mosmed	COVID-C	RICORD	PPMI	Tumor
Method	Params	Eye	Chest	Chest	Chest	Chest	Brain	Brain
VPT ( Jia et al., 2022 )	1.02X	64.78	61.26	63.65	61.78	59.53	56.93	63.37
VPT ( Jia et al., 2022 )	1.04X	65.02	61.53	63.89	62.03	59.82	57.26	63.69
VPT + pMoE (ours)	1.04X	67.35	64.06	66.52	65.38	63.21	60.21	66.82
GaPT ( Yoo et al., 2023 )	1.02X	65.06	61.37	63.69	61.95	59.71	56.97	63.52
GaPT ( Yoo et al., 2023 )	1.04X	65.59	61.82	64.13	62.39	60.23	57.43	63.98

Table 5: Comparison results of visual prompt tuning of DINO v2 pre-trained vision transformers on OCT, CT, and MRI images. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.

Method	MAE		MoCo v3
Method	mIoU (SS)	mIoU (MS)	mIoU (SS)	mIoU (MS)
VPT ( Jia et al., 2022 )	37.76	38.80	35.50	37.15
VPT + pMoE (ours)	38.25	39.56	36.23	38.12
GaPT ( Yoo et al., 2023 )	38.44	39.81	36.81	38.55
VPT-Deep + pMoE (ours)	39.15	40.75	37.92	39.89
LSPT ( Mo et al., 2024b )	39.72	41.51	37.92	39.73

Table 6: Comparison results of visual prompt tuning of MAE & MoCo v3 pre-trained vision transformers on ADE-20K for semantic segmentation. SS and MS denote single-scale and multi-scale, respectively.

Method	MAE		MoCo v3
Method	mIoU (Kvasir-seg)	mIoU (Skin)	mIoU (Kvasir-seg)	mIoU (Skin)
VPT ( Jia et al., 2022 )	42.87	74.23	40.65	72.68
VPT + pMoE (ours)	43.52	75.16	42.08	73.76
GaPT ( Yoo et al., 2023 )	43.95	75.52	42.17	73.89
VPT-Deep + pMoE (ours)	45.38	77.21	43.79	75.86
LSPT ( Mo et al., 2024b )	45.81	77.63	43.92	76.27

Table 7: Comparison results of visual prompt tuning of MAE & MoCo v3 pre-trained vision transformers on Kvasir Polyp and Skin Lesion for medical segmentation. mIoU metrics on a single scale are reported.

EPTs	Dispatcher	CUB	Flowers	Cars	Dogs	NABirds
✗	✗	82.95	94.65	79.37	83.62	76.21
✓	✗	83.07	94.71	79.45	83.69	76.32
✓	✓	83.38	94.85	79.72	83.95	76.73

Table 8: Ablation studies on Expert Prompt Tokens (EPTs) and Dispatcher.

Expert 1	Expert 2	CUB	Flowers	Cars	Dogs	NABirds
MoCo v3	CLIP	83.38	94.85	79.72	83.95	76.73
DINO	CLIP	83.76	95.27	80.52	84.65	77.38
DINO	MoCo v3	83.17	94.72	79.56	83.78	76.58
DINO	MAE	83.28	94.86	79.67	83.89	76.67
MAE	CLIP	83.56	95.01	80.06	84.26	76.95

Table 9: Ablation studies on the type of mixture-of-experts models.

Method	Total Params	Natural	Specialized	Structured
VPT ( Jia et al., 2022 )	1.03X	70.86	83.76	43.05
VPT ( Jia et al., 2022 )	1.06X	71.93	84.98	44.12
VPT + pMoE (ours)	1.06X	72.87	85.97	45.06
GaPT ( Yoo et al., 2023 )	1.03X	75.42	83.97	49.78
GaPT ( Yoo et al., 2023 )	1.06X	76.67	85.06	50.86
GaPT + pMoE (ours)	1.06X	77.73	86.19	51.97

Table 10: Comparison results on ViT-L/16 weights.

Figure 2: Visualization of Mixture-of-Experts path. Our pMoE can dynamically choose a distinct, task-specific path of experts for each benchmark type, demonstrating the ability to adapt to the particularities of each task.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

\mathbf{Z}_{\text{expert}_{k}}^{0}=\text{Patchifier}_{\text{expert}_{k}}(\mathbf{X})

Appendix

Algorithm 1 pMoE : Mixture-of-Experts Prompt Tuning Framework

Method	Total Params	Natural (7)	Specialized (4)	Structured (8)	Average
VPT-Shallow ( Jia et al., 2022 )	1.01X	67.34	82.26	37.55	57.94
VPT-Shallow ( Jia et al., 2022 )	1.02X	67.51	82.39	37.72	58.10
VPT-Shallow + pMoE (ours)	1.02X	68.72	83.81	39.58	59.63

Appendix

Table 11: Comparison results of visual prompt tuning shallow models (VPT-Shallow) on VTAB-1K benchmarks. Numbers in (

\cdot

) denote the number of downstream datasets.

# expert	CUB	Flowers	Cars	Dogs	NABirds
3	83.17	94.76	79.51	83.75	76.45
6	83.38	94.85	79.72	83.95	76.73
9	83.43	94.87	79.75	83.98	76.78

Appendix

Table 12: Ablation studies on the number of experts in MoE-MLPs for the dispatcher.

# layer	CUB	Flowers	Cars	Dogs	NABirds
3	83.08	94.68	79.42	83.68	76.29
6	83.15	94.72	79.53	83.79	76.43
9	83.29	94.78	79.65	83.86	76.62
12	83.38	94.85	79.72	83.95	76.73

Appendix

Table 13: Ablation studies on the number of prompt layers.

Visual Prompt Learning

Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.

See all 122 papers

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
Oct 1, 2026Damiano Marsili, Raphi Kang, Aditya Mehta +2Mixture of Experts Mixture-of-Experts Models
California Institute of Technology
From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation
Apr 19, 2026Evren Çetinkaya, Sangmin Lee, Jung Uk Kim +2Domain Generalization Medical Image Segmentation
Technical University of Munich · Korea University · Kyung Hee University +1
PrAda: Few-Shot Visual Adaptation for Text-Prompted Segmentation
May 19, 2026Gabriele Rosi, Fabio Cermelli, Carlo Masone +1VLM Adaptation Text-Guided Image Segmentation
Politecnico di Torino · Focoos AI

Oct 1, 2026J/K move · Enter open · S save

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

Damiano Marsili, Raphi Kang, Aditya Mehta, Pietro Perona+1

California Institute of Technology

Mixture of Experts Mixture-of-Experts Models Multimodal Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only 21.7 - 47.0% of model parameters and achieving a 4.0x average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.

Abstract

Visual Prompt Learning

Explore similar work

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

PrAda: Few-Shot Visual Adaptation for Text-Prompted Segmentation