Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and segmentation. Typically, prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific prompt tokens and utilizes a dynamic token dispatching mechanism at various prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.
Figures & tables
Figure 1: Illustration of the proposed pMoE framework. Here, we demonstrate how the dispatcher handles tokens and produces integrated prompt tokens for Expert k , with the same method applied to other experts as well. The dynamic dispatching method takes expert prompt tokens from all experts and the state of the current expert as inputs, and outputs dispatching weights for controlling portions to integrate prompt tokens for the next layer. Different colors represent distinct weight groups, applied to corresponding expert prompt tokens, yielding different integrated prompt tokens. This dynamic dispatching mechanism ensures communication and interaction among diverse experts, making the model contribute the most relevant knowledge to the final output.
Method
Natural (7)
Specialized (4)
Structured (8)
Average
VPT ( Jia et al., 2022 )
78.48
82.43
54.98
69.42
EXPRES ( Das et al., 2023 )
79.70
84.00
55.00
70.21
SNF ( Wang et al., 2023 )
83.79
86.13
59.61
74.10
Bi-AdaptFormer ( Jie et al., 2023 )
82.11
86.40
62.43
74.73
LSPT ( Mo et al., 2024b )
85.26
88.57
66.25
77.95
LSPT + pMoE (ours)
87.18
90.25
69.32
80.31
Table 1: Comparison results of visual prompt tuning of supervised ImageNet-21K ViT-B/16 weights on VTAB-1K benchmarks. Numbers in ( ⋅ ) denote the number of downstream datasets.
Method
Total Params
CUB
Flowers
Cars
Dogs
NABirds
Nature
Specialized
Structured
VPT ( Jia et al., 2022 )
1.02X
82.67
94.41
79.18
83.33
75.99
70.27
83.04
42.38
VPT ( Jia et al., 2022 )
1.04X
82.95
94.65
79.37
83.62
76.21
70.55
83.42
42.69
VPT + pMoE (ours)
1.04X
83.38
94.85
79.72
83.95
76.73
71.03
83.85
43.11
GaPT ( Yoo et al., 2023 )
1.02X
82.86
93.71
79.02
83.37
76.02
74.84
83.38
49.10
GaPT ( Yoo et al., 2023 )
1.04X
83.05
93.98
79.25
83.68
76.28
75.13
83.67
49.46
GaPT + pMoE (ours)
1.04X
83.52
94.62
79.69
84.06
76.85
75.68
84.15
50.02
Table 2: Comparison results of DINO v2 pre-trained vision transformers on FGVC and VTAB-1k datasets. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.
Method
Total
Hyper
MESAD
AMLC
APTOS
ISIC
Kvasir
LCBC
MLLB
EyePACS
Params
Polyp
Prosta
Cell
Eye
Skin
Polyp
Cell
Cell
Eye
VPT ( Jia et al., 2022 )
1.02X
62.89
43.78
35.75
57.52
50.89
66.53
42.87
35.37
48.75
VPT ( Jia et al., 2022 )
1.04X
63.15
43.97
35.98
57.83
51.06
66.92
43.11
35.76
48.97
VPT + pMoE (ours)
1.04X
65.23
45.08
37.53
59.62
52.75
68.73
45.68
37.89
50.67
GaPT ( Yoo et al., 2023 )
1.02X
65.18
45.79
37.26
59.37
51.58
67.13
45.16
36.85
51.57
GaPT ( Yoo et al., 2023 )
1.04X
65.57
46.08
37.53
59.68
51.82
67.46
45.52
37.21
51.92
Table 3: Comparison results of visual prompt tuning of DINO v2 pre-trained vision transformers on color images. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.
Method
Total
Vindr
CBIS
COVIDx
SYMH
RSNAB
CheXpert
RSNA
Params
Lung
Breast
Lung
Shoulder
Bone
Chest
Lung
VPT ( Jia et al., 2022 )
1.02X
65.73
74.61
76.18
76.86
51.72
70.85
69.25
VPT ( Mo et al., 2024b )
1.04X
66.02
74.89
76.43
77.12
51.98
71.06
69.52
VPT + pMoE (ours)
1.04X
68.31
76.75
78.56
79.38
54.23
73.58
72.81
GaPT ( Yoo et al., 2023 )
1.02X
66.92
75.15
77.25
77.25
52.83
71.37
70.29
GaPT ( Yoo et al., 2023 )
1.04X
67.37
75.52
77.83
77.79
53.46
71.85
70.78
Table 4: Comparison results of visual prompt tuning of DINO v2 pre-trained vision transformers on X-ray images. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.
Method
Total
Heide
CC-CCII
Mosmed
COVID-C
RICORD
PPMI
Tumor
Params
Eye
Chest
Chest
Chest
Chest
Brain
Brain
VPT ( Jia et al., 2022 )
1.02X
64.78
61.26
63.65
61.78
59.53
56.93
63.37
VPT ( Jia et al., 2022 )
1.04X
65.02
61.53
63.89
62.03
59.82
57.26
63.69
VPT + pMoE (ours)
1.04X
67.35
64.06
66.52
65.38
63.21
60.21
66.82
GaPT ( Yoo et al., 2023 )
1.02X
65.06
61.37
63.69
61.95
59.71
56.97
63.52
GaPT ( Yoo et al., 2023 )
1.04X
65.59
61.82
64.13
62.39
60.23
57.43
63.98
Table 5: Comparison results of visual prompt tuning of DINO v2 pre-trained vision transformers on OCT, CT, and MRI images. Total Params denotes the total number of parameters for the backbone encoder ViT-B, prompt tokens or adapter parameters, and the task heads.
Method
MAE
MoCo v3
mIoU (SS)
mIoU (MS)
mIoU (SS)
mIoU (MS)
VPT ( Jia et al., 2022 )
37.76
38.80
35.50
37.15
VPT + pMoE (ours)
38.25
39.56
36.23
38.12
GaPT ( Yoo et al., 2023 )
38.44
39.81
36.81
38.55
VPT-Deep + pMoE (ours)
39.15
40.75
37.92
39.89
LSPT ( Mo et al., 2024b )
39.72
41.51
37.92
39.73
Table 6: Comparison results of visual prompt tuning of MAE & MoCo v3 pre-trained vision transformers on ADE-20K for semantic segmentation. SS and MS denote single-scale and multi-scale, respectively.
Method
MAE
MoCo v3
mIoU (Kvasir-seg)
mIoU (Skin)
mIoU (Kvasir-seg)
mIoU (Skin)
VPT ( Jia et al., 2022 )
42.87
74.23
40.65
72.68
VPT + pMoE (ours)
43.52
75.16
42.08
73.76
GaPT ( Yoo et al., 2023 )
43.95
75.52
42.17
73.89
VPT-Deep + pMoE (ours)
45.38
77.21
43.79
75.86
LSPT ( Mo et al., 2024b )
45.81
77.63
43.92
76.27
Table 7: Comparison results of visual prompt tuning of MAE & MoCo v3 pre-trained vision transformers on Kvasir Polyp and Skin Lesion for medical segmentation. mIoU metrics on a single scale are reported.
EPTs
Dispatcher
CUB
Flowers
Cars
Dogs
NABirds
✗
✗
82.95
94.65
79.37
83.62
76.21
✓
✗
83.07
94.71
79.45
83.69
76.32
✓
✓
83.38
94.85
79.72
83.95
76.73
Table 8: Ablation studies on Expert Prompt Tokens (EPTs) and Dispatcher.
Expert 1
Expert 2
CUB
Flowers
Cars
Dogs
NABirds
MoCo v3
CLIP
83.38
94.85
79.72
83.95
76.73
DINO
CLIP
83.76
95.27
80.52
84.65
77.38
DINO
MoCo v3
83.17
94.72
79.56
83.78
76.58
DINO
MAE
83.28
94.86
79.67
83.89
76.67
MAE
CLIP
83.56
95.01
80.06
84.26
76.95
Table 9: Ablation studies on the type of mixture-of-experts models.
Method
Total Params
Natural
Specialized
Structured
VPT ( Jia et al., 2022 )
1.03X
70.86
83.76
43.05
VPT ( Jia et al., 2022 )
1.06X
71.93
84.98
44.12
VPT + pMoE (ours)
1.06X
72.87
85.97
45.06
GaPT ( Yoo et al., 2023 )
1.03X
75.42
83.97
49.78
GaPT ( Yoo et al., 2023 )
1.06X
76.67
85.06
50.86
GaPT + pMoE (ours)
1.06X
77.73
86.19
51.97
Table 10: Comparison results on ViT-L/16 weights.
Figure 2: Visualization of Mixture-of-Experts path. Our pMoE can dynamically choose a distinct, task-specific path of experts for each benchmark type, demonstrating the ability to adapt to the particularities of each task.
Table 11: Comparison results of visual prompt tuning shallow models (VPT-Shallow) on VTAB-1K benchmarks. Numbers in ( ⋅ ) denote the number of downstream datasets.
# expert
CUB
Flowers
Cars
Dogs
NABirds
3
83.17
94.76
79.51
83.75
76.45
6
83.38
94.85
79.72
83.95
76.73
9
83.43
94.87
79.75
83.98
76.78
Appendix
Table 12: Ablation studies on the number of experts in MoE-MLPs for the dispatcher.
# layer
CUB
Flowers
Cars
Dogs
NABirds
3
83.08
94.68
79.42
83.68
76.29
6
83.15
94.72
79.53
83.79
76.43
9
83.29
94.78
79.65
83.86
76.62
12
83.38
94.85
79.72
83.95
76.73
Appendix
Table 13: Ablation studies on the number of prompt layers.
Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.
Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only 21.7 - 47.0% of model parameters and achieving a 4.0x average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.
Visual prompting has emerged as a powerful method for adapting pre-trained models to new domains without updating model parameters. However, existing prompting methods typically optimize a single prompt per domain and apply it uniformly to all inputs, limiting their ability to generalize under intra and inter-domain variability, which is especially critical in the medical field. To address this, we propose APEX, an Adaptive Prompt EXtraction framework that retrieves input-specific prompts from a learnable prompt memory. The memory stores diverse, domain-discriminative prompt representations and is queried via domain features extracted from the Fourier spectrum. To learn robust and discriminative domain features, we introduce a novel Low-Frequency Feature Contrastive (LFC) learning framework that clusters representations from the same domain while separating those from different domains. Extensive experiments on two medical segmentation tasks demonstrate that APEX significantly improves generalization across both seen and unseen domains. Furthermore, it complements any existing backbones and consistently enhances performance, confirming its effectiveness as a plug-and-play prompting solution in medical fields. The code is available at https://github.com/cetinkayaevren/apex/
Evren Çetinkaya, Sangmin Lee, Jung Uk Kim +2
Technical University of Munich · Korea University · Kyung Hee University +1
Segmenting images is critical for visual understanding but demands extensive pixel-level annotations. Foundational models have enabled new paradigms for predicting new classes guided by textual prompts, without annotations from the target domain. Yet, on specialized target domains, far from the original pre-training, their performance degrades. We study the errors of existing methods under such domain-shift, finding that misclassification rather than mask generation is the main culprit. To address this, we introduce the novel problem of Few-Shot Visual Adaptation for text-prompted Segmentation. This kind of adaptation has been largely studied for image classification, but it remains unexplored for segmentation. We tackle this task with Prototype Adaptation (PrAda), a novel, parameter-efficient method that adapts a frozen text-prompted segmentation model. Our approach learns class-specific prototypes by combining fine-grained pixel features and high-level transformer representations, which are then fused with the original text-based predictions through a learned importance factor. This preserves the model's zero-shot potential while enabling strong adaptation to new domains. Experiments across semantic, instance, and panoptic segmentation on five benchmarks demonstrate that PrAda yields significant improvements over state-of-the-art and proposed baselines.