Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

Damiano Marsili; Raphi Kang; Aditya Mehta; Pietro Perona; Georgia Gkioxari

cs.CVOct 1, 2026

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

Authors: Damiano Marsili, Raphi Kang, Aditya Mehta, Pietro Perona, Georgia Gkioxari

Organizations: California Institute of Technology

Abstract

Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only 21.7 - 47.0% of model parameters and achieving a 4.0x average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.

Figures & tables

Figure 1 : Experts specialize by modality. (A) : Fraction of each expert’s top- $K$ $(K=8)$ activations from image patches versus text tokens, ordered by optimal leaf ordering (OLO; § A.2.1 ). Most experts strongly favor one modality. (B) : Distribution of each expert’s text-token activation share. For Qwen3-VL-30B-A3B , the distribution is bimodal (Hartigan dip $=0.0136$ , $p<0.001$ ).

Figure 2 : Visual activations exhibit stronger domain structure than textual activations. We project layer-18 text-token and image-patch activations with router weights from Qwen3-VL-30B-A3B using t-SNE. (A) : Datasets are colored and separated by modality; image activations form tighter domain-specific clusters than text. (B) : All domains shown together.

Figure 3 : Visual activations organize by domain; textual activations by linguistic structure. We measure kNN purity across layers, grouping text activations by dataset or part of speech and image activations by dataset, color, or spatial position. Text activations align most with part of speech, while image activations align best with semantic domain.

Figure 4 : Semantically related object classes recruit similar experts. (A) Across all TinyImageNet class pairs, expert-set overlap $Z(a,b)$ decreases with semantic distance (Pearson $r=-0.380$ , $p<0.001$ ). (B) For 10 classes forming five semantic pairs, related classes show greater expert overlap across layers despite visual variation. Solid lines average thresholds $m\in\{3,5,8,12,16\}$ , shading denotes $\pm 1$ standard deviation, and the gray dashed line shows unrelated pairs.

Figure 5 : ExpertLens selects domain-specific experts. (A) ExpertLens -derived cohorts preferentially activate on their target domains. Bar colors denote datasets; heights show normalized cohort firing. Across most cohorts, activation is strongest on the corresponding domain. (B) Representative experts with their decoded logits and highest-activating images: E121 is medical, E76 math, and E93 remote sensing. All results use layer 18 of Qwen3-VL-30B-A3B .

Figure 6 : ExpertLens matches FFT, outperforms LoRA, and accelerates adaptation. (A) ExpertLens matches or surpasses FFT while outperforming LoRA. (B) Highlighted regions show the same stages as (A) , shown against wall-clock time. ExpertLens achieves a

4.0\times

average speedup over FFT while matching performance, and a

2.9\times

speedup over LoRA.

Figure 7 : ExpertLens outperforms matched-budget random expert selection. ExpertLens outperforms both matched-budget random selection baselines while updating the same number of experts.

Selection Method	Math	Medical	Remote Sensing	Mean
Frequency Selection	63.8	52.7	51.6	56.0
Privileged Oracle	63.0	55.5	52.1	56.9
ExpertLens	64.3	53.2	51.9	56.5

Table 1: ExpertLens approaches privileged expert selection without using domain data for expert identification.

	In-domain						Out-of-domain
Trainable Parameters	Math		Medical		Remote Sensing		Math		Medical		Remote Sensing
Selected Experts Only	52.2		49.4		48.7		44.7		47.2		51.7
+ Router Weights	63.4	(+11.2)	50.7	(+1.3)	49.3	(+0.6)	45.7	(+1.0)	54.6	(+7.4)	56.2	(+4.5)
+ QKV Projections	65.8	(+2.4)	53.1	(+2.4)	52.3	(+3.0)	45.5	(-0.2)	55.0	(+0.4)	57.7	(+1.5)
+ LM Head ( ExpertLens )	66.1	(+0.3)	54.1	(+1.0)	52.8	(+0.5)	46.1	(+0.6)	56.1	(+1.1)	59.7	(+2.0)

Table 2 : Shared decoder parameters are important for effective adaptation. Progressively training the router, attention weights, and language-model head alongside the selected experts improves performance on both in-domain and out-of-domain samples, with the full ExpertLens configuration performing best across domains.

Domain Ordering	Method	Math	Medical	Remote Sensing	Mean
Math $\rightarrow$ Med. $\rightarrow$ Rem. Sens.	FFT	62.4	52.6	51.0	55.3
Math $\rightarrow$ Med. $\rightarrow$ Rem. Sens.	ExpertLens	65.2 (+2.8)	53.2 (+0.6)	52.6 (+1.6)	57.0 (+1.7)
Med. $\rightarrow$ Math $\rightarrow$ Rem. Sens.	FFT	57.1	51.5	51.9	53.5
Med. $\rightarrow$ Math $\rightarrow$ Rem. Sens.	ExpertLens	61.3 (+4.2)	52.3 (+0.8)	52.7 (+0.8)	55.5 (+2.0)
Rem. Sens. $\rightarrow$ Med. $\rightarrow$ Math	FFT	62.6	52.9	53.1	56.2
Rem. Sens. $\rightarrow$ Med. $\rightarrow$ Math	ExpertLens	67.2 (+4.6)	51.0 (-1.9)	50.9 (-2.2)	56.4 (+0.2)

Table 3: ExpertLens is robust to domain ordering. ExpertLens achieves higher mean performance than full fine-tuning across all three domain orderings, with less variation across sequences. Parentheses denote the difference relative to FFT.

Method	Math	Medical	Remote Sensing	Mean
Random (full)	40.2	46.5	48.4	45.0
FFT	40.0	50.6	51.5	47.4
ExpertLens	41.5	51.4	51.0	48.0

Table 4: ExpertLens is competitive with full fine-tuning and outperforms random expert selection on Kimi-VL-A3B .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Sensitivity to score threshold. We vary the ExpertLens selection threshold (

S_{\mathrm{EL}}

) and report the resulting number of selected experts for each domain in Qwen3-VL-30B-A3B. Expert counts decrease smoothly as the threshold becomes more stringent. Our default threshold of

S_{\mathrm{EL}}\geq 0.9

(dashed line) selects 2,768 math, 1,103 medical, and 1,290 remote-sensing experts.

Medical	Math	Remote Sensing
ly	1	<id 220>
他	<id 220>	...
<id 220>	2	<id 4102>
方面的	0	/or
/or	(	wards
ness	=	\n

Appendix

Table 5: Representative noisy ExpertLens token classifications. Because ExpertLens infers expert specialization by classifying decoded vocabulary tokens, ambiguous tokens, token fragments, punctuation, and control tokens can occasionally be assigned to a semantic domain. We show representative examples for each domain.

Figure 9: ExpertLens-derived expert specialization persists across layers. We extend the analysis from Fig. 5 A across decoder layers of Qwen3-VL-30B-A3B . Rows correspond to ExpertLens -identified medical, math, and remote-sensing expert cohorts, and columns show routing for text tokens and image patches. Curves report the normalized dataset-hit ratio per expert for each of the six evaluation datasets; asterisks mark layers where at least one of the top-two preferred datasets belongs to the expert cohort’s corresponding domain. Across layers, expert cohorts preferentially activate on inputs from their matching domain, showing that the domain alignment observed in Fig. 5 A is not specific to a single layer.

Figure 10: Mean routing weights for the top-k selected experts from each model, on image patches and text tokens. The difference between the top-1 and top-k expert weight is larger for text tokens than image patches in all models. Routing distributions across domains is similar within each modality. Overall, no model exhibits a mean top-1 routing weight over 0.5 or a top-k routing weight below 0.05.

Figure 11: Expanding on Fig. 1 , Gemma-4 26B-A4B shows a strong modality-preference with a bimodal frequency histogram, while Kimi-VL-A3B with fewer net experts appears to consist of mostly neutral experts.

Figure 12: Robustness to image scale. We repeat the experiment in § 3.1 but vary the scale of images. Specifically, we fix the total number of patches per domain and vary only image width ( original : $[50,1031]$ px; mean : $462$ px; max : $1031$ px), so smaller images contribute fewer patches each and we sample from more images. We then count the number of image-leaning versus text-leaning experts in the different experimental runs. Cells show the fraction of experts that retain the same $\geq 50\%$ image/text side across a pair of settings. High agreement across scale changes implies modality specialization is stable under image scale change.

Figure 13: Expanding on Fig. 3 , which showed that grouping by dataset showed highest kNN purity for image activations, while grouping by POS showed highest kNN purity for text tokens. Clustering by 3 domains (medical, math, remote sensing) with images from 2 datasets per domain, we see that grouping by domain is still the most kNN-pure method in image activations, while POS narrowly exceeds grouping by domain in text tokens. All kNN purity values are higher due to the fewer number of classes. The gray dotted line indicates chance kNN purity.

Figure 14: Expanding on Fig. 4 , object level categorization can be observed in Kimi-VL-A3B and Gemma-4 26B-A4B . The left hand for each plot shows Z(a, b) plotted against the semantic distance between object class pairs (x axis). Semantically unrelated object classes share fewer experts, as with Qwen3-VL-30B-A3B . The right hand side for each plot shows the category level expert specialization compared to unrelated category pairs. Images from objects in the same semantic category share more experts than any two unrelated object classes.

Figure 15: t-SNE projections for additional layers of Qwen3-VL-30B-A3B . We visualize layers 12 and 29 of Qwen3-VL-30B-A3B , supplementing the plot included in the main paper. Across various layers we find that visual activations exhibit stronger domain structure than textual activations.

Figure 16: t-SNE projections for Gemma-4 26B-A4B . We visualize Layers 12 and 24 and find image patches are more tightly clustered than text tokens.

Figure 17: t-SNE projections for Kimi-VL-A3B . We visualize Layers 12 and 24 and find image patches are more tightly clustered than text tokens.

Domain Ordering	Method	Math	Medical	Remote Sensing	Mean
Math $\rightarrow$ Med. $\rightarrow$ Rem. Sens.	FFT	1.3	0.7	0.4	0.8
Math $\rightarrow$ Med. $\rightarrow$ Rem. Sens.	ExpertLens	0.4	0.7	0.2	0.4
Med. $\rightarrow$ Math $\rightarrow$ Rem. Sens.	FFT	0.7	0.9	0.2	0.6
Med. $\rightarrow$ Math $\rightarrow$ Rem. Sens.	ExpertLens	2.0	0.2	0.5	0.9
Rem. Sens. $\rightarrow$ Med. $\rightarrow$ Math	FFT	1.6	1.0	0.2	0.9
Rem. Sens. $\rightarrow$ Med. $\rightarrow$ Math	ExpertLens	0.1	0.1	0.3	0.2

Appendix

Table 6: Domain ordering experimental variability. We report standard deviations across three seeds for our domain ordering experiments.

Hyperparameter	Value
Training objective	GSPO
Training steps per domain	200
Batch size	32
Group size	5
Learning rate	$1\times 10^{-5}$
Optimizer	AdamW

Appendix

Table 7: Hyperparameters for the Qwen3-VL experiments.

Explore similar work

EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinMixture-Of-Experts Experts
University of California, Berkeley · Allen Institute for AI
FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
May 10, 2026Xing Han, Shravan Chaudhari, Tanvi Ranade +2Mixture-Of-Experts Architectures
Department of Computer Science Johns Hopkins University · Department of ECE Johns Hopkins University · Department of Computer Science Johns Hopkins University, Bayesian Health
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vlm-To-Vla Capability Transfer Multimodal Fusion
Li Auto Inc.

May 7, 2026, cs.CLJ/K move · Enter open · S save

EMO: Pretraining Mixture of Experts for Emergent Modularity

Ryan Wang, Akshita Bhagia, Sewon Min

University of California, Berkeley · Allen Institute for AI

Mixture-Of-Experts Experts Pretraining Modular Framework Mixtures Domain-Specific Language+1

Open paper AskPDF arXiv

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document often share a domain, EMO restricts them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint enables coherent expert groupings to emerge during pretraining using document boundaries alone. We pretrain a 1B-active, 14B-total EMO on 1T tokens. As a full model, it matches standard MoE performance. Crucially, it enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting. We further find that expert subsets in EMO specialize at semantic levels (e.g., domains such as math or code), in contrast to the low-level syntactic specialization observed in standard MoEs. Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large, sparse models and open new opportunities for composable architectures.

Abstract

Explore similar work

EMO: Pretraining Mixture of Experts for Emergent Modularity

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs