Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling

Sanghyeok Chu; Pyunghwan Ahn; Gwangmo Song; SeungHwan Kim; Honglak Lee; Bohyung Han

cs.CVApr 15, 2026

Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling

Authors: Sanghyeok Chu, Pyunghwan Ahn, Gwangmo Song, SeungHwan Kim, Honglak Lee, Bohyung Han

Organizations: ECE, Seoul National University · LG AI Research · University of Michigan · IPAI, Seoul National University

Abstract

Sparse Upcycling provides an efficient way to initialize a Mixture-of-Experts (MoE) model from pretrained dense weights instead of training from scratch. However, since all experts start from identical weights and the router is randomly initialized, the model suffers from expert symmetry and limited early specialization. We propose Cluster-aware Upcycling, a strategy that incorporates semantic structure into MoE initialization. Our method first partitions the dense model's input activations into semantic clusters. Each expert is then initialized using the subspace representations of its corresponding cluster via truncated SVD, while setting the router's initial weights to the cluster centroids. This cluster-aware initialization breaks expert symmetry and encourages early specialization aligned with the data distribution. Furthermore, we introduce an expert-ensemble self-distillation loss that stabilizes training by providing reliable routing guidance using an ensemble teacher. When evaluated on CLIP ViT-B/32 and ViT-B/16, Cluster-aware Upcycling consistently outperforms existing methods across both zero-shot and few-shot benchmarks. The proposed method also produces more diverse and disentangled expert representations, reduces inter-expert similarity, and leads to more confident routing behavior. Project page: https://sanghyeokchu.github.io/cluster-aware-upcycling/

Figures & tables

Figure 1: Comparison of Sparse Upcycling and Cluster-aware Upcycling. Unlike Sparse Upcycling, which inherently introduces expert symmetry, our method leverages semantic structure to initialize both experts and the router, promoting expert specialization.

Panel of Figure 2 — Figure 2: An illustration of how Cluster-aware Upcycling initializes the MoE layer. (a) Input activations are clustered to obtain whitening matrices and cluster centroids, which are used to initialize expert and router parameters, respectively. (b) The resulting MoE layer, where experts capture the subspace of their corresponding clusters, and the router aligns with the underlying semantic structure of the data.

Figure 3: Expert-ensemble self-distillation (EESD). The dense EMA ensemble (right) performs soft routing over all experts and provides stable supervision for the sparse top- $k$ MoE model (left), particularly for tokens with high routing uncertainty.

Model			MSCOCO			ImageNet-1k							VTAB
Arch.	MoE Init	Samples	I $\rightarrow$ T	T $\rightarrow$ I	Avg.	Val	V2	A	R	Sketch	ObjNet	Avg.	Natural
ViT-B/32
Dense	-	4.0B	25.5	42.5	34.0	49.6	41.9	9.7	56.7	34.9	31.0	37.3	52.4
Dense	-	4.0B+1.3B	30.8	47.5	39.2	56.7	48.5	13.9	64.0	41.2	36.1	43.4	58.3
MoE	Drop-Upcycling [ 30 ]	4.0B+1.3B	29.7	46.5	38.1	56.0	47.7	12.9	63.4	40.8	34.3	42.5	57.8
	Sparse Upcycling [ 20 ]		30.8	48.0	39.4	57.1	49.1	13.8	64.3	41.8	36.0	43.7	58.0

Table 1: Comparison of upcycling methods on CLIP ViT-B/32 and ViT-B/16 across zero-shot retrieval and classification benchmarks, measured by Recall@1 and Accuracy, respectively. Cluster-aware Upcycling achieves the best performance across most benchmarks.

Model		ImageNet-1k
Arch.	MoE Init	5-shot	10-shot	FT
Dense	-	50.4	57.1	72.8
MoE	Sparse Upcycling [ 20 ]	50.9	57.8	73.0
	Drop-Upcycling [ 30 ]	51.1	57.9	73.1
	CLIP-MoE [ 48 ]	51.3	58.0	73.2
	Cluster-aware Upcycling	51.5	58.2	73.3

Table 2: ImageNet-1k few-shot and full fine-tuning results for the upcycled ViT-B/16 model. Cluster-aware Upcycling consistently outperforms other upcycling methods.

Model		MSCOCO		ImageNet-1k
Cluster-init.	EESD	I $\rightarrow$ T	T $\rightarrow$ I	Val	10-shot
		34.9	50.9	63.0	57.8
✓		35.1	51.1	63.2	58.1
	✓	34.6	51.4	62.7	57.8
✓	✓	35.4	51.6	63.2	58.2

Table 3: Ablation study on cluster-aware initialization and EESD loss for the upcycled ViT-B/16 model.

Panel of Figure 4 — Figure 4: (a) Relative Compactness measures the overlap between intra- and inter-expert variance, where lower values indicate more disentangled subspaces. (b) Expert similarity shows pairwise cosine similarity between expert weights, with Cluster-aware Upcycling maintaining higher parameter diversity. (c) Routing entropy captures routing uncertainty, with our model achieving lower entropy and more stable expert assignments. (d) Expert utilization patterns across experts, showing balanced routing without routing collapse.

Panel of Figure 5 — Figure 5: Detailed expert utilization across mixture-of-experts layers in vision encoders. Best viewed in color.

Model	MSCOCO			ImageNet-1k							VTAB
MoE Init	I $\rightarrow$ T	T $\rightarrow$ I	Avg.	Val	V2	A	R	Sketch	ObjNet	Avg.	Natural
Drop-Upcycling ( $r$ =0.75) [ 30 ]	29.1	45.6	37.4	55.7	47.3	12.4	62.3	40.1	33.0	41.8	58.6
Drop-Upcycling ( $r$ =0.50) [ 30 ]	29.7	46.5	38.1	56.0	47.7	12.9	63.4	40.8	34.3	42.5	57.8
Drop-Upcycling ( $r$ =0.25) [ 30 ]	30.6	47.6	39.1	56.7	48.4	13.1	63.5	41.0	35.7	43.1	57.7
Sparse Upcycling [ 20 ]	30.8	48.0	39.4	57.1	49.1	13.8	64.3	41.8	36.0	43.7	58.0
CLIP-MoE [ 48 ]	29.5	46.8	38.2	56.6	48.1	14.3	64.2	41.4	35.7	43.4	58.8

Table A.1 : Additional comparison of upcycling methods for ViT-B/32. Cluster-aware Upcycling achieves the strongest overall performance across most benchmarks.

Panel of Figure 10 — Figure B.2 : Expert utilization across mixture-of-experts layers for Sparse Upcycling and Cluster-aware Upcycling. Left: Vision towers. Right: Text towers. Best viewed in color.

Panel of Figure 11 — Figure B.3 : Training dynamics and epoch-wise performance.

Panel of Figure 12 — Figure B.4 : Layer-wise expert similarity and routing entropy across MoE layers in the vision and text towers of Sparse Upcycling (blue) and Cluster-aware Upcycling (red).

Panel of Figure 13 — Figure B.5 : Expert routing probabilities for Sparse Upcycling and Cluster-aware Upcycling.

Mixture-of-Experts Expert Specialization

See all 21 papers

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
Apr 21, 2026Chaitanya Dwivedi, Binxuan Huang, Himanshu Gupta +3Efficient Language Model Training Language Model Scaling
1Amazon Stores Foundation AI · 3Anthropic · 2Carnegie Mellon University
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
Jun 15, 2026Weiqiao Shan, Ruixiang Mao, Yuang Li +10Sparse Mixture-of-Experts
Northeastern University, China · Huawei TSC, China · CUHK-Shenzhen, China +4
DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
Oct 8, 2026Yuxuan Lou, Kai Yang, Geng Zhang +2Mixture-of-Experts Models Sparse Mixture-of-Experts
School of Computing, National University of Singapore, Singapore · Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

Apr 21, 2026J/K move · Enter open · S save

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

Chaitanya Dwivedi, Binxuan Huang, Himanshu Gupta, Pratik Jayarao+2

1Amazon Stores Foundation AI · 3Anthropic · 2Carnegie Mellon University

Efficient Language Model Training Language Model Scaling Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models: frontier models routinely decouple total parameters from per-token computation through sparse expert routing. Scaling laws show that under fixed active computation, model quality scales predictably with total parameters, and MoEs realize this by increasing expert count. However, training large MoEs is expensive, as memory requirements and inter-device communication both scale with total parameter count. We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts during continued pre-training (CPT). Given a trained E-expert model, the upcycling operator constructs an mE-expert model through expert duplication and router extension while holding top-K routing fixed, preserving per-token inference cost. Duplication provides a warm initialization: the expanded model inherits the source checkpoint's learned representations, starting from a substantially lower loss than random initialization. Subsequent CPT then breaks the symmetry among duplicated experts to drive specialization. We formalize the upcycling operator and develop a theoretical framework decomposing the quality gap into a capacity term and an initialization term. We further introduce utility-based expert selection, which uses gradient-based importance scores to guide non-uniform duplication, more than tripling gap closure when CPT is limited. In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours. Comprehensive ablations across model scales, activation ratios, MoE architectures, and training budgets yield a practical recipe for deploying expert upcycling, establishing it as a principled, compute-efficient alternative to training large MoE models from scratch.

Abstract

Mixture-of-Experts Expert Specialization

Explore similar work

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition