Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization

Zheng Lin; Shaoke Fang; Yuxin Zhang; Jinfeng Xu; Zihan Fang; Zhe Chen; Wei Ni; Jun Luo

cs.LGSep 30, 2026

Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization

Authors: Zheng Lin, Shaoke Fang, Yuxin Zhang, Jinfeng Xu, Zihan Fang, Zhe Chen, Wei Ni, Jun Luo, +1 more

Organizations: Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Luxembourg · Department of Computer Science, Peking University, China · Institute of Space Internet, Fudan University, China · Department of Electrical and Computer Engineering, The University of British Columbia, Canada · Department of Computer Science, City University of Hong Kong, Hong Kong, China · School of Engineering, Edith Cowan University, Australia · College of Computing and Data Science, Nanyang Technological University, Singapore

Abstract

While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ranking, which isolates the evaluation of individual experts and ignores the intricate inter-expert dependencies introduced by the MoE gating network. In this paper, we propose DS-MoE, a theoretically grounded framework that redefines expert selection via difference-of-submodular (DS) optimization. By analyzing the second-order Taylor expansion of the loss degradation, we reveal functional duality within expert combinations: redundancy (where experts encode overlapping representations) and synergy (where experts provide complementary error cancellation). To navigate this duality, we mathematically decouple redundancy reduction from synergy maximization by formulating the selection objective as a DS function. Furthermore, we devise a tailored majorization-minimization (MM) algorithm with provable monotonicity guarantees to efficiently identify the optimal expert subset. Extensive experiments demonstrate that DS-MoE effectively preserves indispensable expert combinations, achieving superior performance compared to the state-of-the-art baselines.

Figures & tables

Figure 1: Overview of the DS-MoE framework.

Panel of Figure 2 — Figure 2: First-order gradients versus second-order Hessian. (a) First-order proxy fits a flat tangent plane at local stationary points, failing to distinguish between experts. (b) Hessian captures steep curvature, revealing the massive penalty of removing expert 1 ( $\Delta\mathcal{L}(\mathbf{m})=100$ ) versus expert 2 ( $\Delta\mathcal{L}(\mathbf{m})=1$ ).

Model	Method	Test Accuracy (%) $\uparrow$					Convergence Rounds $\downarrow$
Model	Method	AGNews	PIQA	HellaSwag	MMLU	Avg.	AGNews	PIQA	HellaSwag	MMLU	Avg.
Switch-Base-32	Rand-MoE	87.3 $\pm$ 0.3	68.1 $\pm$ 0.8	65.9 $\pm$ 0.7	39.8 $\pm$ 1.1	65.3 $\pm$ 0.7	84 $\pm$ 6	26 $\pm$ 3	28 $\pm$ 4	34 $\pm$ 5	43.0 $\pm$ 4.5
	S-MoE	92.6 $\pm$ 0.2	74.3 $\pm$ 0.5	71.8 $\pm$ 0.4	46.4 $\pm$ 0.6	71.3 $\pm$ 0.4	59 $\pm$ 3	15 $\pm$ 2	17 $\pm$ 2	19 $\pm$ 3	27.5 $\pm$ 2.5
	SEER-MoE	93.1 $\pm$ 0.3	75.2 $\pm$ 0.4	72.7 $\pm$ 0.5	46.9 $\pm$ 0.5	72.0 $\pm$ 0.4	49 $\pm$ 3	12 $\pm$ 1	13 $\pm$ 2	17 $\pm$ 2	22.8 $\pm$ 2.0
	DiEP	93.3 $\pm$ 0.2	75.8 $\pm$ 0.3	73.2 $\pm$ 0.4	47.5 $\pm$ 0.3	72.5 $\pm$ 0.3	48 $\pm$ 2	12 $\pm$ 2	12 $\pm$ 1	15 $\pm$ 2	21.8 $\pm$ 1.8
	DS-MoE	94.7 $\pm$ 0.1	78.6 $\pm$ 0.2	75.6 $\pm$ 0.2	51.8 $\pm$ 0.3	75.2 $\pm$ 0.2	32 $\pm$ 1	8 $\pm$ 1	10 $\pm$ 1	11 $\pm$ 2	15.3 $\pm$ 1.3

Table 1: Experimental results on test accuracy (%) (

\uparrow

) and convergence rounds (

\downarrow

). We evaluate DS-MoE and four baselines on Switch-Base-32, Switch-Base-64, and DeepSeek-MoE-16B.

Model	$k$	Method	Test Accuracy (%) $\uparrow$					Convergence Rounds $\downarrow$
Model	$k$	Method	AGNews	PIQA	HellaSwag	MMLU	Avg.	AGNews	PIQA	HellaSwag	MMLU	Avg.
DeepSeek-16B	1	Rand-MoE	83.2 $\pm$ 0.6	66.8 $\pm$ 1.2	65.9 $\pm$ 0.9	28.9 $\pm$ 1.5	61.2 $\pm$ 1.1	162 $\pm$ 14	59 $\pm$ 7	56 $\pm$ 9	57 $\pm$ 11	83.5 $\pm$ 10.3
		S-MoE	90.1 $\pm$ 0.5	73.4 $\pm$ 0.9	72.1 $\pm$ 0.7	37.7 $\pm$ 1.2	68.3 $\pm$ 0.8	118 $\pm$ 9	26 $\pm$ 4	23 $\pm$ 5	29 $\pm$ 6	49.0 $\pm$ 6.0
		SEER-MoE	90.9 $\pm$ 0.4	75.7 $\pm$ 0.7	73.2 $\pm$ 0.8	39.2 $\pm$ 0.9	69.8 $\pm$ 0.7	112 $\pm$ 6	19 $\pm$ 4	19 $\pm$ 3	21 $\pm$ 4	42.8 $\pm$ 4.3
		DiEP	91.6 $\pm$ 0.3	75.9 $\pm$ 0.6	73.8 $\pm$ 0.7	40.8 $\pm$ 0.8	70.5 $\pm$ 0.6	104 $\pm$ 5	18 $\pm$ 3	17 $\pm$ 4	18 $\pm$ 3	39.3 $\pm$ 3.8
		DS-MoE	93.2 $\pm$ 0.3	79.8 $\pm$ 0.4	76.3 $\pm$ 0.3	43.9 $\pm$ 0.5	73.3 $\pm$ 0.4	66 $\pm$ 3	12 $\pm$ 2	13 $\pm$ 2	15 $\pm$ 3	26.5 $\pm$ 2.5

Table 2: Robustness and Scalability Analysis on DeepSeek-MoE-16B. The training performance versus (a) expert sparsity

k

with

N=64

and (b) expert pool size

N

with

k=2

Panel of Figure 3 — Figure 3: Comprehensive ablation study of the DS-MoE framework on DeepSeek-MoE-16B with $k=2$ and $N=64$ . (a)-(d): Effectiveness of second-order Hessian over first-order proxies. (e)-(h): Necessity of DS decomposition. (i)-(l): Optimization superiority of the proposed MM algorithm.

Explore similar work

DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
May 31, 2026Jiarui Feng, Hanqing Zeng, Karish Grover +11Mixture-Of-Experts Architectures Mixture-Of-Experts
Meta MRS · Washington University in St. Louis · Carnegie Mellon University +1
ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
Jun 1, 2026Heng Zhao, Zilei Shao, Guy Van den Broeck +1Large Language Model Routing Probabilistic Inference
Department of Computer Science, University of Virginia, Charlottesville, USA · Department of Computer Science, University of California, Los Angeles, Los Angeles, USA
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
Aug 11, 2026Gongli Zhang, Zhulin Liu, C. L. Philip ChenMixture-Of-Experts Unified Framework
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510335, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou 510641, China

May 31, 2026, cs.AIJ/K move · Enter open · S save

DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts

Jiarui Feng, Hanqing Zeng, Karish Grover, Ruizhong Qiu+10

Meta MRS · Washington University in St. Louis · Carnegie Mellon University+1

Mixture-Of-Experts Architectures Mixture-Of-Experts Localized Lora-Moe

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge. Prior work shows that fine-grained experts enlarge the space of expert combinations and improve flexibility, but they also impose substantial routing overhead, creating a new scalability bottleneck. In this paper, we explore a complementary axis for scaling -- how expert outputs are aggregated. We theoretically show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router, and enables possible multi-step reasoning within a single MoE layer. To this end, we propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts. Extensive experiments under standard language modeling settings show that DAG-MoE consistently improves performance in both pretraining and fine-tuning, surpassing traditional MoE baselines.

Abstract

Explore similar work

DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts

ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts

Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation