Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Xing Chen; Hengshuai Yao

cs.LGSep 4, 2026

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Authors: Xing Chen, Hengshuai Yao

Abstract

Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top- $k$ : reducing $k$ at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top $k_1$ experts while normalizing by the probability mass of the top $k_2$ experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with $k_2=16$ , while halving routed-expert compute. The result replicates on the $11\times$ larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different $k_2$ , cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.

Figures & tables

	Qwen3.6	Qwen3.5
	35B-A3B	397B-A17B
Layers (all MoE)	40	60
Hidden size	2048	4096
Experts per layer $E$	256	512
Trained $k$	8	10
Expert interm. size	512	1024

Table 1: The two models. “Act. expert params” counts routed experts activated per token plus the always-on shared expert, summed over layers. Router mass and gain are measured on WikiText.

Figure 1: Routing structure of Qwen3.6-35B-A3B measured over 500K tokens per domain. (a) Cross-domain (WikiText vs. code) Jensen–Shannon divergence of the per-layer expert distribution rises with depth; full-attention layers are marked separately. (b) Fraction of routing mass retained when keeping only the top-

N

experts per layer; shaded bands span the best and worst layer. Even keeping half the experts loses 11–19% of routing mass, and the worst layer far more.

	WikiText		Code
Routing rule	PPL	$\Delta$	PPL	$\Delta$
top- $k$ (native)	6.64	—	2.59	—
top- $k$ , uniform $w$	9.95	+49.7%	3.11	+20.3%
sampled without repl.	27.22	+310%	6.09	+135%
uniformly random	117.39	+1667%	18.69	+623%

Table 2: Routing ablations at fixed compute (8 experts activated in every row) on Qwen3.6-35B-A3B. Replacing the router’s weights with uniform ones is far less damaging than replacing its choices .

Domain	rank 1	top-4	top-6	top-8	top-16
WikiText	0.046	0.122	0.155	0.182	0.262
Code	0.046	0.116	0.146	0.170	0.245

Table 3: Cumulative router softmax mass by rank on Qwen3.6-35B-A3B, averaged over tokens and layers. The top-8 of 256 experts carry under a fifth of the total mass, so the renormalization in Eq. 1 applies a large,

k

-dependent gain.

Qwen3.6-35B-A3B (trained $k{=}8$ , $E{=}256$ )
Configuration	WikiText	Held-out	Code
$k_{1}{=}8$ native	6.643	6.986	2.585
$k_{1}{=}6$ , $k_{2}{=}6$	+4.08%	+4.69%	+1.72%
$k_{1}{=}6$ , $k_{2}{=}8$	–1.61%	–0.57%	+0.68%
$k_{1}{=}6$ , $k_{2}{=}16$	+1.81%	+1.42%	+3.29%
$k_{1}{=}6$ , $k_{2}{=}32$	+23.25%	+19.04%	+13.00%

Table 4: Perplexity relative to the native baseline (absolute PPL in the native rows). Held-out is a WikiText split disjoint from the split used to choose

k_{2}

. Perplexity is minimized at

k_{2}

equal to the trained

k

on both models and all three corpora.

Figure 2: (a) WikiText perplexity (blue, left) and MMLU accuracy (orange, right) as a function of the reference set size

k_{2}

k_{1}{=}6

on Qwen3.6-35B-A3B; stars mark the optima, dashed lines the native top-8 baselines. The two metrics disagree: perplexity is minimized at

k_{2}{=}8

, MMLU at

k_{2}{=}16

. (b) MMLU change versus

k_{2}/k

for both models at two values of

k_{1}

each; hollow markers are significant at

p<0.05

under a paired McNemar test, filled markers are not. Shrinking the reference set together with

k_{1}

(leftmost point of each curve) is significantly harmful in three of four settings;

k_{2}\geq k

recovers it. The

k_{2}{=}E

endpoint is off-scale and omitted.

Qwen3.6-35B-A3B , $n{=}2000$
Configuration	Acc.	$\Delta$ (pp)	flips $-/+$	McNemar $p$
$k_{1}{=}8$ native	81.65%	—	—	—
$k_{1}{=}6$ , $k_{2}{=}6$	79.05%	–2.60	82 / 30	$9.3\times 10^{-7}$
$k_{1}{=}6$ , $k_{2}{=}8$	80.55%	–1.10	53 / 31	0.021
$k_{1}{=}6$ , $k_{2}{=}16$	82.40%	+0.75	74 / 89	0.27
$k_{1}{=}6$ , $k_{2}{=}256$	54.20%	–27.45	637 / 88	$10^{-103}$

Table 5: MMLU, 5-shot, greedy, paired McNemar test against each model’s native baseline. “flips” counts questions the baseline answered correctly and the variant did not, versus the reverse. Bold rows halve or nearly halve activated routed experts and are statistically indistinguishable from the baseline.

Qwen3.6-35B-A3B , $n{=}500$
Configuration	Acc.	$\Delta$ (pp)	McNemar $p$	trunc.
$k_{1}{=}8$ native	95.00%	—	—	2.8%
$k_{1}{=}6$ , $k_{2}{=}6$	92.80%	–2.20	0.027	6.0%
$k_{1}{=}6$ , $k_{2}{=}8$	95.00%	+0.00	1.00	2.2%
$k_{1}{=}4$ , $k_{2}{=}4$	88.40%	–6.60	$10^{-6}$	10.4%
$k_{1}{=}4$ , $k_{2}{=}16$	94.40%	–0.60	0.68	1.4%

Table 6: GSM8K, 0-shot, greedy decoding, paired McNemar test against each model’s native baseline. “trunc.” is the fraction of generations that reach the 512-token limit without emitting an answer. Bold rows halve the activated experts with no detectable loss; the default reduction (

k_{2}{=}k_{1}

) is significantly worse and truncates more often.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Configuration	Acc.	$\Delta$ (pp)	McNemar $p$
$k_{1}{=}8$ native	80.40%	—	—
$k_{1}{=}6$ , $\alpha{=}1.00$	78.10%	–2.30	0.0018
$k_{1}{=}6$ , $\alpha{=}0.90$	78.60%	–1.80	0.015
$k_{1}{=}6$ , $\alpha{=}0.85$	78.60%	–1.80	0.020
$k_{1}{=}6$ , $\alpha{=}0.80$	79.10%	–1.30	0.079
$k_{1}{=}6$ , $\alpha{=}0.75$	78.80%	–1.60	0.033

Appendix

Table 7: Gemma-4-26B-A4B, MMLU

n{=}1000

, scalar-gain scan (preliminary protocol). Only

\alpha{=}0.80

is not significantly worse than the baseline.

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 221 papers

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Aug 8, 2026Ali Janati, Kaoutar El Maghraoui, Chengke Zou +2Mixture-of-Experts Pruning Mixture-of-Experts Models
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
Shape Mutating Expert Compression:LorExperts and BTExperts
Aug 7, 2026Inesh Chakrabarti, Sourjya Roy, Bowen Bao +3Mixture-of-Experts Language Models Low-Rank Matrix Decomposition
Advanced Micro Devices

Aug 8, 2026J/K move · Enter open · S save

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

Ali Janati, Kaoutar El Maghraoui, Chengke Zou, Xinyi Luo+1

Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA

Mixture-of-Experts Pruning Mixture-of-Experts Models Mixture-of-Experts Fine-Tuning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models decouple parameter count from per-token compute, but deployment still requires hosting every expert in memory. Recent theory shows that experts whose router weights change least during fine-tuning can be pruned with provable accuracy preservation, yet the guarantee assumes full fine-tuning. We show that the signal can be elicited through a brief parameter-efficient adaptation. We fine-tune with a lightweight adapter, rank experts by the induced router change, and prune the least-changed experts in one shot. On Mixtral-8

\times

7B-Instruct, router-only LoRA trains 0.002% of parameters and retains 27.54% MMLU-Pro accuracy with half the experts removed, against roughly 16% for magnitude and random pruning. Signal quality improves monotonically with adapter size, reaching 28.76%, and declines as adaptation spreads beyond the router. Under their shared budget, IA3 reaches 28.04% while Houlsby reaches 25.39%. The criterion transfers to Qwen1.5-MoE fine-tuned for mathematical reasoning, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed. Structural pruning reduces memory by 49% and per-token latency by 37%. Lightweight router sensitivity therefore makes provably motivated, task-conditioned expert pruning practical at scale.

Abstract

Mixture-of-Experts Language Models

Explore similar work

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Shape Mutating Expert Compression:LorExperts and BTExperts