Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

Zhiyuan Wang; Han Wang; Shuo Zhang

cs.CLOct 15, 2026

Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

Authors: Zhiyuan Wang, Han Wang, Shuo Zhang

Organizations: Independent Researcher

Abstract

Running prompt tokens through fewer routed experts during prefill makes mixture-of-experts (MoE) serving cheaper, but what it damages has not been measured. We locate the damage with token-, layer- and attention-path interventions in four MoE families on short-answer question answering. At half the experts the F1 loss is mostly answer form: the model still finds the answer but no longer answers briefly. The loss is made at a few tokens: the two chat-template tokens before the answer in Qwen3-30B-A3B, and the first 8-32 tokens, where the instruction sits, in DeepSeek-V2-Lite. Keeping them at full experts removes most of the F1 loss and about half and a fifth of the increase in answer NLL. These tokens are not the familiar suspects: reduction leaves the first-token sink within 4%, and restoring super experts or massive activations recovers at most a quarter of the loss. On real prompts the localisation becomes a placement rule: with about 4% of tokens at full experts, putting them on the task interface keeps F1 within one point of the unmodified model, while the same budget scattered in the documents is about 3 points worse in both prespecified primary tests of this comparison. Time to first token falls about 14% on 11K-token prompts.

Figures & tables

Figure 1: Form and content separate. Qwen3-30B-A3B, HotpotQA, 2,000 questions; change against the unmodified model. (a, b) Reducing every prompt token mostly lowers precision; reducing only the last eight tokens lowers only precision; reducing all but the last eight lowers recall and precision alike. (c) The answer lengthens exactly when the last tokens are reduced.

Figure 2: Which tokens need their experts (Qwen3-30B-A3B, HotpotQA F1 against the unmodified model). (a) Keeping the last

N

prompt tokens full: the two-token boundary accounts for much of the format effect; a 32-token tail preserves the question as well. (b) The same number of full tokens placed last, first, or at random on this template. (c) Reducing only the named tokens to two experts: the damage of the last eight is the damage of the last two, which are redundant with each other.

Model	Uniform reduction	What is lost	Where it is made	First and last 32 full
Qwen3-30B-A3B	F1 $-$ 3.34 [ $-$ 4.45, $-$ 2.26]	form	last two tokens	$-$ 0.41 [ $-$ 1.23, +0.39]
DeepSeek-V2-Lite	F1 $-$ 8.87 [ $-$ 10.38, $-$ 7.37]	form	first 8–32 tokens (instruction or prompt start)	+0.95 [ $-$ 0.17, +2.12]
Qwen1.5-MoE-A2.7B	containment $-$ 4.50 (F1 +2.13)	content	last two tokens	containment +0.05 [ $-$ 1.25, +1.35]
OLMoE-1B-7B	F1 $-$ 5.11 [ $-$ 6.76, $-$ 3.43]	form (precision)	both ends, beyond the last two	$-$ 0.94 [ $-$ 2.25, +0.34]

Table 1: Where the loss is made, by family (HotpotQA, half the experts, earlier unmarked batch; change against the unmodified model with 95% intervals; DeepSeek’s 8–32 comes from the vLLM 0.31 prefix sweep of Section 3.3 ). Qwen1.5’s loss is measured by containment (whether the gold answer appears), since its F1 rises.

Figure 3: Why the last tokens matter in Qwen3. (a) The account. (b) Attention each token receives from the query that predicts the first answer token (log scale). (c) Increase in the loss of the model’s own answer when one attention path is blocked: the first answer token needs the last two prompt tokens to see the instruction and documents; the stop token needs the answer to see those two tokens. (d) Damage left after repairing one band of layers when the last eight tokens run two experts: the mean increase in the answer’s NLL as a share of the increase with no repair, over 300 prompts. A value above 100% means that the partial repair made the loss larger; the 95% prompt-bootstrap interval of this ratio excludes 100% for experts kept in layers 0–3 (120% [103, 141]) and 44–47 (113% [105, 121]) and for the clean state restored in layers 8–11 (126% [107, 148]). Repairing one band mixes repaired and reduced computation, a state the model sees under neither condition. The ratio is of means, which a few prompts dominate.

	DeepSeek-V2-Lite		Qwen3-30B-A3B
Is the sink disturbed? (change under uniform reduction)
First token: attention / norm / largest channel	$+0.3$ % / $-0.2$ % / $-0.1$ %		$+1.5$ % / $-3.2$ % / $-3.2$ %
Last eight tokens: attention / norm / largest channel	$-4.3$ % / $-4.3$ % / $-1.4$ %		$-0.3$ % / $-8.3$ % / $-10.0$ %
Hide first token from last eight and answer, $\Delta$ NLL (random token)	$+1.07$ ( $+0.00$ ) nats		$+40.6$ ( $+0.01$ ) nats
Super experts (layer/expert); boundary selections removed	L3/E54, L4/E38; 12.5%		L1/E68, L2/E92, L3/E82; 10.8%
	Recovery	Minus position	Recovery	Minus position

Table 2: Direct tests of sink-related explanations (HuggingFace, 200 HotpotQA questions per model; DeepSeek

6\to 3

, Qwen3

8\to 4

). Change : ratio of the statistic under uniform reduction to the unmodified model, minus one, averaged over layers 1 onward and heads (first token: attention from the last eight prompt positions, hidden norm, largest absolute channel; last eight prompt tokens: attention from the answer, mean norm and largest channel; for DeepSeek these are not its sensitive tokens). Recovery : share of the uniform-reduction increase in answer NLL removed,

\overline{\mathrm{NLL}_{\text{unif}}-\mathrm{NLL}_{\text{cond}}}/\overline{\mathrm{NLL}_{\text{unif}}-\mathrm{NLL}_{\text{orig}}}

, in percent; rules keep 32 (DeepSeek) or 8 (Qwen3) tokens full. Minus position : paired difference to keeping the sensitive positions full (post hoc, except for the two signal rules, for which it is the quantity of the prespecified rule (h)). Restorations and random full tokens mix clean and reduced states (Section 4.2 ). 95% bootstrap intervals over questions. Answer NLL rises from 1.10 to 2.64 nats (DeepSeek) and 0.22 to 1.64 (Qwen3).

Qwen3-30B-A3B, $8\to 4$
Comparison	Batch	Difference	Status
Uniform $-$ unmodified	joint / neg. / place.	$-3.51$ / $-2.15$ / $-1.83$	—
Interface-Full $-$ unmodified	neg. / place.	$+0.46$ [ $-0.62$ , $+1.57$ ] / $+0.76$ [ $-0.49$ , $+2.06$ ]	sec
	joint	$+0.78$ , 90% [ $-0.31$ , $+1.89$ ]	post
Interface-Full $-$ negative control	neg.	$\mathbf{+3.22}$ [ $+1.74$ , $+4.73$ ]	pri (rule met)
	place., five placements	$\mathbf{+3.05}$ [ $+1.47$ , $+4.68$ ]; per placement $+2.33$ to $+3.88$	pri (rule met)

Table 3: Where the full-expert budget goes (vLLM 0.31, F1 points, task-averaged, 95% bootstrap intervals over questions unless marked 90%). Batches: joint (760 prompts, H100, exploratory), neg. (negative control), interf. (interface), place. (placement; H200), A100 (2,000 long-system HotpotQA questions). Status: pri / sec prespecified primary/secondary, post post hoc; the prespecified equivalence margins were

\pm 2

. Seven placements of the negative control: one in neg., one in interf., five in place.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	NLL	F1	NLL	F1
	DeepSeek, standard		Qwen3
Unmodified	1.10	49.2	0.22	67.1
Uniform	2.64	38.9	1.64	66.0
Positional rule (first 32 / last 8)	2.31	50.7	0.95	66.4
Interface-Full	—	—	0.39	68.8
Restore super experts, all reduced tokens	2.62	37.7	1.53	65.6

Appendix

Table 4: Further results of the direct-test batch. Scores and answer NLL averaged over the 200 questions; F1 of each condition’s greedy generation.

Method	Slots	Quality	90%	Interface-Full minus	TTFT, 760	TTFT, 128	End-to-end, 128	Throughput, 128
Unmodified (reference)	1.00	0			1	1 / 1	1 / 1	1 / 1
Unmodified, unmarked prompt	1.00	$+0.39^{\geq}$ [ $-0.89$ , $+1.78$ ]	[ $-0.69$ , $+1.53$ ]		0.99	0.99 / 1.01	1.00 / 1.01	1.00 / 1.00
Full-budget control	1.00	$+0.38^{\geq}$ [ $-0.82$ , $+1.65$ ]	[ $-0.61$ , $+1.44$ ]		1.03	1.04 / 1.05	1.53 / 1.57	0.91 / 0.92
Uniform	0.50	$\mathbf{-3.51}$ [ $-5.67$ , $-1.26$ ]	[ $-5.29$ , $-1.63$ ]	$\mathbf{+4.28}$ [ $+2.11$ , $+6.50$ ]	0.89	0.86 / 0.86	0.87 / 0.87	1.22 / 1.22
KeepEnds (32, 32)	0.51	$-0.37$ [ $-1.89$ , $+1.12$ ]	[ $-1.68$ , $+0.91$ ]	$+1.15$ [ $-0.09$ , $+2.42$ ]	0.93	0.87 / 0.87	0.88 / 0.87	1.21 / 1.21
Template role	0.51	$-0.45$ [ $-2.64$ , $+1.82$ ]	[ $-2.29$ , $+1.42$ ]	$+1.23$ [ $-0.59$ , $+3.07$ ]	0.95	0.87 / 0.88	0.88 / 0.88	1.22 / 1.20

Appendix

Table 5: Quality and latency in one batch (Qwen3-30B-A3B, vLLM 0.31, one H100, 760 marked prompts,

8\to 4

on reduced tokens). Slots : routed-expert work relative to the unmodified model. Quality : F1 change against the unmodified model on the marked prompts (task-averaged, 95% interval; a positive value means a higher score than the reference row, so the unmarked prompt scores 0.39 above the marked one). 90% : the 90% interval that decides non-inferiority (

\geq

: lower bound above

-1

). Interface-Full minus : the same-question difference between Interface-Full and that method. Latency columns: median per-prompt ratio to the unmodified model in the same cycle. 760 : cycle 1, all prompts, the same prompts as the quality column. 128 : cycles 2 / 3 (restarted servers) on a fixed timing subset of 16 prompts per task, which over-weights long LongBench prompts (median prompt 7.6K tokens in the subset against 2.6K over all 760 prompts); TTFT and end-to-end latency are measured one request at a time, throughput at concurrency four. Full cycle-1 ratios are in Appendix G .

Comparison	Difference [95%]	Holm $p$
Interface-Full $-$ negative control (primary)	$\mathbf{+3.22}$ [ $+1.74$ , $+4.73$ ]	—
Inverted control $-$ unmodified	$-1.86$ [ $-3.45$ , $-0.33$ ]	0.070
Negative control $-$ equal budget, ends	$-3.50$ [ $-5.07$ , $-1.90$ ]	$<$ 0.001
Negative control $-$ unmodified	$-2.76$ [ $-4.33$ , $-1.21$ ]	0.003
Interface-Full $-$ unmodified	$+0.46$ [ $-0.62$ , $+1.57$ ]	0.63
Primary, question-first HotpotQA only	$+2.01$ [ $-1.18$ , $+5.19$ ]	0.63

Appendix

Table 6: Negative-control batch (Qwen3-30B-A3B, vLLM 0.31, one H100, 2,098 marked prompts,

8\to 4

, quality only). F1 differences on seven QA datasets, equally weighted; code is edit similarity. Intervals are unadjusted 95% bootstrap intervals with HotpotQA questions resampled across their three layouts;

p

is Holm-adjusted over the six prespecified secondary comparisons. The last three rows were added after the data were seen and are not part of the correction.

Figure 4: Prompt roles and expert budgets. Left: the tokens each rule reduces in the illustrated prompt; generated tokens retain all experts. Right: segment loss density measured with an exploratory label-free probe (Supplement S3.4; thresholds not prospectively validated), relative to segment token share; task segments carry far more of the model’s answer loss than their share of tokens. For simplicity panel (a) draws the demonstrations as wholly reduced under Interface-Full; the serving rule reduces only the documents inside them.

Claim	Model	Engine, GPU	Mask	Measured
Loss is form; where it is made (§ 3 )	Qwen3, DeepSeek-V2-Lite, OLMoE, Qwen1.5	HF hooks / vLLM 0.10.2; RTX PRO 6000 and H100 (App. L )	unmarked templates, positional	quality
Protecting task tokens; equal budget (§ 5 , App. C )	Qwen3-30B-A3B	vLLM 0.31, H100 SXM	marked spans, equal-budget positions	quality, TTFT, end-to-end, throughput (joint batch)
Necessity of the task interface (§ 5 , App. C )	Qwen3-30B-A3B	vLLM 0.31, H100	marked spans, negative and inverted controls	quality (negative-control batch)
Second placement, assistant header, DeepSeek (§ 5 , App. C )	Qwen3-30B-A3B, DeepSeek-V2-Lite	vLLM 0.31, H100	marked spans, header and negative controls	quality (interface batch)
Five placements, DeepSeek screen and prefix sweep (§ 5 , App. C )	Qwen3-30B-A3B, DeepSeek-V2-Lite	vLLM 0.31, H200	marked spans, negative controls, prefix and suffix rules	quality (placement batch)
Sinks, massive activations, super experts (§ 4.2 )	Qwen3-30B-A3B, DeepSeek-V2-Lite	HF hooks, H100 PCIe; vLLM 0.31 for the prefix sweep	uniform, positional, signal rules, restorations	answer NLL, F1, attention and activation statistics

Appendix

Table 7: Where each claim comes from. Quality and latency come from the same mask, implementation and prompts only in the joint batch. Other rows are separate batches, each compared with its own unmodified model.

Method	TTFT	End-to-end	Throughput
Unmodified, unmarked prompt	0.987	0.990	1.011
Full-budget control	1.030	1.666	0.919
Uniform	0.889	0.904	1.221
KeepEnds (32, 32)	0.932	0.927	1.208
Template role	0.946	0.938	1.209
Equal budget, ends	0.959	0.955	1.200

Appendix

Table 8: Joint batch, cycle 1 (all 760 prompts): median TTFT ratio, median end-to-end ratio and throughput ratio at concurrency four, each against the unmodified model in the same cycle.

Method	All 760	LongBench	HotpotQA
Uniform	0.89	0.85	0.96
KeepEnds (32, 32)	0.93	0.86	1.01
Template role	0.95	0.86	1.04
Equal budget, ends	0.96	0.87	1.05
Equal budget, rand	0.94	0.87	1.02
Equal budget, span	0.90	0.85	1.00

Appendix

Table 9: Cycle-1 TTFT by prompt length (joint batch, Qwen3-30B-A3B): median per-prompt ratio to the unmodified model. LongBench: 360 prompts, median 10.9K tokens; HotpotQA: 400 prompts, medians 1.8K and 2.3K for the two layouts. One server start per method; no interval across starts. Throughput was not measured by length.

Method	HP sys	HP std	HP q-first	2Wiki	LB HotpotQA	MuSiQue	MFQA	Narr.	Qasper	LCC	RepoB.
Uniform	$-2.8$	$-1.7$	$-3.4$	$-1.3$	$+2.8$	$-4.6$	$-2.9$	$-3.8$	$-2.6$	$-5.8$	$-1.4$
Equal budget, ends	$-0.7$	$+1.4$	$-1.0$	$+3.2$	$+1.4$	$+1.8$	$+0.4$	$-1.4$	$-0.2$	$-2.2$	$-0.1$
Negative control	$-2.2$	$-1.2$	$-3.1$	$-4.8$	$+0.3$	$-3.7$	$-2.1$	$-3.6$	$-3.3$	$-6.2$	$+0.4$
Inverted control	$-1.0$	$-2.9$	$-2.3$	$-0.6$	$-1.5$	$-2.6$	$-1.5$	$-1.3$	$-3.5$	$-2.2$	$+0.4$
Interface-Full	$+0.8$	$+1.9$	$-1.0$	$+2.6$	$+0.5$	$+2.0$	$-0.6$	$-2.0$	$+0.2$	$-2.1$	$-0.4$

Appendix

Table 10: Negative-control batch, per task : change against the unmodified model (F1 points; edit similarity for LCC and RepoBench-P). Descriptive only.

Comparison	Difference [95%]	Holm $p$
DeepSeek-V2-Lite, $6\to 3$ , 800 prompts
Instruction in the middle: Interface-Full $-$ equal budget, ends (primary)	$\mathbf{-1.67}$ [ $-3.56$ , $+0.19$ ]	—
Standard layout: Interface-Full $-$ equal budget, ends	$-0.62$ [ $-2.41$ , $+1.17$ ]	0.53
Interface-Full $-$ negative control	$+4.28$ [ $+1.87$ , $+6.72$ ]	0.002
Interface-Full $-$ unmodified	$+1.34$ [ $-0.64$ , $+3.36$ ]	0.38
Uniform $-$ unmodified	$-2.48$ [ $-5.05$ , $+0.07$ ]	0.17

Appendix

Table 11: Interface batch (vLLM 0.31, one H100, quality only, F1 points). Unadjusted 95% bootstrap intervals;

p

is Holm-adjusted within each model over the listed secondary comparisons. Both primaries are undecided under their prespecified rules (Appendix I ). Header only : the eight-token assistant header full and the rest of Interface-Full’s budget at random in the documents. No header : the Interface-Full mask with the header reduced and its budget moved into the documents. The last three rows are post hoc.

Qwen3 method	HP std	HP sys	HP q-first	HP tool	HP mid	2Wiki	LB HotpotQA	MuSiQue	MFQA	Narr.	Qasper
Interface-Full	$+1.9$	$+0.8$	$-1.0$	$+1.8$	$-1.8$	$+4.2$	$+1.5$	$+2.3$	$-1.2$	$-1.4$	$+0.4$
Header only	$+0.3$	$-0.3$	$-0.4$	$+4.1$	$-1.6$	$+3.7$	$+3.8$	$-2.1$	$-2.6$	$-2.8$	$-1.3$
No header	$-0.8$	$+0.2$	$-2.5$	$-3.8$	$-2.3$	$+0.7$	$+2.5$	$-3.4$	$-2.7$	$-2.4$	$+0.3$
Negative control, seed 2	$-0.6$	$-0.5$	$-3.3$	$+1.8$	$-2.0$	$+3.5$	$+0.8$	$-2.3$	$-2.0$	$-2.2$	$-2.3$
Equal budget, ends	$+1.4$	$-0.7$	$-1.0$	$+2.4$	$-1.8$	$+4.0$	$+0.9$	$+1.9$	$+1.2$	$-1.4$	$+0.6$
DeepSeek method	HP std / mid

Appendix

Table 12: Interface batch, per task : change against the unmodified model (F1 points). Descriptive only. DeepSeek rows: standard / instruction-in-the-middle HotpotQA.

Comparison	Difference [95%]	Holm $p$
DeepSeek-V2-Lite screen, uniform $-$ unmodified, 400 questions each
10 / 30 words before the instruction	$-1.45$ [ $-4.88$ , $+2.01$ ] / $-1.87$ [ $-5.10$ , $+1.33$ ]	descr.
100 / 300 words before the instruction	$+0.14$ [ $-3.22$ , $+3.49$ ] / $-0.18$ [ $-3.42$ , $+3.06$ ]	—
DeepSeek-V2-Lite, standard layout, 400 questions
Interface-Full $-$ negative control (two placements)	$+8.09$ [ $+5.05$ , $+11.26$ ]	$<$ 0.001
Uniform $-$ unmodified	$-6.79$ [ $-10.16$ , $-3.40$ ]	descr.

Appendix

Table 13: Placement batch (vLLM 0.31, one H200, quality only, F1 points). Unadjusted 95% bootstrap intervals;

p

is Holm-adjusted within each model over the listed secondary comparisons. Rows marked descr. are descriptive. First

k

full : the first

k

prompt tokens at full experts, the rest uniform; last

k

full likewise at the end.

Reduced tokens	Qwen3-30B-A3B (F1)	DeepSeek-V2-Lite (F1)	Qwen1.5-MoE (containment)	OLMoE-1B-7B (F1)
All prompt tokens (uniform)	$-$ 3.34 [ $-$ 4.45, $-$ 2.26]	$-$ 8.87 [ $-$ 10.38, $-$ 7.37]	$-$ 4.50 [ $-$ 6.20, $-$ 2.80]	$-$ 5.11 [ $-$ 6.76, $-$ 3.43]
Only the first token	—	+0.11 [ $-$ 0.54, +0.76]	$-$ 0.95 [ $-$ 1.60, $-$ 0.30]	$-$ 0.07 [ $-$ 0.67, +0.53]
Only the first 20	+0.31 [ $-$ 0.25, +0.88]	$-$ 9.45 [ $-$ 10.67, $-$ 8.19]	$-$ 1.45 [ $-$ 2.25, $-$ 0.70]	$-$ 0.69 [ $-$ 1.52, +0.13]
Only the head	+0.57 [ $-$ 0.11, +1.27]	$-$ 9.86 [ $-$ 11.17, $-$ 8.54]	$-$ 2.65 [ $-$ 3.70, $-$ 1.65]	$-$ 1.21 [ $-$ 2.18, $-$ 0.20]
Only the last two	—	+3.66 [+2.61, +4.73]	$-$ 4.40 [ $-$ 5.75, $-$ 3.00]	+0.03 [ $-$ 0.96, +1.04]
Only the last 32	$-$ 2.71 [ $-$ 3.69, $-$ 1.76]	+0.78 [ $-$ 0.52, +2.08]	$-$ 4.80 [ $-$ 6.35, $-$ 3.25]	$-$ 3.02 [ $-$ 4.47, $-$ 1.57]

Appendix

Table 14: Reducing single spans, by family (family batch, HotpotQA, 2,000 questions, half of each model’s experts; change against the unmodified model with 95% intervals). F1, except containment for Qwen1.5, whose F1 rises under reduction. “Only

X

”:

X

reduced, every other prompt token full; “keep

X

”:

X

full, every other prompt token reduced. The head is the chat prefix, the instruction and the two demonstrations. Qwen3’s single-token rows come from a separate batch (Section 3.2 ).

Question	Unmodified	Uniform $8\to 4$	KeepEnds
Qwen3-30B-A3B
What was the nickname of the hitman hired by an Italian American Criminal Organization?	The Crowned Prince of the Philadelphia Mob	The nickname of the hitman hired by an Italian American criminal organization was “The Crowned Prince of the Philadelphia Mob.”	The Crowned Prince of the Philadelphia Mob
Which movie directed by James Gunn portrayed the fictional character, Ego the Living Planet?	Guardians of the Galaxy Vol. 2	Answer: Guardians of the Galaxy Vol. 2	Guardians of the Galaxy Vol. 2
DeepSeek-V2-Lite
Tysons Galleria is located in what county?	Fairfax County	Tysons Galleria is located in Fairfax County, Virginia.	Fairfax County
In which year was the King who made the 1925 Birthday Honours born?	1865	The King who made the 1925 Birthday Honours was born in 1865.	1865

Appendix

Table 15: What a loss of form looks like. HotpotQA answers of the unmodified model, with every prompt token at half of its experts, and under KeepEnds. The reduced model still finds the answer; it no longer gives “only the short answer”.

Figure 5: Where the loss is made, by model family. HotpotQA F1 against each unmodified model when the named prompt tokens run half of their experts (2,000 questions, 95% intervals). The head is the chat prefix, the instruction and the two demonstrations. Which end carries the loss differs by model; with 32 tokens kept at each end (rightmost bar, the KeepEnds rule) no model shows a detectable loss on this task (Qwen1.5’s F1 rises).

Figure 6: The effect follows the answer boundary. (a) Reducing only the last two tokens (the last eight for the prompt end under reasoning) to

8\to 2

in four settings: it damages precision whenever those tokens are followed directly by the answer, and not when generated reasoning separates them from it (shaded). (b) With reasoning on, a prompt reduced to four experts is not separated from the unmodified model by the 95% interval in this earlier run; that alone does not establish equivalence. At two experts, prompt reduction again changes answer form.

Input Order Sensitivity in Language Models

See all 14 papers

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
Sep 4, 2026Xing Chen, Hengshuai YaoMixture-of-Experts Language Models Mixture-of-Experts Inference
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI

Sep 22, 2026J/K move · Enter open · S save

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Yuanteng Chen, Qiwei Lai, Chen Tianqi, Peisong Wang+6

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy+1

Mixture-of-Experts Pruning Mixture-of-Experts Language Models Mixture-of-Experts Inference

Open paper AskPDF arXiv

Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly many selected per token. This shift makes dynamic expert pruning an attractive route to cheaper inference. Yet existing evidence comes largely from coarser architectures and likelihood-scored multiple-choice benchmarks, leaving three central questions open in the fine-grained regime: how redundant per-token expert selection is, how effectively existing pruning methods exploit that redundancy, and what governs a model's sensitivity to pruning. We fill this gap with a systematic empirical study of twelve fine-grained MoE checkpoints spanning nine architecture families, with a core suite of eleven benchmarks covering knowledge QA, mathematics, code generation, and general reasoning. We find that expert selection is far more redundant than the field's operating points assume: uniformly retaining about two thirds of the selected experts preserves 98.8% of unpruned performance on average, requiring only a one-integer change and delivering 1.2-1.7x measured speedup across two serving backends. This simple baseline leaves little room for dynamic allocation at conservative budgets: even the best published rules differ from it by under 1% at matched expert budgets. Their value emerges under aggressive pruning, where the best rules recover up to 3.0% over uniform truncation, with gains concentrated in the generative tasks that suffer the sharpest degradation. Sensitivity to aggressive pruning also depends on the model: larger and thinking models are more resilient, whereas multimodal models are more vulnerable. Together, these findings reveal how much expert computation fine-grained MoEs can dispense with, and establish when dynamic allocation earns its complexity, informing both practical deployment and future pruning methods.

Abstract

Input Order Sensitivity in Language Models

Explore similar work

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Speculating Experts Accelerates Inference for Mixture-of-Experts