Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Geunhyoung Kim; Sung Ju Hwang

cs.LGOct 27, 2026

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Authors: Geunhyoung Kim, Sung Ju Hwang

Organizations: KAIST

Abstract

Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data. We show that channel selection needs no calibration data. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs; Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% on average across our benchmarks. Second, top-k keeps each token's k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4. Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget. We furthermore introduce a reader built entirely from the model's weights, which screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels running MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one and up to 2.69x at larger batches on the same three models.

Figures & tables

Figure 1: At heavy sparsity, calibrated selectors lose most of their accuracy on at least one task, whereas data-free top-

k

selectors stay accurate and our reader performs best and decodes faster. (a) Worst accuracy over three MoE models and three benchmarks; dashed lines use calibrated thresholds, placed at the budget they were calibrated for. See Appendix Figure 11 for the same curves with calibrated selectors placed at the reads they actually used. (b) Whole-model decoding speedup of our reader at 17.8% of reads over optimized vLLM.

Figure 2: A channel selector computes only a subset of each routed expert’s channels, and hardware charges every 32-byte sector it touches. (a) Channel

j

owns row

j

of the gate and up projections (

G_{e},V_{e}

) and column

j

of the down projection (

D_{e}

);

H

is the hidden width,

I

the number of channels per expert,

\phi

the gate activation. (b) On the same scores, a calibrated threshold

\tau

keeps six channels on token A and two on token B, while top-

k

keeps four on both. (c) A selector that needs 80 bytes spread over four sectors is charged 128 bytes.

Figure 3: Prior channel selectors, grouped by what they read before choosing channels. Dashed rows or columns are skipped; chips show each keep rule, and a dot marks budgets or rotations fitted on data. HiRE and Mixture-of-Channels (MoC) train the model for top-

k

; TopK-Guided bounds a threshold by a top-

k

window. Our reader (bold) fits nothing to the model.

Figure 4: Under a calibrated threshold, the number of kept channels changes from token to token, and selectors read less than their budget on benchmarks. (a) Share of channels each token keeps under an exact-score threshold calibrated on UltraChat for about 12%, on held-out chat and on the model’s answers to GPQA-Diamond ( Rein et al., 2024 ) , IFEval ( Zhou et al., 2023 ) , and MMLU-Pro ( Wang et al., 2024 ) ; top-

k

keeps the same share on every token (line). (b) Median kept count per 500-token window, relative to the calibrated count (about 94,000 answer tokens per model). (c) In all 72 benchmark runs, Prox and our reader with calibrated thresholds read below their budget (the diagonal).

Figure 5: On the same score, a selector’s top-

k

version matches or beats its threshold version in 40 of 43 comparisons and avoids Gemma 4’s collapse. Each dot is the accuracy of a selector’s threshold version minus that of its top-

k

version, at the budget the threshold was calibrated for, with

\pm 2

paired SE. Gray marks ties, blue top-

k

leads (the four on Gemma 4 by 38 to 45 points), and orange threshold leads (at most 2.8 points). See Appendix Figure 8 for the same comparisons with each top-

k

version run at the reads its threshold version actually used.

Figure 6: Our reader screens candidate channels with a ternary sketch, computes the top-

k

channels exactly, and recovers the rest using only the full model’s weights. Top: the full model reads every channel of a routed expert; our reader reads a ternary sketch, the

k

kept channels, and a low-rank recovery map. Middle: the three stages. Bottom: reads at 17.8% on Qwen3-Thinking.

	About 13% of reads				About 18% of reads
Selector	Target	Realized	Mean	Worst	Target	Realized	Mean	Worst
Our reader	–	12.8	94.5	86.9	–	17.8	98.1	95.0
Prox score, top- $k$	–	13.5	87.0	61.1	–	18.5	96.1	90.4
Prox as published, calibrated	13	8.5	51.5	0.0	18	12.9	68.5	12.1
WINA as published, top- $k$	–	15	2.4	0.0	–	20	28.4	1.1
WINA score, calibrated	15	7.8	0.2	0.0	20	11.2	5.0	0.0

Table 1: Our reader is the most accurate tested selector at the two lowest budgets. Mean and worst accuracy over nine model-benchmark pairs, as a percentage of the full model’s accuracy. Selectors with calibrated thresholds list the budget they were calibrated for (Target) and the reads they used (Realized); WINA as published uses top-

k

and meets its budget to the precision shown.

Figure 7: Our reader accelerates MoE blocks by 1.60 to 2.69

\times

over optimized vLLM across three post-trained models. At 17.8% of routed-expert reads, median of three processes. (a) MoE-block decode speedup; both arms share the unmodified router, and Gemma 4’s dense MLP and Qwen3.5’s shared expert are timed inside the block. (b) Whole model, batch sizes 1 to 128: right is faster per user, up is more output per GPU.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Regime	Models	Same score	Matched	Ahead
La RoSA ( 2025b )	dense, PPL	LLaMA2-7B	yes	target	top- $k$ (PPL)
Batch-wise pruning ( 2026c )	dense, CoT	2 distilled 7–8B	yes	target	top- $k$ (batch 4)
Sparsing Law ( 2025 )	dense, short tasks	0.1B, 0.8B	no	target	threshold
Prox ( 2026 )	dense, short tasks	Qwen3-8B, 14B	yes	target	threshold (+1.8)
TopK-Guided ( 2026 )	dense, short tasks	Llama-2/3, 7–8B	no	target	hybrid; top- $k$ vs. $\tau$ mixed
Sparser-MoE ( 2026d )	MoE, short tasks	OLMoE	unclear	target	top- $k$ ( $<$ 1 pt)

Appendix

Table 2: Prior keep-rule comparisons reach opposite conclusions, focus mostly on short tasks on dense models, and none runs top-

k

at the reads a threshold actually uses. Unlike most prior comparisons, ours holds the channel score fixed ( Same score ), tests long chain-of-thought reasoning on MoE models ( Regime ), and runs top-

k

both at the target budget and at realized reads ( Matched ).

Model	Total	Active	Expert	Hidden	layers	full	Sketch	Recovery
	Routed experts		Width		MoE	Always	Added memory (%)
Qwen3-Thinking	128	8	768	2048	48	–	7.6	0.50
Gemma 4	128	8	704	2816	30	dense MLP	7.0	0.35
Qwen3.5	256	8	512	2048	40	shared expert	7.0	0.47
OLMoE	64	8	1024	2048	16	–	7.3	0.50

Appendix

Table 3: Evaluated MoE models and the resident memory our reader adds. Each row gives the number of routed experts and how many each token uses, the channels per expert and the hidden width, the number of MoE layers, any module that always runs in full, and the resident memory of our reader’s ternary sketch and recovery map as a percentage of the BF16 checkpoint. OLMoE’s sketch size is computed analytically.

				Sampling
Model	Benchmark	Items	Samples	Temperature	Top- $p$	Top- $k$	Max tokens
Qwen3-Thinking	GPQA-D	198	4	0.6	0.95	20	32,768
	IFEval	541	1	0.6	0.95	20	32,768
	MMLU-Pro	1,000	1	0.6	0.95	20	32,768
Gemma 4	GPQA-D	198	4	1.0	0.95	64	32,768
	IFEval	541	1	1.0	0.95	64	32,768

Appendix

Table 4: Evaluation protocol on the post-trained models. Each model runs in thinking mode with its model card’s sampling parameters and random seed 20260923. GPQA-Diamond reports accuracy over four samples per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro reports accuracy on a fixed 1,000-question subset.

		Routed-expert reads (%)
Model	Sketch read	Sketch	Exact channels	Recovery	Metadata	Total
Qwen3-Thinking	partial	4.86	7.29	0.66	0.04	12.85
	partial	4.86	12.30	0.63	0.04	17.83
	full	7.99	20.00	0.59	0.10	28.68
	full	7.99	30.00	0.54	0.10	38.62
Gemma 4	partial	4.78	7.49	0.54	0.03	12.84

Appendix

Table 5: Most of our reader’s reads go to exact channels, and the sketch takes most of the rest. Routed-expert reads per token at batch one, as a percentage of the full model, by component at the four main budgets. At the two smallest budgets, the partial screen reads part of the ternary sketch.

Sketch read	partial	partial	full	full	full	full
Channels computed exactly (%)	7.0–7.5	12.0–12.5	20	30	50	70
Reads (% of full model)	12.8	17.8	28.7	38.7	58.6	78.5

Appendix

Table 6: Our reader reads part of the sketch at the two smallest budgets and all of it above. Each column is one budget. It gives the share of candidate channels computed exactly across each token’s routed experts and the resulting total reads, which include the ternary sketch, exact channels, recovery map, and metadata, averaged over the three post-trained models and three benchmarks.

Batch size	1	4	8
Full model	75.6	229.1	359.3
Our reader	13.9	49.9	92.3
Share	18.3%	21.8%	25.7%

Appendix

Table 7: At batch one, the profiler measures 18.3% of the full model’s DRAM reads, close to the 17.8% we count. Measured DRAM reads per MoE layer on Qwen3-Thinking in MB, averaged over 12 layers, without the router, with a cold cache.

		Realized reads (%)
Method	Calibrated for (%)	Min	Mean	Max
CATS	53.3	51.1	52.1	53.0
	66.7	63.8	65.1	66.0
Our reader, calibrated	12.8	11.2	11.8	12.4
	17.8	15.2	16.5	17.1
	28.7	24.5	26.5	27.7

Appendix

Table 8: Every calibrated threshold reads less than the budget it was calibrated for. Each row is one selector at one calibrated budget. The columns give the lowest, mean, and highest realized routed-expert reads (%) over the nine model-benchmark pairs.

Context before the window	Kept channels	Change
Question and all 6,000 earlier answer tokens	9,983	–
Question and the last 1,000 answer tokens	11,423	$+15.0$	${\scriptstyle\pm 1.2}\%$
Question only	15,422	$+56.6$	${\scriptstyle\pm 2.7}\%$
Start of the answer (different tokens)	17,283

Appendix

Table 9: The same tokens keep more channels when the earlier answer is removed. The first three rows change only the context before the 256-token windows at answer position 6,000 of Qwen3-Thinking GPQA-Diamond traces, under Prox’s threshold calibrated for 13%. Kept channels are per token, summed over 48 layers (of 294,912 candidates) and averaged over 48 traces;

\pm

is one paired SE. The last row shows the start of the answer, which has different tokens.

Model	Task	Score	Budget (%)	$\tau$	$k$	$\tau-k$
Gemma 4	GPQA-D	Prox, output-scaled	13	16.2	54.0	$-37.9$	${\scriptstyle\pm 3.8}$	$k$
		Prox	13	6.1	51.0	$-44.9$	${\scriptstyle\pm 3.5}$	$k$
	IFEval	Prox, output-scaled	13	23.1	63.6	$-40.5$	${\scriptstyle\pm 2.3}$	$k$
		Prox	13	3.5	45.8	$-42.3$	${\scriptstyle\pm 2.2}$	$k$
		TEAL	18	0.0	7.0	$-7.0$	${\scriptstyle\pm 1.1}$	–
OLMoE	BBH	Prox, output-scaled	13	25.4	26.1	$-0.7$	${\scriptstyle\pm 0.6}$	=

Appendix

Table 10: Keep-rule comparisons on generation tasks. Each row fixes one model, task, score, and budget. It gives the accuracy of the threshold version (

\tau

) and the top-

k

version (

k

), their paired difference

\pm

one standard error, and which version is ahead (

=

within two standard errors, – both collapsed).

Model	Task	Score	Budget (%)	$\tau$	$k$	$\tau-k$
Gemma 4	GPQA-D	Prox, output-scaled	13	16.2	30.3	$-14.1$	${\scriptstyle\pm 3.5}$	$k$
		Prox	13	6.1	24.2	$-18.2$	${\scriptstyle\pm 2.9}$	$k$
	IFEval	Prox, output-scaled	13	23.1	34.0	$-10.9$	${\scriptstyle\pm 2.1}$	$k$
		Prox	13	3.5	16.5	$-12.9$	${\scriptstyle\pm 1.6}$	$k$
		TEAL	18	0.0	0.2	$-0.2$	${\scriptstyle\pm 0.2}$	–
OLMoE	BBH	Prox, output-scaled	13	25.4	25.3	$+0.1$	${\scriptstyle\pm 0.6}$	=

Appendix

Table 11: The generation-task comparisons with each top-

k

version run at its threshold version’s realized reads. Columns follow Table 10 . The top-

k

version is rerun only where the threshold version’s realized reads differ from the budget by more than one percentage point. † The threshold version read within one point of its budget, so the top-

k

run at the budget is reused.

Prompt	Task	Budget (%)	$\tau$	$k$	$\tau-k$
full model	ARC-C	13	49.7	50.0	$-0.3$	${\scriptstyle\pm 0.9}$	=
	ARC-E	13	73.9	73.2	$+0.7$	${\scriptstyle\pm 0.6}$	=
	HellaSwag	13	75.4	74.9	$+0.6$	${\scriptstyle\pm 0.3}$	$\tau$
	MMLU	13	50.3	50.3	$-0.0$	${\scriptstyle\pm 0.3}$	=
	PIQA	13	77.7	76.4	$+1.3$	${\scriptstyle\pm 0.7}$	=
	ARC-C	18	51.9	51.4	$+0.5$	${\scriptstyle\pm 0.9}$	=

Appendix

Table 12: Keep-rule comparisons on Prox’s five likelihood tasks on OLMoE. Prox’s threshold and top-

k

versions, with the prompt either sparse or computed by the full model. Columns follow Table 10 . At 38%, the top-

k

version keeps 50% of inputs and the threshold version 30%; these verdicts are in parentheses and not counted.

Prompt	Task	Budget (%)	$\tau$	$k$	$\tau-k$
full model	ARC-C	13	49.7	50.0 †	$-0.3$	${\scriptstyle\pm 0.9}$	=
	ARC-E	13	73.9	73.2 †	$+0.7$	${\scriptstyle\pm 0.6}$	=
	HellaSwag	13	75.4	74.9 †	$+0.6$	${\scriptstyle\pm 0.3}$	$\tau$
	MMLU	13	50.3	49.7	$+0.5$	${\scriptstyle\pm 0.3}$	=
	PIQA	13	77.7	76.4 †	$+1.3$	${\scriptstyle\pm 0.7}$	=
	ARC-C	18	51.9	52.8	$-0.9$	${\scriptstyle\pm 0.9}$	=

Appendix

Table 13: The likelihood-task comparisons with each top-

k

version run at its threshold version’s realized reads. Columns follow Table 10 . † The threshold version read within one point of its budget, so the top-

k

run at the budget is reused.

	At the calibrated budget			At realized reads
Model	Comparisons	$\tau-k$		Comparisons	$\tau-k$
Qwen3-Thinking	4	$-0.3$	${\scriptstyle\pm 1.0}$	4	$+4.5$	${\scriptstyle\pm 1.1}$
Qwen3.5	3	$-1.7$	${\scriptstyle\pm 0.7}$	2	$-0.2$	${\scriptstyle\pm 0.8}$
OLMoE	32	$-0.3$	${\scriptstyle\pm 0.1}$	31	$+0.0$	${\scriptstyle\pm 0.1}$
Gemma 4	4	$-41.5$	${\scriptstyle\pm 1.3}$	4	$-13.3$	${\scriptstyle\pm 1.1}$

Appendix

Table 14: No model favors the threshold version at the calibrated budget. Threshold-version minus top-

k

-version accuracy, pooled over each model’s counted comparisons with inverse-variance weights, at the calibrated budget and at realized reads. The

\pm

is one SE that treats comparisons as independent; accounting for shared questions changes each SE by at most 0.1. The realized-read columns favor the threshold version (Section 2 ). Only Qwen3-Thinking leads beyond two SE there, a lead that comes from Prox on IFEval and reverses at the calibrated budget.

Figure 8: Figure 5 with each top-

k

version run at the reads its threshold version actually used. Each dot is one comparison, as in Figure 5 . When a threshold version read more than one percentage point away from its calibrated budget, the top-

k

version is rerun at those realized reads. The figure has 41 comparisons at 13% and 18% budgets; four TEAL comparisons at heavy sparsity, where both versions collapse, are left out. Threshold versions read less than their budget, so this rerun lowers the top-

k

version’s reads too, yet the top-

k

version matches or beats the threshold version in 37 of 41.

Task	Shots	Metric	Questions	Full model (%)
MMLU	5	accuracy	14,042	52.71
HellaSwag	10	normalized accuracy	10,042	79.40
ARC-E	0	accuracy	2,376	78.11
ARC-C	25	normalized accuracy	1,172	55.20
PIQA	0	accuracy	1,838	79.98

Appendix

Table 15: Prox’s five likelihood tasks on OLMoE. Few-shot examples, metric, number of questions, and full-model accuracy for each task.

Reads (%)	13.5	18.5	28.5	38.4
Full sketch, no recovery	0.553	0.277	0.107	0.051
Full sketch with recovery	0.488	0.245	0.096	0.047
Partial sketch with recovery	0.310	0.187	0.090	0.053
Prox’s 4-bit copy, output-scaled top- $k$ , recovery	0.305	0.187	0.070	0.035

Appendix

Table 16: Partial screening helps at small budgets, and recovery helps at every budget. Next-token KL divergence from the full model on held-out OLMoE documents at equal reads (lower is better). Each row is one variant of our reader.

Variant at 17.8% of reads (OLMoE, development)	KL
Full sketch, no recovery (rotated, top- $k$ over all routed experts)	0.286
Without rotation	0.348
Fixed share of channels per expert	0.491
Score from an INT4 copy of the gate projection, same top- $k$	0.811
Score from TEAL’s input magnitudes, top- $k$ inputs	1.471

Appendix

Table 17: Rotation and ranking channels across all routed experts lower KL divergence, and reading 80 of 128 coordinates is at or near the best. Next-token KL divergence from the full model on development data (lower is better). Top: at 17.8% of reads on OLMoE without recovery, the full sketch, two of its design variants, and two other scores under top-

k

. Bottom: coordinates the partial screen reads per 128-coordinate block, with recovery, at equal reads; bold marks the lowest in each row.

Figure 9: Accuracy against reads on every model and task. Each panel is one model-benchmark pair, and its vertical axis runs from zero to above the full model’s accuracy (dotted line). Blue solid lines use no calibration data. Warm dashed lines are selectors with thresholds fitted on calibration data, placed at the budget they were calibrated for. WINA as published (red solid) keeps a top-

k

per token, with a sparsity level for each projection fitted on calibration data as in TEAL. GPQA-Diamond averages four samples per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro uses a fixed subset of 1,000 questions.

Figure 10: Figure 9 with calibrated selectors placed at the reads they actually used. Selectors with calibrated thresholds move left, because they read less at test time than the budget they were calibrated for.

Figure 11: Figure 1 with calibrated selectors placed at the reads they actually used. Each curve is the worst accuracy over the nine model-benchmark pairs, as in Figure 1 , but each calibrated selector sits at the mean reads it used on the pair that gives its worst accuracy, instead of the budget it was calibrated for. Selectors with calibrated thresholds read less than that budget (Section 4.1 ), so their points move left.

Qwen3-Thinking
Method	Budget (%)	GPQA-D	IFEval	MMLU-Pro
Full model	100	70.8	87.8	80.5
Our reader	12.8	63.8	85.0	76.9
	17.8	67.3	87.2	78.2
	28.7	67.9	85.4	78.6
	38.7	70.8	86.1	80.3

Appendix

Table 18: Accuracy of every selector on every model and task. Scores in percent at heavy sparsity. Budget is the mean reads for selectors without calibration data and the calibrated budget for the others. GPQA-Diamond (GPQA-D) averages four samples per question.

Budget	Method	Reads (%)	KL ( $\downarrow$ )	Top-1 ( $\uparrow$ )
${\sim}13\%$	Our reader	12.8	0.334	0.751
	Prox, calibrated, division tuned on development data	13.2	0.488	0.709
	Prox as published, calibrated	13.7	0.462	0.723
	HiRE-style gate screen	12.8	1.709	0.469
${\sim}18\%$	Our reader	17.8	0.199	0.808
	Prox, calibrated, division tuned on development data	18.3	0.298	0.774

Appendix

Table 19: On OLMoE, our reader is closer to the full model than every baseline evaluated at each budget. Next-token KL divergence from the full model (lower is better) and top-1 agreement on 192 held-out Pile documents, at the reads each method realizes. HiRE-style screens channels with an INT4 copy of the gate projection under our top-

k

. R-Sparse adds a low-rank term for skipped inputs. CHESS, CATS, M-COUNTDOWN (the up-projection variant of COUNTDOWN ( Cheon & Kang, 2025 ) ), and Park et al. (2026) compute one projection in full before thresholding, so they read at least a third of each routed expert.

Batch size		1	2	4	8	16	32	64	128
Qwen3-Thinking
MoE block (ms)	Full model	3.084	5.525	8.517	11.969	18.370
	Our reader	1.619	2.604	3.562	5.131	9.260	not measured
	Speedup ( $\times$ )	1.90	2.12	2.39	2.33	1.98
Per output token (ms)	Full model	6.253	9.454	12.450	15.666	22.044	27.366	34.107	38.196
	Our reader	4.605	6.609	7.453	8.957	12.850	19.831	28.925	38.363

Appendix

Table 20: Speedups peak at batch sizes four to sixteen, and the whole-model gain fades by batch 128. Decode times at 17.8% of routed-expert reads, in milliseconds per decoding step, median of three processes. The MoE block covers all MoE layers with the router, Gemma 4’s dense MLP, and Qwen3.5’s shared expert. Whole-model times use 128-token prompts.

Figure 12: Our reader raises per-GPU throughput at the same per-user speed. Whole-model decoding of three post-trained models at 17.8% of routed-expert reads, against optimized vLLM, from Table 20 (128-token prompts, one RTX PRO 6000 Blackwell GPU). Each point is one batch size, labeled for our reader. Per-user speed is the inverse of the time per output token, and per-GPU throughput is the batch’s total output rate.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding
Sep 28, 2026Ali Abbasi, Justin Shi, Soheil KolouriMixture-of-Experts Inference LLM Inference Acceleration
College of Connected Computing, Vanderbilt University
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
LinkedIn · MIT · Google

Mar 6, 2026J/K move · Enter open · S save

MoE Lens -- An Expert Is All You Need

Marmik Chaudhari, Idhant Gulati, Nishkal Hundia, Pranav Karra+1

The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA

Mixture-of-Experts Expert Specialization Mixture-of-Experts Models Mixture-of-Experts Language Models Mixture of Experts Mixture-of-Experts Interpretability LLM Routing

Open paper AskPDF arXiv

Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.

Abstract

Mixture-of-Experts Inference

Explore similar work

MoE Lens -- An Expert Is All You Need

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models