In Mixture-of-Experts (MoE) models, the nature of expert specialization remains controversial. Reports of domain-associated expert preferences contrast with analyses attributing routing primarily to token identity or syntactic patterns. To clarify these differing accounts, we look beyond expert-level routing to investigate the neuron-level organization of domain-specialized computation using Routing-Aware Magnitude Profiling (RAMP), a measure developed for this analysis. Using mixed-domain calibration data, we identify domain-specialized neuron sets with 58-74% overlap across multi-source samples of the same domain, compared with 15-24% across domains. To understand how this organization evolves during training, we trace neuron importance throughout pre-training and post-training across three fully open MoE families. Neuron importance continues to reorganize between pre-training checkpoints despite comparatively stable expert-level importance rankings, while post-training largely preserves neuron rankings. Building on these neuron-level perspectives, we further introduce RAMP-based Pruning (RAMPr) to select neurons across experts for balanced multi-domain capability preservation. At 75% neuron retention, RAMPr preserves average 91-98% of original full performance across six models. At 50% retention, our pruning method can reduce resident GPU memory by 36-43% and improves batched prefill throughput by 17-39% effectively. Our code is available at https://github.com/ydyjya/Neuron-Level-Specialization-in-MoE
Figures & tables
Figure 1: Routing-weighted activation magnitudes and output projection norms quantify domain-wise neuron importance, revealing domain-specialized sets that span mixed experts. Their enrichment correlates with domain routing preferences. Aggregating these important scores guides multi-domain neuron pruning across experts, retaining different numbers of neurons in each expert.
Figure 2: Neuron-set overlap at four selection rates across six models. Blue and orange bars show same-domain and cross-domain overlap. Dashed lines and connected center points denote six-model means, while gray dotted lines mark random-set expectations. Dark ticks show each model’s overlap at 1% selection, and upper strips give the change in the same–cross gap relative to that setting, in percentage points.
Figure 3: (a) Share of each domain’s selected neurons held by the top ⌈0.1E⌉ experts, with the gray line marking uniform allocation and the arrow giving the gap in percentage points. (b) Observed overlap against the count-preserving expectation in Equation 5 , with one marker per model, domain and selection rate, and the dashed line marking parity. (c) Validation dispatch share of each domain’s enriched experts, from the same domain in color and from other domains in gray, with black ticks marking their share of all experts.
Model
∣Hd∣/E
M
ρ
ΔH
Qwen3
16.18
83.11
0.6668
26.01
Qwen3.5
13.61
83.02
0.5671
19.19
Qwen3.6
13.39
83.40
0.5677
20.71
GLM-4.7
18.12
83.29
0.6498
19.61
Qwen1.5
23.26
66.40
0.3682
3.65
OLMoE
17.47
72.43
0.6357
13.51
Table 1: Enrichment and validation routing at 6.25% selection. ∣Hd∣/E is the share of experts enriched for a domain, M is their share of the domain’s exclusive neurons, ρ is the median Spearman correlation between F and Δr , and ΔH is the target-minus-other-domain dispatch-share difference, with all but ρ in percent.
Figure 4: Importance-rank correlations between adjacent checkpoints within each domain–layer–pair cell on validation text. (a) Neuron correlations ρN and (b) expert-aggregate correlations ρE for pre- and post-training. (c) Paired differences Δρ=ρE−ρN , with red dots where neurons are more stable. (d) Neuron and expert medians for each post-training stage pair on the scale of (a) and (b).
RAMPr-L
RAMPr-D
Model
50
75
90
50
75
90
Qwen3
61.04
92.64
97.44
44.47
93.99
98.74
Qwen3.5
83.05
96.82
98.52
77.53
98.20
99.54
Qwen3.6
83.83
97.63
98.66
80.01
97.63
99.15
GLM-4.7
66.16
91.12
100.77
67.33
95.61
100.67
Qwen1.5
88.47
93.93
95.26
85.73
92.95
103.60
Table 2: Mean performance retention (%) of RAMPr relative to the original model at 50%, 75% and 90% neuron retention.
Model
Base
F-G
F-M
REAP
L
D
Qwen3
78.45
29.25
29.37
60.77
52.66
40.18
Qwen3.5
85.20
49.75
56.34
–
76.12
74.21
Qwen3.6
84.10
51.65
58.26
–
75.23
75.71
GLM-4.7
74.70
29.87
31.01
–
49.05
52.11
Qwen1.5
45.92
32.59
33.76
–
39.60
39.15
OLMoE
46.37
26.89
30.60
–
38.29
38.05
Table 3: Raw-score 11-task macro-average against Fisher and REAP at 50% neuron retention, with GLM-4.7 also compared at 75%. F-G and F-M denote Fisher calibrated on GSM8K and MBPP, and REAP retains the same fraction of experts.
Figure 5: RAMPr-L with fixed128 layout at batch 32 and 2048 input tokens, relative to the original model on the same GPU. The Mean row averages six models arithmetically in (a) and (b) and geometrically in (c) and (d). (a) Active parameters and (b) resident memory, with colored bars for 50% retention, gray bands for 75% and 90%, ticks at the original and diamonds for REAP. (c) Prefill and (d) decode speedups on log axes, with points from faint to solid for 90%, 75% and 50% retention and the original model at 1× .
Appendix figures & tables53 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9Table 10
Model
MCQ
Gen
Cross
Qwen3
66.71
60.60
34.74
Qwen3.5
66.81
58.99
33.65
Qwen3.6
67.23
59.56
33.78
GLM-4.7
70.80
63.99
37.56
Qwen1.5
58.51
52.09
26.92
OLMoE
73.81
70.36
43.83
Appendix
Table 7: Task-format overlap (%) averaged over four selection fractions. MCQ and Gen compare discovery and validation sets within a bucket; Cross compares the two formats within a category.
Figure 7: Within-format overlap and dominant-domain identity under routing-weighted RAMP. (a, b) Same-domain replication and cross-domain overlap within generation and multiple-choice inputs at 6.25% selection, with the dashed line marking the uniform-set reference; Table 8 reports all four selection fractions. (c) Size of the strong-margin group relative to all routed neurons. (d) Validation dominant-label agreement within this group, which is selected on discovery text.
1%
3.125%
6.25%
12.5%
Model
Same
Cross
Same
Cross
Same
Cross
Same
Cross
Generation
Qwen3
49.72
7.18
58.40
9.97
64.02
13.75
70.69
21.16
Qwen3.5
48.39
6.31
58.24
10.62
64.34
16.02
70.49
25.69
Qwen3.6
48.58
5.98
58.41
10.19
64.42
15.62
70.52
25.05
GLM-4.7
55.93
8.81
63.32
10.88
67.79
14.77
72.41
21.94
Appendix
Table 8: Routing-weighted fixed-format overlap (%) at four selection fractions: six models, four budgets and two formats. Same compares discovery and validation sets within a domain; Cross compares discovery sets across domains within the same format. Generation averages five domains or ten unordered pairs per layer, and multiple-choice averages four domains or six unordered pairs, followed by equal layer averaging.
Figure 8: Routing-weighted task-set overlap at 6.25% selection across all six models. Orange outlines identify diagonal entries, which measure same-domain discovery–validation replication. Off-diagonal entries compare discovery sets across domains. Cell annotations are percentages, rounded only for display. All panels use a 0–100% color scale.
Figure 9: Routing-weighted task-set overlap at 12.5% selection. The uniform-set reference is 12.5%.
Model
h/E
1%
3.125%
6.25%
12.5%
Qwen3
13/128
76.53
72.32
67.22
57.36
Qwen3.5
26/256
82.72
78.68
73.24
61.60
Qwen3.6
26/256
83.70
79.69
74.16
62.27
GLM-4.7
7/64
75.73
73.59
68.73
58.74
Qwen1.5
6/60
53.27
47.53
42.47
35.98
OLMoE
7/64
74.58
70.71
64.29
53.79
Appendix
Table 9: Routing-weighted selected-neuron coverage (%) of the top h=⌈0.1E⌉ experts. h/E is the actual expert fraction and also the coverage under uniform allocation.
Figure 10: Ranked selected-neuron coverage curves for the routing-weighted discovery sets. The horizontal axis is the fraction of experts included after sorting by selected-neuron count, and the vertical axis is the covered fraction G of the domain’s selected set. Darker blue indicates a smaller selection rate. Dots mark the top-10% point at the actual fraction h/E , indicated by the dotted vertical line, and the dashed diagonal is the uniform-allocation reference. Curves are computed within each domain–layer unit and averaged equally over layers and buckets.
Figure 11: Routing-weighted spatial statistics for each model and bucket at 6.25% and 12.5% selection, as layer means. (a, b) Normalized effective expert count, the inverse concentration of selected-neuron shares across experts. (c, d) Observed overlap above the count-preserving expectation, which is positive in every reported layer–bucket cell at both budgets.
1%
3.125%
6.25%
12.5%
Model
Obs.
Bound
Obs.
Bound
Obs.
Bound
Obs.
Bound
Qwen3
9.02
22.86
12.39
25.63
16.33
28.92
23.81
34.97
Qwen3.5
9.28
22.51
14.46
26.92
19.52
31.20
28.42
38.17
Qwen3.6
9.21
21.68
14.21
26.03
19.25
30.39
27.98
37.33
GLM-4.7
12.23
27.75
14.93
28.56
18.74
31.78
25.51
37.37
Qwen1.5
8.47
41.52
11.28
44.92
15.23
48.51
22.47
53.59
Appendix
Table 10: Observed cross-domain overlap (Obs.) and the routing-only bound (Bound) in Equation 17 , in percent. Both quantities use routing-weighted discovery top sets and are averaged over the 36 domain pairs within each native layer and then over layers. The bound is the overlap expected if neurons within each expert kept the same order for every domain.
All tokens
Shared tokens
Model
Same
Cross
Same
Cross
Gap retained
Share
Types
Qwen3
66.13
16.33
71.67
28.82
86.0
12.13
10
Qwen3.5
66.18
19.52
72.67
30.48
90.4
16.27
11
Qwen3.6
66.40
19.25
72.76
30.05
90.6
16.27
11
GLM-4.7
69.72
18.74
75.03
32.97
82.5
8.70
6
Qwen1.5
57.77
15.23
73.62
36.27
87.8
12.13
10
Appendix
Table 11: Same-domain and cross-domain overlap at 6.25% selection under all tokens and under token types shared by all 18 text groups, in percent. Gap retained is the shared-token same–cross gap divided by the all-token gap, and the last two columns give the shared tokens’ share of token positions and their number of types.
Coverage
Same
Cross
ρ
ΔH
Model
Raw
Norm.
Raw
Norm.
Raw
Norm.
Raw
Norm.
Raw
Norm.
Qwen3
67.22
41.21
66.13
49.21
16.33
21.45
0.667
0.277
26.01
12.50
Qwen3.5
73.24
49.95
66.18
51.69
19.52
23.96
0.567
0.320
19.19
15.03
Qwen3.6
74.16
49.18
66.40
50.17
19.25
23.11
0.568
0.296
20.71
15.40
GLM-4.7
68.73
47.97
69.72
59.54
18.74
27.34
0.650
0.379
19.61
13.98
Qwen1.5
42.47
36.40
57.77
55.23
15.23
15.61
0.368
0.159
3.65
1.93
Appendix
Table 12: Statistics at 6.25% selection with RAMP (Raw) and with per-dispatch importance in Equation 18 (Norm.). Coverage is the share of each domain’s selected neurons held by the top 10% of experts, overlaps and coverage are in percent, ρ is the median correlation between F and Δr , and ΔH is in percentage points.
Figure 12: Paired Spearman gains over original top-set counts (left) and exclusive counts (right) for validation relative dispatch preference. Each panel includes all 1926 domain–layer units, using routing-weighted 6.25% discovery sets and matched finite- F expert support. ECDFs retain negative gains; no units are selected by their routing outcome.
Figure 13: Routing-weighted enrichment–routing sensitivity at four selection rates. Panels show the median ρ(F,Δr) , median within-unit advantage over top-set counts, and mean frozen-list preference difference. All six models are shown; the selection rates are discrete measured thresholds, not pruning settings.
Model
M1%
M3.125%
M6.25%
M12.5%
(∣H∣/E)6.25%
(∣U∣/k)6.25%
Qwen3
76.21
80.92
83.11
85.06
16.18
38.03
Qwen3.5
75.52
80.63
83.02
85.68
13.61
35.04
Qwen3.6
76.49
80.93
83.40
85.89
13.39
34.92
GLM-4.7
78.29
81.83
83.29
85.06
18.12
34.29
Qwen1.5
58.18
64.76
66.40
66.90
23.26
39.51
OLMoE
67.11
72.15
72.43
72.95
17.47
27.75
Appendix
Table 13: Routing-weighted enrichment at four selection rates. M is the mean exclusive-neuron share carried by composition-enriched experts. The last two columns report mean list size and exclusive fraction at 6.25% only. All entries are percentages averaged equally over domain–layer units; each rate covers all 1926 units with nonempty enriched lists.
Model
N
ρ(F,r)
ρ(F,Δr)
n−r/n−Δ
ΔρF−T
ρ(F,Δrw)
Qwen3
432
0.7018 [0.625, 0.761]
0.6668 [0.586, 0.721]
0/0
0.0538
0.6718
Qwen3.5
360
0.5562 [0.461, 0.627]
0.5671 [0.495, 0.628]
0/0
0.0874
0.5715
Qwen3.6
360
0.5808 [0.498, 0.647]
0.5677 [0.509, 0.630]
0/0
0.0793
0.5712
GLM-4.7
414
0.6419 [0.566, 0.703]
0.6498 [0.568, 0.713]
0/0
0.0719
0.6714
Qwen1.5
216
0.3548 [0.193, 0.503]
0.3682 [0.228, 0.499]
13/9
0.0611
0.6582
OLMoE
144
0.5574 [0.430, 0.635]
0.6357 [0.520, 0.706]
3/0
0.0681
0.6616
Appendix
Table 14: Domain–layer enrichment–routing associations at routing-weighted 6.25% selection. r is validation dispatch share and Δr=r−b is relative domain preference. Correlations report medians and [25th, 75th percentiles] across units. N is valid coverage, equal to the total; n−r/n−Δ counts negative correlations. ΔρF−T is the median within-unit gain over original top counts on matched finite- F support. The last column uses effective routing-weight shares without changing discovery membership.
Model
Cross-domain valid/total
ρcross
n−cross
se=0
RˉH
BˉH
ΔH
n−H
Qwen3
6014/6144
0.7197
125
130
0.4012
0.1410
0.2601
0
Qwen3.5
9612/10240
0.6333
443
628
0.3084
0.1165
0.1919
0
Qwen3.6
9604/10240
0.6390
420
636
0.3214
0.1143
0.2071
0
GLM-4.7
2893/2944
0.7030
93
51
0.3582
0.1621
0.1961
1
Qwen1.5
1439/1440
0.4000
237
1
0.2653
0.2289
0.0365
8
OLMoE
1015/1024
0.6946
38
9
0.3007
0.1655
0.1351
0
Appendix
Table 15: Fixed-expert and frozen-list validation checks at routing-weighted 6.25% selection. ρcross correlates F with dispatch share across nine domains within a native layer–expert. se=0 counts empty exclusive pools. RˉH , BˉH and ΔH are domain–layer means for the discovery-frozen list H={e:F>1,q<0.05} . Shares are fractions. All 1926 lists are nonempty; n−H counts negative list differences.
Table 27
Task
Setting
Qwen3
Qwen3.5
Qwen3.6
GLM-4.7
Qwen1.5
OLMoE
WinoGrande
Original
70.32
75.06
74.66
73.16
65.59
67.72
RAMPr 90%
71.59
74.19
74.43
73.01
65.98
67.40
Ablate chat-gen
51.38
72.06
71.43
62.04
55.64
51.70
Ablate chat-mcq
49.17
63.93
64.96
52.64
51.78
52.80
HellaSwag
Original
77.59
82.64
83.52
80.13
78.90
80.75
RAMPr 90%
77.40
82.35
83.51
80.34
78.68
80.17
Appendix
Table 18: Task scores (%) after zeroing the top-10% neuron set Td(10%) of one domain, compared with the original model and with RAMPr at 90% retention, which removes the same number of neurons. Each block lists the tasks matched to one domain; gen and mcq denote the generation and multiple-choice buckets of that domain. Shaded rows ablate the domain matched to the task.
10% removed
25% removed
Task
Qwen3
Qwen3.5
Qwen3.6
GLM-4.7
Qwen1.5
OLMoE
Qwen3
Qwen3.5
Qwen3.6
GLM-4.7
Qwen1.5
OLMoE
WinoGrande
− 2.37
− 1.72
+0.11
− 1.03
− 0.87
− 2.21
− 13.42
− 1.74
− 2.29
− 12.08
+0.47
− 1.66
HellaSwag
− 2.42
− 1.59
− 1.64
− 2.52
− 0.90
− 2.42
− 9.03
− 5.19
− 4.60
− 16.56
− 1.82
− 5.34
CEval
− 2.08
− 0.89
− 0.59
− 4.83
− 3.05
− 2.01
− 8.54
− 4.16
− 4.68
− 11.74
− 7.13
− 4.90
CMMLU
− 2.35
− 1.20
− 1.21
− 3.51
− 2.23
− 1.46
− 9.20
− 4.65
− 3.89
− 9.10
− 5.33
− 4.55
PubMedQA
− 5.80
− 5.80
− 0.20
− 1.60
− 1.20
− 0.60
− 7.20
− 0.40
− 1.40
− 13.40
+0.80
− 27.40
Appendix
Table 19: Score difference (percentage points) between random removal and RAMPr at the same budget: random 10% against RAMPr at 90% retention, and random 25% against RAMPr at 75%. Negative values mean random removal scores lower.
Stage
Model
Cells
Median ρE
Median ρN
Median Δρ
Δρ>0
Pre
OLMoE
5808
0.989789
0.939291
0.048084
5775/5808
Pre
OpenMoE
264
0.916415
0.841811
0.053208
201/264
Post
OLMoE
528
0.998077
0.996918
0.000715
365/528
Post
Instella
858
0.998947
0.997497
0.001597
737/858
Appendix
Table 20: Unfiltered validation/all cell medians and positive-gap counts. Paired-gap medians summarize within-cell differences, not differences between marginal medians.
Table 31
Figure 14: Additional multi-granularity interval changes and local replication. Panels (a), (c) and (d) compare validation rank correlations for routing counts, summed neuron importance, full expert-output energy and neuron importance, using domain–layer means for each interval. Pre-training endpoint comparisons in (a) are separate intervals. Panels (c) and (d) show within-family stage comparisons. Panel (b) reports the original candidate counts retained by the full validation set and both halves. Connecting lines follow categorical comparisons and do not encode interval duration. Each panel labels its own vertical scale.
Interval
C
S
E
OLMoE 20B–5033B
.543
.433
.458
OpenMoE 200B–1.1T
.190
.526
.485
OLMoE stage2 21B–101B
.991
.996
.996
Appendix
Table 23: Validation expert-rank correlations over long pre-training intervals and OLMoE stage2. Columns correspond to dispatch counts, summed neuron importance and complete-output energy. Each entry is a domain–layer mean.
Figure 15: Stage comparisons for dispatch counts, summed neuron importance and complete-output energy. Values are domain–layer mean rank correlations for each interval, computed separately on fixed discovery and validation text. Stage-pair positions are categorical and do not encode interval length. OLMoE’s P and S2 labels denote its separate pre-training and stage2 step counters. The three statistics summarize expert-level rank stability.
Figure 16: Endpoint and released-stage neuron comparisons, including Base-to-final comparisons and Instella’s earlier stages. Points are domain–layer mean Spearman correlations. Lines join categorical comparisons only. Discovery and validation markers can overlap when their values are close.
Figure 17: Validation outcomes for all 267 frozen domain–layer transitions from OLMoE and OpenMoE pre-training. (a, b) Validation routing-load shift against validation neuron-rank correlation; dashed lines are the selection thresholds, and orange points fail at least one criterion and remain in the denominator. (c, d) Discovery versus validation neuron-rank correlation, marking outcomes on the full validation set and on both halves of 32 documents per bucket; the two tests are not nested and the panels use different axis ranges.
Figure 18: Cross-text consistency and within-expert structure of neuron changes. (a) Early- and late-checkpoint cross-text rank correlations and the correlation of importance-share differences. (b) Median within-expert L1 share ηc on jointly observed experts; pre-training groups are frozen candidates, while Instella groups cover all 286 domain–layer cells per stage pair. (c) Instella’s mean cross-checkpoint rank correlation contrasted with the median cross-text correlation of Δq . (d) ηc alongside pooled direction agreement of the top-1% Δq changes, retaining unavailable validation coordinates in the denominator.
Figure 19: Exploratory replication of discovery-selected changes. All five selection fractions are shown for layer-share and within-expert-share differences. Direction agreement is the pooled same-direction count divided by the pooled selected count, retaining unavailable validation coordinates in the denominator. Signed-change and absolute-magnitude Spearman correlations are computed separately on available selected coordinates and summarized by valid-cell medians.
Figure 20: Domain neuron-set statistics of Section 3 computed at every checkpoint, at 6.25% selection under routing-weighted RAMP. Same- and cross-domain overlaps follow Equation 4 , the count-preserving reference follows Equation 5 , and M is the share of each domain’s exclusive neurons held by its enriched experts, as in Table 1 . Values are averaged over domains and layers. (a) OLMoE pre-training labelled by tokens seen, followed by the stage-2 and post-training checkpoints, ending at the OLMoE-0125-Instruct model of Section 3 . (b) OpenMoE pre-training. (c) Instella’s released stages.
Method
Unit
Operation
Ranking signal
Calibration
NAEE
Expert
Remove
Reconstruction of layer output
General text
REAP
Expert
Remove
Router-weighted output norm
General and code
RAZOR
Expert
Remove
Output change after deletion
Seven-domain pool
EASY-EP
Expert
Remove
Router-weighted output norm, token-weighted
Target domains
HC-SMoE
Expert
Merge
Clusters of expert outputs
General text
DERN
Expert, neuron
Remove, merge
Routing statistics, segment similarity
General text
Appendix
Table 24: MoE compression methods compared by the unit they act on, the operation, the signal that ranks units and the calibration text.
Figure 21: n128 mean performance retention for both aggregation rules at all three neuron-retention rates. Ratios are averaged over tasks with nonzero Base scores. Qwen1.5 and OLMoE use eleven tasks, and the remaining complete rows use thirteen. Values exceeding 100% are retained. Lines connect only the evaluated budgets.
Figure 22: All 264 task-score differences (six models, eleven non-AIME tasks and four RAMPr–Fisher comparisons) at 50% routed-neuron retention, in percentage points. RAMPr-L/D use 11×128 calibration texts and layer-wise allocation, while each Fisher baseline uses 128 GSM8K or MBPP texts and expert-local allocation. Macro11 averages raw task scores rather than Base-relative retention. All 24 macro differences are positive, and 246 of 264 task differences are non-negative, with the remaining negative differences retained.
Figure 23: GLM-4.7-Flash task scores and RAMPr-D-minus-REAP differences for all thirteen tasks. RAMPr-D retains 75% of routed neurons, whereas REAP-25 retains 75% of experts (48/64). RAMPr-D has the higher eleven-task raw macro-average (71.90% versus 68.99%, a difference of +2.91 percentage points). Its thirteen-task mean Base-relative retention including AIME is nearly identical (95.61% versus 95.71%).
Figure 24: Complete empirical distributions of normalized retained width for all 36 n64 masks. Rows show RAMPr-L/D and columns show neuron-retention budgets; the six-model legend is shared across panels. Every native integer width is represented without smoothing, including zero-width experts. Fixed expert counts within each model make pooled and layer-equally weighted ECDFs identical. Shared experts, GLM’s initial dense layer, and layout padding are excluded.
Figure 25: Layer-equally weighted retained-coordinate coverage, 100G(k) , at every integer expert count k , with the same panel and model ordering as Figure 24 . The diagonal gives uniform-subset expected percentage coverage 100k/E ; markers identify k=⌊rE+1/2⌋ for neuron-retention fraction r . Expert budgets are exactly matched at 50% and 75%; at nominal 90%, actual fractions are 89.84375% for Qwen3/3.5/3.6, 90% for Qwen1.5, and 90.625% for GLM-4.7/OLMoE.
Figure 26: Routed-FFN active retention at 50% neuron retention: (a) logical and (b) fixed128 execution-dimension estimates for RAMPr-L/D with n64 calibration. Both panels reuse original-model dispatch on the nine-domain validation probe; neither reports post-pruning active measurements. Fisher’s 50% reference is shown only for the four models covered by the available task-conditioned n128 masks, not for Qwen1.5 or OLMoE; its calibration differs from RAMPr’s. Axes show retained active percentages relative to each original model (100%), with a restricted display range. REAP is not plotted across six models; its two inspected configurations are reported in Table 28 .
Model
Rule
Keep (%)
Pstored (B)
Alog (%)
A128 (%)
Pˉlog (B)
Fˉlog (G)
Fˉ128 (G)
Qwen3
L
50
14.4955
66.39
72.09
1.2030
2.4060
2.6123
Qwen3
L
75
21.7433
88.02
90.42
1.5949
3.1898
3.2768
Qwen3
L
90
26.0920
96.55
97.38
1.7495
3.4989
3.5290
Qwen3
D
50
14.4955
62.84
69.57
1.1386
2.2771
2.5212
Qwen3
D
75
21.7433
85.16
89.08
1.5431
3.0861
3.2282
Qwen3
D
90
26.0920
95.46
96.79
1.7297
3.4595
3.5077
Appendix
Table 25: All 36 RAMPr n64 configurations. All active values use original-model dispatch on the nine-domain validation probe. Stored counts concern logically sliced routed weights only; A is retained active relative to the original model, Pˉ is weights invoked per token, and Fˉ is matrix-multiply FLOPs/token. Subscripts log and 128 distinguish logical and fixed128 execution-dimension estimates. B and G denote 109 . Keep is the nominal routed-neuron percentage; native integer rounding changes the 90% routed-storage fraction by at most 0.0007 percentage points.
Model
Rule
Keep (%)
Pstored (B)
Alog (%)
A128 (%)
Pˉlog (B)
Fˉlog (G)
Fˉ128 (G)
Qwen3
L
50
14.4955
66.16
71.94
1.1988
2.3976
2.6072
Qwen3
D
50
14.4955
62.77
69.47
1.1374
2.2747
2.5176
GLM-4.7
L
75
20.8373
83.28
85.02
1.4460
2.8921
2.9528
GLM-4.7
D
75
20.8373
81.40
83.87
1.4134
2.8269
2.9128
Appendix
Table 26: Four supplementary RAMPr n128 configurations, separate from the n64 grid. All active values use original-model dispatch on the nine-domain validation probe. Stored counts concern logically sliced routed weights only; A is retained active relative to the original model, Pˉ is weights invoked per token, and Fˉ is matrix-multiply FLOPs/token. Subscripts log and 128 distinguish logical and fixed128 execution-dimension estimates. B and G denote 109 .
Model
Calibration task
w
Pstored (B)
Alog (%)
A128 (%)
Pˉlog (B)
Fˉlog=Fˉ128 (G)
Qwen3
C-Eval (valid)
384
14.4955
50.00
50.00
0.9060
1.8119
Qwen3
CMMLU
384
14.4955
50.00
50.00
0.9060
1.8119
Qwen3
GSM8K
384
14.4955
50.00
50.00
0.9060
1.8119
Qwen3
HellaSwag
384
14.4955
50.00
50.00
0.9060
1.8119
Qwen3
MBPP
384
14.4955
50.00
50.00
0.9060
1.8119
Qwen3
minerva_math
384
14.4955
50.00
50.00
0.9060
1.8119
Appendix
Table 27: All 24 available Fisher rule-E masks. Each uses 128 task-conditioned calibration texts; retained width w is exactly half of each expert’s original width, unlike RAMPr’s cross-expert allocation. Both logical and fixed128 routed-active retention are 50%; retained widths are multiples of 128, so the two matrix-multiply estimates coincide. Columns use the units and original-routing probe of Table 25 .
Model
n
Rule
PREAP
Plog
ΔP (%)
P128
Alog (%)
A128 (%)
Qwen3
64
L
16.030317
16.036608
+0.0392
18.176489
66.39
72.09
Qwen3
64
D
16.030317
16.036608
+0.0392
18.275580
62.84
69.57
Qwen3
128
L
16.030317
16.036608
+0.0392
18.206374
66.16
71.94
Qwen3
128
D
16.030317
16.036608
+0.0392
18.276366
62.77
69.47
GLM-4.7
64
L
22.996116
22.997624
+0.0066
23.642498
83.39
85.17
GLM-4.7
64
D
22.996116
22.997624
+0.0066
23.815513
81.35
83.87
Appendix
Table 28: Backbone-parameter reconciliation with the two inspected REAP checkpoints. Qwen3 uses 50% neuron/expert retention and GLM-4.7 uses 75%; RAMPr rules L/D are separate. PREAP , Plog and P128 are backbone parameter counts in billions. ΔP is the logical RAMPr budget difference relative to REAP. Alog and A128 are original-routing RAMPr active percentages relative to REAP’s fixed- K , full-width routed structure (100% of its original model). Auxiliary MTP tensors and non-parameter buffers are excluded.
Figure 27: Real-minus-zero task-score differences for all 180 n64 comparisons. Each model has six method–retention settings per task. HumanEval uses source-matched v3 scores, and the other tasks retain their recorded metrics. Horizontal offsets separate configurations without implying another quantitative axis.
Figure 28: Full fixed128 efficiency grid for RAMPr-L with n64 calibration. Every panel reports all six models at 50%, 75% and 90% neuron retention under four batch–input settings, giving 216 displayed values. Speedups compare the pruned model with its same-GPU Full baseline and share a color scale centered at one. Memory reductions use a separate 0–50% scale. Numbers are rounded only for display. Loading and layout conversion are outside inference timing.
Figure 29: Eleven-task raw macro scores over calibration-set size. Blue denotes RAMPr-L and orange RAMPr-D. Solid, dashed and dotted lines show 50%, 75% and 90% neuron retention. The n64 HumanEval scores use the v3 protocol. Curves show raw task-score averages.
Figure 30: All thirteen task scores in the Qwen3 calibration-composition comparison. Mixed and generation-only calibration contain different total numbers of texts. Both RAMPr variants and the available full-model and REAP results are shown. All task scores correspond to the same calibration-composition experiment.
Method
Wino
Hella
CEval
cmmlu
mmlu
PubMed
mbpp
base
70.32
77.59
83.43
83.77
79.53
69.20
88.00
fisher-gsm8k
60.62
47.39
45.02
43.39
49.71
20.00
0.00
fisher-mbpp
61.56
49.29
49.03
48.20
52.49
57.60
1.40
fisher-winogrande
64.01
NA
NA
NA
NA
NA
NA
fisher-hellaswag
NA
60.85
NA
NA
NA
NA
NA
fisher-ceval-valid
NA
NA
65.82
NA
NA
NA
NA
Appendix
Table 29: Qwen3 main-table scores. Missing entries remain NA. Scores are percentages under the recorded task protocols.
Method
Wino
Hella
CEval
cmmlu
mmlu
PubMed
mbpp
base
75.06
82.64
84.40
85.21
84.96
65.40
86.60
fisher-gsm8k
67.32
61.67
61.96
62.23
64.11
70.80
11.20
fisher-mbpp
67.48
59.65
61.96
64.15
66.30
70.80
52.40
fisher-winogrande
71.19
NA
NA
NA
NA
NA
NA
fisher-hellaswag
NA
70.03
NA
NA
NA
NA
NA
fisher-ceval-valid
NA
NA
72.59
NA
NA
NA
NA
Appendix
Table 30: Qwen3.5 main-table scores. Missing entries remain NA. Scores are percentages under the recorded task protocols.
Method
Wino
Hella
CEval
cmmlu
mmlu
PubMed
mbpp
base
74.66
83.52
83.58
85.19
82.45
76.40
91.80
fisher-gsm8k
69.38
63.93
64.49
64.38
65.87
73.20
21.20
fisher-mbpp
66.46
60.50
65.16
65.77
66.73
74.80
61.00
fisher-winogrande
71.59
NA
NA
NA
NA
NA
NA
fisher-hellaswag
NA
71.69
NA
NA
NA
NA
NA
fisher-ceval-valid
NA
NA
72.88
NA
NA
NA
NA
Appendix
Table 31: Qwen3.6 main-table scores. Missing entries remain NA. Scores are percentages under the recorded task protocols.
Method
Wino
Hella
CEval
cmmlu
mmlu
PubMed
mbpp
base
73.16
80.13
69.24
72.24
72.43
53.40
80.60
fisher-gsm8k
58.64
51.15
41.38
38.66
41.06
42.60
1.20
fisher-mbpp
56.20
50.05
42.72
41.13
39.10
53.40
10.80
fisher-winogrande
60.38
NA
NA
NA
NA
NA
NA
fisher-hellaswag
NA
57.88
NA
NA
NA
NA
NA
fisher-ceval-valid
NA
NA
51.49
NA
NA
NA
NA
Appendix
Table 32: GLM-4.7 main-table scores. Missing entries remain NA. Scores are percentages under the recorded task protocols.
Method
Wino
Hella
CEval
cmmlu
mmlu
PubMed
mbpp
base
65.59
78.90
76.30
77.47
59.47
14.60
33.20
fisher-gsm8k
60.69
66.35
52.30
51.36
49.30
16.00
11.00
fisher-mbpp
61.96
64.81
51.71
52.58
48.59
13.40
21.00
fisher-winogrande
61.96
NA
NA
NA
NA
NA
NA
fisher-hellaswag
NA
72.75
NA
NA
NA
NA
NA
fisher-ceval-valid
NA
NA
57.65
NA
NA
NA
NA
Appendix
Table 33: Qwen1.5 main-table scores. Missing entries remain NA. Scores are percentages under the recorded task protocols.
Method
Wino
Hella
CEval
cmmlu
mmlu
PubMed
mbpp
base
67.72
80.75
37.44
36.40
53.77
60.60
28.80
fisher-gsm8k
56.51
53.89
29.35
28.01
36.16
32.40
0.80
fisher-mbpp
56.43
52.92
29.57
28.67
39.66
55.20
14.00
fisher-winogrande
60.77
NA
NA
NA
NA
NA
NA
fisher-hellaswag
NA
63.10
NA
NA
NA
NA
NA
fisher-ceval-valid
NA
NA
32.54
NA
NA
NA
NA
Appendix
Table 34: OLMoE main-table scores. Missing entries remain NA. Scores are percentages under the recorded task protocols.
Model
Thinking
Sampling
AIME
MATH
GPQA
GSM8K
HumanEval
MBPP
Qwen3
On
T=0.6 , top- p 0.95, top- k 20
38,912
16,384
16,384
8,192
8,192
8,192
Qwen3.5
On
T=1.0 , top- p 0.95, top- k 20, presence 1.5
81,920
16,384
16,384
8,192
8,192
8,192
Qwen3.6
On
T=1.0 , top- p 0.95, top- k 20, presence 1.5
81,920
16,384
16,384
8,192
8,192
8,192
GLM-4.7
On
T=1.0 , top- p 0.95
32,768
16,384
16,384
8,192
8,192
8,192
Qwen1.5
Off
T=0.7 , top- p 0.8, top- k 20, repetition 1.05
8,192
16,384
8,192
8,192
8,192
8,192
OLMoE
Off
Greedy
2,048
2,048
1,024
2,048
2,048
1,024
Appendix
Table 35: Generation settings for the six models. Budgets are maximum output tokens. For Qwen3.5 and Qwen3.6, HumanEval and MBPP use temperature 0.6, top- p 0.95 and top- k 20 without a presence penalty. Qwen1.5 additionally uses the system prompt “You are a helpful assistant.”
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
Micro-expert pruning reduces the capacity of mixture-of-experts (MoE) models by removing coupled channels within routed experts. However, finer pruning granularity alone does not guarantee the preservation of downstream capabilities, as similar language-modeling perplexity can accompany substantially different task performance. We revisit micro-expert mask construction as two distinct decisions: allocating retained capacity across experts and selecting channels within that capacity. We propose UC-RAMP, a two-stage post-training pruning framework that explicitly separates these decisions. CAPE uses cross-domain calibration evidence to redistribute expert widths under an exact layer-wise budget, while RAMP refines channel composition at fixed widths through one-out, one-in exchanges guided by a channel-interaction surrogate. Across three MoE models at 25% and 50% routed-expert channel pruning, UC-RAMP achieves the highest ten-task mean accuracy among evaluated pruning methods in five of six settings. At 50% pruning, it outperforms the strongest evaluated baseline by 8.19, 3.96, and 6.14 percentage points across the three models, respectively. In a single-A100 deployment at 25% pruning, UC-RAMP reduces measured time to first token by 19.7--23.9% and time per output token by 20.2--22.3% relative to the unpruned model. The code is available at https://github.com/xzktx003/UC_RAMP.
Zukang Xu, Xinpei Gao, Xing Hu +2
Houmo AI · University of Science and Technology of China · Southeast University
Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions (∼60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.