Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries
Organizations: Nanyang Technological University · Foundation Model Department, Tencent · National University of Singapore
Abstract
In Mixture-of-Experts (MoE) models, the nature of expert specialization remains controversial. Reports of domain-associated expert preferences contrast with analyses attributing routing primarily to token identity or syntactic patterns. To clarify these differing accounts, we look beyond expert-level routing to investigate the neuron-level organization of domain-specialized computation using Routing-Aware Magnitude Profiling (RAMP), a measure developed for this analysis. Using mixed-domain calibration data, we identify domain-specialized neuron sets with 58-74% overlap across multi-source samples of the same domain, compared with 15-24% across domains. To understand how this organization evolves during training, we trace neuron importance throughout pre-training and post-training across three fully open MoE families. Neuron importance continues to reorganize between pre-training checkpoints despite comparatively stable expert-level importance rankings, while post-training largely preserves neuron rankings. Building on these neuron-level perspectives, we further introduce RAMP-based Pruning (RAMPr) to select neurons across experts for balanced multi-domain capability preservation. At 75% neuron retention, RAMPr preserves average 91-98% of original full performance across six models. At 50% retention, our pruning method can reduce resident GPU memory by 36-43% and improves batched prefill throughput by 17-39% effectively. Our code is available at https://github.com/ydyjya/Neuron-Level-Specialization-in-MoE
Figures & tables
| Model | ||||
|---|---|---|---|---|
| Qwen3 | 16.18 | 83.11 | 0.6668 | 26.01 |
| Qwen3.5 | 13.61 | 83.02 | 0.5671 | 19.19 |
| Qwen3.6 | 13.39 | 83.40 | 0.5677 | 20.71 |
| GLM-4.7 | 18.12 | 83.29 | 0.6498 | 19.61 |
| Qwen1.5 | 23.26 | 66.40 | 0.3682 | 3.65 |
| OLMoE | 17.47 | 72.43 | 0.6357 | 13.51 |
| RAMPr-L | RAMPr-D | |||||
|---|---|---|---|---|---|---|
| Model | 50 | 75 | 90 | 50 | 75 | 90 |
| Qwen3 | 61.04 | 92.64 | 97.44 | 44.47 | 93.99 | 98.74 |
| Qwen3.5 | 83.05 | 96.82 | 98.52 | 77.53 | 98.20 | 99.54 |
| Qwen3.6 | 83.83 | 97.63 | 98.66 | 80.01 | 97.63 | 99.15 |
| GLM-4.7 | 66.16 | 91.12 | 100.77 | 67.33 | 95.61 | 100.67 |
| Qwen1.5 | 88.47 | 93.93 | 95.26 | 85.73 | 92.95 | 103.60 |
| Model | Base | F-G | F-M | REAP | L | D |
|---|---|---|---|---|---|---|
| Qwen3 | 78.45 | 29.25 | 29.37 | 60.77 | 52.66 | 40.18 |
| Qwen3.5 | 85.20 | 49.75 | 56.34 | – | 76.12 | 74.21 |
| Qwen3.6 | 84.10 | 51.65 | 58.26 | – | 75.23 | 75.71 |
| GLM-4.7 | 74.70 | 29.87 | 31.01 | – | 49.05 | 52.11 |
| Qwen1.5 | 45.92 | 32.59 | 33.76 | – | 39.60 | 39.15 |
| OLMoE | 46.37 | 26.89 | 30.60 | – | 38.29 | 38.05 |
Appendix figures & tables53 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | MCQ | Gen | Cross |
|---|---|---|---|
| Qwen3 | 66.71 | 60.60 | 34.74 |
| Qwen3.5 | 66.81 | 58.99 | 33.65 |
| Qwen3.6 | 67.23 | 59.56 | 33.78 |
| GLM-4.7 | 70.80 | 63.99 | 37.56 |
| Qwen1.5 | 58.51 | 52.09 | 26.92 |
| OLMoE | 73.81 | 70.36 | 43.83 |
| 1% | 3.125% | 6.25% | 12.5% | |||||
| Model | Same | Cross | Same | Cross | Same | Cross | Same | Cross |
| Generation | ||||||||
| Qwen3 | 49.72 | 7.18 | 58.40 | 9.97 | 64.02 | 13.75 | 70.69 | 21.16 |
| Qwen3.5 | 48.39 | 6.31 | 58.24 | 10.62 | 64.34 | 16.02 | 70.49 | 25.69 |
| Qwen3.6 | 48.58 | 5.98 | 58.41 | 10.19 | 64.42 | 15.62 | 70.52 | 25.05 |
| GLM-4.7 | 55.93 | 8.81 | 63.32 | 10.88 | 67.79 | 14.77 | 72.41 | 21.94 |
| Model | 1% | 3.125% | 6.25% | 12.5% | |
|---|---|---|---|---|---|
| Qwen3 | 13/128 | 76.53 | 72.32 | 67.22 | 57.36 |
| Qwen3.5 | 26/256 | 82.72 | 78.68 | 73.24 | 61.60 |
| Qwen3.6 | 26/256 | 83.70 | 79.69 | 74.16 | 62.27 |
| GLM-4.7 | 7/64 | 75.73 | 73.59 | 68.73 | 58.74 |
| Qwen1.5 | 6/60 | 53.27 | 47.53 | 42.47 | 35.98 |
| OLMoE | 7/64 | 74.58 | 70.71 | 64.29 | 53.79 |
| 1% | 3.125% | 6.25% | 12.5% | |||||
|---|---|---|---|---|---|---|---|---|
| Model | Obs. | Bound | Obs. | Bound | Obs. | Bound | Obs. | Bound |
| Qwen3 | 9.02 | 22.86 | 12.39 | 25.63 | 16.33 | 28.92 | 23.81 | 34.97 |
| Qwen3.5 | 9.28 | 22.51 | 14.46 | 26.92 | 19.52 | 31.20 | 28.42 | 38.17 |
| Qwen3.6 | 9.21 | 21.68 | 14.21 | 26.03 | 19.25 | 30.39 | 27.98 | 37.33 |
| GLM-4.7 | 12.23 | 27.75 | 14.93 | 28.56 | 18.74 | 31.78 | 25.51 | 37.37 |
| Qwen1.5 | 8.47 | 41.52 | 11.28 | 44.92 | 15.23 | 48.51 | 22.47 | 53.59 |
| All tokens | Shared tokens | ||||||
|---|---|---|---|---|---|---|---|
| Model | Same | Cross | Same | Cross | Gap retained | Share | Types |
| Qwen3 | 66.13 | 16.33 | 71.67 | 28.82 | 86.0 | 12.13 | 10 |
| Qwen3.5 | 66.18 | 19.52 | 72.67 | 30.48 | 90.4 | 16.27 | 11 |
| Qwen3.6 | 66.40 | 19.25 | 72.76 | 30.05 | 90.6 | 16.27 | 11 |
| GLM-4.7 | 69.72 | 18.74 | 75.03 | 32.97 | 82.5 | 8.70 | 6 |
| Qwen1.5 | 57.77 | 15.23 | 73.62 | 36.27 | 87.8 | 12.13 | 10 |
| Coverage | Same | Cross | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Raw | Norm. | Raw | Norm. | Raw | Norm. | Raw | Norm. | Raw | Norm. |
| Qwen3 | 67.22 | 41.21 | 66.13 | 49.21 | 16.33 | 21.45 | 0.667 | 0.277 | 26.01 | 12.50 |
| Qwen3.5 | 73.24 | 49.95 | 66.18 | 51.69 | 19.52 | 23.96 | 0.567 | 0.320 | 19.19 | 15.03 |
| Qwen3.6 | 74.16 | 49.18 | 66.40 | 50.17 | 19.25 | 23.11 | 0.568 | 0.296 | 20.71 | 15.40 |
| GLM-4.7 | 68.73 | 47.97 | 69.72 | 59.54 | 18.74 | 27.34 | 0.650 | 0.379 | 19.61 | 13.98 |
| Qwen1.5 | 42.47 | 36.40 | 57.77 | 55.23 | 15.23 | 15.61 | 0.368 | 0.159 | 3.65 | 1.93 |
| Model | ||||||
|---|---|---|---|---|---|---|
| Qwen3 | 76.21 | 80.92 | 83.11 | 85.06 | 16.18 | 38.03 |
| Qwen3.5 | 75.52 | 80.63 | 83.02 | 85.68 | 13.61 | 35.04 |
| Qwen3.6 | 76.49 | 80.93 | 83.40 | 85.89 | 13.39 | 34.92 |
| GLM-4.7 | 78.29 | 81.83 | 83.29 | 85.06 | 18.12 | 34.29 |
| Qwen1.5 | 58.18 | 64.76 | 66.40 | 66.90 | 23.26 | 39.51 |
| OLMoE | 67.11 | 72.15 | 72.43 | 72.95 | 17.47 | 27.75 |
| Model | ||||||
|---|---|---|---|---|---|---|
| Qwen3 | 432 | 0.7018 [0.625, 0.761] | 0.6668 [0.586, 0.721] | 0/0 | 0.0538 | 0.6718 |
| Qwen3.5 | 360 | 0.5562 [0.461, 0.627] | 0.5671 [0.495, 0.628] | 0/0 | 0.0874 | 0.5715 |
| Qwen3.6 | 360 | 0.5808 [0.498, 0.647] | 0.5677 [0.509, 0.630] | 0/0 | 0.0793 | 0.5712 |
| GLM-4.7 | 414 | 0.6419 [0.566, 0.703] | 0.6498 [0.568, 0.713] | 0/0 | 0.0719 | 0.6714 |
| Qwen1.5 | 216 | 0.3548 [0.193, 0.503] | 0.3682 [0.228, 0.499] | 13/9 | 0.0611 | 0.6582 |
| OLMoE | 144 | 0.5574 [0.430, 0.635] | 0.6357 [0.520, 0.706] | 3/0 | 0.0681 | 0.6616 |
| Model | Cross-domain valid/total | |||||||
|---|---|---|---|---|---|---|---|---|
| Qwen3 | 6014/6144 | 0.7197 | 125 | 130 | 0.4012 | 0.1410 | 0.2601 | 0 |
| Qwen3.5 | 9612/10240 | 0.6333 | 443 | 628 | 0.3084 | 0.1165 | 0.1919 | 0 |
| Qwen3.6 | 9604/10240 | 0.6390 | 420 | 636 | 0.3214 | 0.1143 | 0.2071 | 0 |
| GLM-4.7 | 2893/2944 | 0.7030 | 93 | 51 | 0.3582 | 0.1621 | 0.1961 | 1 |
| Qwen1.5 | 1439/1440 | 0.4000 | 237 | 1 | 0.2653 | 0.2289 | 0.0365 | 8 |
| OLMoE | 1015/1024 | 0.6946 | 38 | 9 | 0.3007 | 0.1655 | 0.1351 | 0 |
| Task | Setting | Qwen3 | Qwen3.5 | Qwen3.6 | GLM-4.7 | Qwen1.5 | OLMoE |
|---|---|---|---|---|---|---|---|
| WinoGrande | Original | 70.32 | 75.06 | 74.66 | 73.16 | 65.59 | 67.72 |
| RAMPr 90% | 71.59 | 74.19 | 74.43 | 73.01 | 65.98 | 67.40 | |
| Ablate chat-gen | 51.38 | 72.06 | 71.43 | 62.04 | 55.64 | 51.70 | |
| Ablate chat-mcq | 49.17 | 63.93 | 64.96 | 52.64 | 51.78 | 52.80 | |
| HellaSwag | Original | 77.59 | 82.64 | 83.52 | 80.13 | 78.90 | 80.75 |
| RAMPr 90% | 77.40 | 82.35 | 83.51 | 80.34 | 78.68 | 80.17 |
| 10% removed | 25% removed | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Task | Qwen3 | Qwen3.5 | Qwen3.6 | GLM-4.7 | Qwen1.5 | OLMoE | Qwen3 | Qwen3.5 | Qwen3.6 | GLM-4.7 | Qwen1.5 | OLMoE |
| WinoGrande | 2.37 | 1.72 | +0.11 | 1.03 | 0.87 | 2.21 | 13.42 | 1.74 | 2.29 | 12.08 | +0.47 | 1.66 |
| HellaSwag | 2.42 | 1.59 | 1.64 | 2.52 | 0.90 | 2.42 | 9.03 | 5.19 | 4.60 | 16.56 | 1.82 | 5.34 |
| CEval | 2.08 | 0.89 | 0.59 | 4.83 | 3.05 | 2.01 | 8.54 | 4.16 | 4.68 | 11.74 | 7.13 | 4.90 |
| CMMLU | 2.35 | 1.20 | 1.21 | 3.51 | 2.23 | 1.46 | 9.20 | 4.65 | 3.89 | 9.10 | 5.33 | 4.55 |
| PubMedQA | 5.80 | 5.80 | 0.20 | 1.60 | 1.20 | 0.60 | 7.20 | 0.40 | 1.40 | 13.40 | +0.80 | 27.40 |
| Stage | Model | Cells | Median | Median | Median | |
|---|---|---|---|---|---|---|
| Pre | OLMoE | 5808 | 0.989789 | 0.939291 | 0.048084 | 5775/5808 |
| Pre | OpenMoE | 264 | 0.916415 | 0.841811 | 0.053208 | 201/264 |
| Post | OLMoE | 528 | 0.998077 | 0.996918 | 0.000715 | 365/528 |
| Post | Instella | 858 | 0.998947 | 0.997497 | 0.001597 | 737/858 |
| Interval | |||
|---|---|---|---|
| OLMoE 20B–5033B | .543 | .433 | .458 |
| OpenMoE 200B–1.1T | .190 | .526 | .485 |
| OLMoE stage2 21B–101B | .991 | .996 | .996 |
| Method | Unit | Operation | Ranking signal | Calibration |
|---|---|---|---|---|
| NAEE | Expert | Remove | Reconstruction of layer output | General text |
| REAP | Expert | Remove | Router-weighted output norm | General and code |
| RAZOR | Expert | Remove | Output change after deletion | Seven-domain pool |
| EASY-EP | Expert | Remove | Router-weighted output norm, token-weighted | Target domains |
| HC-SMoE | Expert | Merge | Clusters of expert outputs | General text |
| DERN | Expert, neuron | Remove, merge | Routing statistics, segment similarity | General text |
| Model | Rule | Keep (%) | (B) | (%) | (%) | (B) | (G) | (G) |
|---|---|---|---|---|---|---|---|---|
| Qwen3 | L | 50 | 14.4955 | 66.39 | 72.09 | 1.2030 | 2.4060 | 2.6123 |
| Qwen3 | L | 75 | 21.7433 | 88.02 | 90.42 | 1.5949 | 3.1898 | 3.2768 |
| Qwen3 | L | 90 | 26.0920 | 96.55 | 97.38 | 1.7495 | 3.4989 | 3.5290 |
| Qwen3 | D | 50 | 14.4955 | 62.84 | 69.57 | 1.1386 | 2.2771 | 2.5212 |
| Qwen3 | D | 75 | 21.7433 | 85.16 | 89.08 | 1.5431 | 3.0861 | 3.2282 |
| Qwen3 | D | 90 | 26.0920 | 95.46 | 96.79 | 1.7297 | 3.4595 | 3.5077 |
| Model | Rule | Keep (%) | (B) | (%) | (%) | (B) | (G) | (G) |
|---|---|---|---|---|---|---|---|---|
| Qwen3 | L | 50 | 14.4955 | 66.16 | 71.94 | 1.1988 | 2.3976 | 2.6072 |
| Qwen3 | D | 50 | 14.4955 | 62.77 | 69.47 | 1.1374 | 2.2747 | 2.5176 |
| GLM-4.7 | L | 75 | 20.8373 | 83.28 | 85.02 | 1.4460 | 2.8921 | 2.9528 |
| GLM-4.7 | D | 75 | 20.8373 | 81.40 | 83.87 | 1.4134 | 2.8269 | 2.9128 |
| Model | Calibration task | (B) | (%) | (%) | (B) | (G) | |
|---|---|---|---|---|---|---|---|
| Qwen3 | C-Eval (valid) | 384 | 14.4955 | 50.00 | 50.00 | 0.9060 | 1.8119 |
| Qwen3 | CMMLU | 384 | 14.4955 | 50.00 | 50.00 | 0.9060 | 1.8119 |
| Qwen3 | GSM8K | 384 | 14.4955 | 50.00 | 50.00 | 0.9060 | 1.8119 |
| Qwen3 | HellaSwag | 384 | 14.4955 | 50.00 | 50.00 | 0.9060 | 1.8119 |
| Qwen3 | MBPP | 384 | 14.4955 | 50.00 | 50.00 | 0.9060 | 1.8119 |
| Qwen3 | minerva_math | 384 | 14.4955 | 50.00 | 50.00 | 0.9060 | 1.8119 |
| Model | Rule | (%) | (%) | (%) | ||||
|---|---|---|---|---|---|---|---|---|
| Qwen3 | 64 | L | 16.030317 | 16.036608 | +0.0392 | 18.176489 | 66.39 | 72.09 |
| Qwen3 | 64 | D | 16.030317 | 16.036608 | +0.0392 | 18.275580 | 62.84 | 69.57 |
| Qwen3 | 128 | L | 16.030317 | 16.036608 | +0.0392 | 18.206374 | 66.16 | 71.94 |
| Qwen3 | 128 | D | 16.030317 | 16.036608 | +0.0392 | 18.276366 | 62.77 | 69.47 |
| GLM-4.7 | 64 | L | 22.996116 | 22.997624 | +0.0066 | 23.642498 | 83.39 | 85.17 |
| GLM-4.7 | 64 | D | 22.996116 | 22.997624 | +0.0066 | 23.815513 | 81.35 | 83.87 |
| Method | Wino | Hella | CEval | cmmlu | mmlu | PubMed | mbpp |
|---|---|---|---|---|---|---|---|
| base | 70.32 | 77.59 | 83.43 | 83.77 | 79.53 | 69.20 | 88.00 |
| fisher-gsm8k | 60.62 | 47.39 | 45.02 | 43.39 | 49.71 | 20.00 | 0.00 |
| fisher-mbpp | 61.56 | 49.29 | 49.03 | 48.20 | 52.49 | 57.60 | 1.40 |
| fisher-winogrande | 64.01 | NA | NA | NA | NA | NA | NA |
| fisher-hellaswag | NA | 60.85 | NA | NA | NA | NA | NA |
| fisher-ceval-valid | NA | NA | 65.82 | NA | NA | NA | NA |
| Method | Wino | Hella | CEval | cmmlu | mmlu | PubMed | mbpp |
|---|---|---|---|---|---|---|---|
| base | 75.06 | 82.64 | 84.40 | 85.21 | 84.96 | 65.40 | 86.60 |
| fisher-gsm8k | 67.32 | 61.67 | 61.96 | 62.23 | 64.11 | 70.80 | 11.20 |
| fisher-mbpp | 67.48 | 59.65 | 61.96 | 64.15 | 66.30 | 70.80 | 52.40 |
| fisher-winogrande | 71.19 | NA | NA | NA | NA | NA | NA |
| fisher-hellaswag | NA | 70.03 | NA | NA | NA | NA | NA |
| fisher-ceval-valid | NA | NA | 72.59 | NA | NA | NA | NA |
| Method | Wino | Hella | CEval | cmmlu | mmlu | PubMed | mbpp |
|---|---|---|---|---|---|---|---|
| base | 74.66 | 83.52 | 83.58 | 85.19 | 82.45 | 76.40 | 91.80 |
| fisher-gsm8k | 69.38 | 63.93 | 64.49 | 64.38 | 65.87 | 73.20 | 21.20 |
| fisher-mbpp | 66.46 | 60.50 | 65.16 | 65.77 | 66.73 | 74.80 | 61.00 |
| fisher-winogrande | 71.59 | NA | NA | NA | NA | NA | NA |
| fisher-hellaswag | NA | 71.69 | NA | NA | NA | NA | NA |
| fisher-ceval-valid | NA | NA | 72.88 | NA | NA | NA | NA |
| Method | Wino | Hella | CEval | cmmlu | mmlu | PubMed | mbpp |
|---|---|---|---|---|---|---|---|
| base | 73.16 | 80.13 | 69.24 | 72.24 | 72.43 | 53.40 | 80.60 |
| fisher-gsm8k | 58.64 | 51.15 | 41.38 | 38.66 | 41.06 | 42.60 | 1.20 |
| fisher-mbpp | 56.20 | 50.05 | 42.72 | 41.13 | 39.10 | 53.40 | 10.80 |
| fisher-winogrande | 60.38 | NA | NA | NA | NA | NA | NA |
| fisher-hellaswag | NA | 57.88 | NA | NA | NA | NA | NA |
| fisher-ceval-valid | NA | NA | 51.49 | NA | NA | NA | NA |
| Method | Wino | Hella | CEval | cmmlu | mmlu | PubMed | mbpp |
|---|---|---|---|---|---|---|---|
| base | 65.59 | 78.90 | 76.30 | 77.47 | 59.47 | 14.60 | 33.20 |
| fisher-gsm8k | 60.69 | 66.35 | 52.30 | 51.36 | 49.30 | 16.00 | 11.00 |
| fisher-mbpp | 61.96 | 64.81 | 51.71 | 52.58 | 48.59 | 13.40 | 21.00 |
| fisher-winogrande | 61.96 | NA | NA | NA | NA | NA | NA |
| fisher-hellaswag | NA | 72.75 | NA | NA | NA | NA | NA |
| fisher-ceval-valid | NA | NA | 57.65 | NA | NA | NA | NA |
| Method | Wino | Hella | CEval | cmmlu | mmlu | PubMed | mbpp |
|---|---|---|---|---|---|---|---|
| base | 67.72 | 80.75 | 37.44 | 36.40 | 53.77 | 60.60 | 28.80 |
| fisher-gsm8k | 56.51 | 53.89 | 29.35 | 28.01 | 36.16 | 32.40 | 0.80 |
| fisher-mbpp | 56.43 | 52.92 | 29.57 | 28.67 | 39.66 | 55.20 | 14.00 |
| fisher-winogrande | 60.77 | NA | NA | NA | NA | NA | NA |
| fisher-hellaswag | NA | 63.10 | NA | NA | NA | NA | NA |
| fisher-ceval-valid | NA | NA | 32.54 | NA | NA | NA | NA |
| Model | Thinking | Sampling | AIME | MATH | GPQA | GSM8K | HumanEval | MBPP |
|---|---|---|---|---|---|---|---|---|
| Qwen3 | On | , top- 0.95, top- 20 | 38,912 | 16,384 | 16,384 | 8,192 | 8,192 | 8,192 |
| Qwen3.5 | On | , top- 0.95, top- 20, presence 1.5 | 81,920 | 16,384 | 16,384 | 8,192 | 8,192 | 8,192 |
| Qwen3.6 | On | , top- 0.95, top- 20, presence 1.5 | 81,920 | 16,384 | 16,384 | 8,192 | 8,192 | 8,192 |
| GLM-4.7 | On | , top- 0.95 | 32,768 | 16,384 | 16,384 | 8,192 | 8,192 | 8,192 |
| Qwen1.5 | Off | , top- 0.8, top- 20, repetition 1.05 | 8,192 | 16,384 | 8,192 | 8,192 | 8,192 | 8,192 |
| OLMoE | Off | Greedy | 2,048 | 2,048 | 1,024 | 2,048 | 2,048 | 1,024 |
Mixture-of-Experts Language Models
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
See all 249 papers