FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach

Ning Liao; Xiaoxing Wang; Xiaohan Qin; Junchi Yan

cs.CVMar 9, 2026

FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach

Authors: Ning Liao, Xiaoxing Wang, Xiaohan Qin, Junchi Yan

Organizations: Shanghai Jiao Tong University

Abstract

As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.

Figures & tables

Figure 1: The proposed FineRMoE (FineR-grained MoE) architecture. The MoE layer is composed of the shared expert and multiple sparse experts, in which fine-grained design is applied to both the intermediate and output dimensions. The forward process of the sparse experts consists of a sparse sum layer and a sparse concatenation layer. A single router with specially designed routing mechanism simultaneously steers the activation in the two sparse layers.

	#P/B	#AP/B	MMLU	BBH	Hell.	Wino.	ARCC.	ARCE.	AGIE.	MBPP	GSM8K	GPQA	AVG
Base Model: Qwen2.5-0.5B
PT	0.49	0.49	47.50	23.18	51.84	56.67	34.81	68.90	28.07	29.40	41.70	29.80	41.19
CT	0.49	0.49	46.18	32.04	52.71	57.46	35.58	68.06	27.31	28.00	38.51	27.23	41.31
C32A2	10.36	0.95	44.52	31.12	51.82	55.56	34.84	67.42	28.31	33.80	37.32	28.12	41.28
S16A4	0.63	0.40	26.04	12.95	30.45	51.70	22.95	46.46	25.47	0.00	1.90	23.88	24.18
DU	2.83	0.94	24.36	6.93	28.51	50.67	21.93	40.70	25.86	0.00	2.27	25.00	22.62

Table 1: The performance comparison of the proposed FineRMoE against baselines. #P/B: Total parameter size. #AP/B: Activated parameter size. Hell.: HellaSwag. Wino.: WinoGrande. ARCC.: ARC-Challenge. ARCE.: ARC-Easy. AGIE.: AGIEval.

Settings	Pre-Trained	No FG	Inter FG	Out FG	Inter&Out FG
Intermediate FG	-	$\times$	✓	$\times$	✓
Output FG	-	$\times$	$\times$	✓	✓
$\bm{G_{I}}$ , $\bm{R_{I}}$ , $\bm{G_{O}}$ , $\bm{R_{O}}$	-	1,1,1,2	32,1,1,2	1,1,2,2	32,1,2,2
Intermediate Dim	8960	8960	280	8960	280
Output Dim	1536	1536	1536	768	768
#A-Experts / #Experts	- / -	1 / 2	1 / 64	2 / 4	2 / 128

Table 2: The effectiveness validation of the finer-grained design in the proposed FineRMoE based on Qwen2.5-1.5B by traning on 10B tokens. FG: Fine-Grained. Dim: Dimension.

Settings	MMLU	BBH	Hell.	Wino.	ARCC.	ARCE.	AGIE.	MBPP	GSM8K	GPQA	AVG
Separate Router	59.31	43.62	67.73	64.88	52.05	79.80	38.62	45.00	66.19	32.37	54.96
Single Router	59.30	45.97	67.51	65.27	53.41	80.35	41.81	50.00	66.34	32.14	56.21

Table 3: The ablation study on router design based on Qwen2.5-1.5B.

Panel of Figure 2 — Figure 2: The ablation study on the architecture of FineRMoE based on Qwen2.5-1.5B.

$\bm{G_{I}}$	$\bm{G_{O}}$	#Experts	#A-Experts	#Param/B	#A-Param/B	Inter-Dim	Out-Dim	Performance
Pre-trained		-	-	1.54	1.54	8960	1536	55.39
2	2	8	2	5.63	2.74	4480	768	53.71
4	2	16	2	5.63	2.26	2240	768	54.13
4	4	32	4	8.72	2.65	2240	384	55.33
8	2	32	2	5.63	2.02	1120	768	55.00
8	4	64	4	8.72	2.22	1120	384	56.09

Table 4: The ablation study on fine-grained configurations based on Qwen2.5-1.5B.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The forward computation process of a sequence of tokens in the sparse experts of FineRMoE. For a given input sequence, the router first calculates the set of activated experts for each token. The tokens are then permuted to allow for parallel expert computation. After processing by the experts, the outputs are unpermuted to their original token order. For each token, the outputs from its activated experts are combined via a weighted sum to form dimension-reduced components. Finally, these components are concatenated to produce the final, dimension-restored output.

Hyper-parameter	Setting
Optimizer	Adam
Adam_beta1	0.9
Adam_beta2	0.95
Clip Grad	1.0
Learning Rate	1e-5
Minimum Learning Rate	1e-7

Appendix

Table 5: The training hyper-parameters in experiments.

Models	C32A2	S16A4	DU	NVShard	FineRMoE
#Param (B)	184.42	7.62	47.54	47.55	26.65
#A-Param (B)	13.33	3.34	13.32	7.63	7.94
#Generated Tokens	107	47	32000	332	110
TTFT (ms) $\downarrow$	50245.9	78.5	84.8	137.8	178.3
Throughput (tokens/s) $\uparrow$	0.2	24.0	25.6	18.9	27.3
Generated Text

Appendix

Table 6: The comparisons of the inference efficiency between C32A2, S16A4, DU, NVShard and our FineRMoE. The models are constructed by continued training on 50B tokens on Qwen2.5-7B.

Figure 4: The average similarity among the sparse experts across all layers in the effectiveness validation of finer-grained design. The corresponding models are trained based on Qwen2.5-1.5B on 10B tokens for efficiency.

$\bm{G_{I}}$	$\bm{G_{O}}$	MMLU	BBH	HellaSwag	WinoGrande	ARC-C	ARC-E	AGIEval	MBPP	GSM8K	GPQA
2	2	58.19	42.45	66.56	64.09	51.19	78.28	40.41	46.00	63.84	26.12
4	2	58.15	43.46	66.44	63.69	51.88	80.13	41.80	44.40	63.46	27.90
4	4	58.97	44.68	67.14	65.11	51.62	80.30	41.15	47.20	66.11	31.03
8	2	58.16	45.48	66.79	64.48	50.77	79.12	39.85	46.60	65.50	33.26
8	4	59.17	47.00	67.46	64.88	53.24	80.39	42.16	50.00	66.03	30.58
16	2	59.06	45.51	67.24	64.01	52.82	79.88	41.55	48.00	66.11	30.36

Appendix

Table 7: Evaluation details across 10 benchmarks in the ablation study on fine-grained configurations based on Qwen2.5-1.5B.

Settings	Pre-Trained	Base	Ablation on $\bm{T_{I}}$		Ablation on $\bm{R_{I}}$		Ablation on $\bm{R_{O}}$
$\bm{G_{I}}$ , $\bm{R_{I}}$ , $\bm{G_{O}}$ , $\bm{R_{O}}$	-	32,1,2,2	32,1,2,2	32,1,2,2	32, 2 ,2,2	32, 4 ,2,2	32,1,2, 4	32,1,2, 8
$\bm{T_{I}}$	-	1	2	4	1	1	1	1
#Experts	-	128	128	128	256	512	256	512
#A-Experts	-	2	4	8	2	2	2	2
#Param / B	1.54	5.64	5.64	5.64	9.51	17.24	9.51	17.24
#A-Param / B	1.54	1.85	1.91	2.03	1.86	1.88	1.86	1.88

Appendix

Table 8: The ablation study on

T_{I}

R_{I}

R_{O}

based on Qwen2.5-1.5B by training on 10B tokens.

Sparse Mixture-of-Experts

Sparse mixture-of-experts models route each input to a small subset of specialized networks to increase capacity while limiting computation.

See all 121 papers

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
Oct 8, 2026Yuxuan Lou, Kai Yang, Geng Zhang +2Mixture-of-Experts Models Sparse Mixture-of-Experts
School of Computing, National University of Singapore, Singapore · Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1

Feb 5, 2026J/K move · Enter open · S save

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

Jingze Shi, Zhangyang Peng, Yizhang Zhu, Yifan Wu+2

The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence

Mixture-of-Experts Inference LLM Inference Acceleration Expert Routing Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency. However, existing MoE designs face an inherent trade-off between the granularity of expert specialization and hardware execution efficiency. We propose OmniMoE, a system-algorithm co-designed framework that pushes expert granularity to its logical extreme. OmniMoE introduces vector-level Atomic Experts, enabling scalable routing and execution within a single MoE layer, while retaining a shared dense MLP branch for general-purpose processing. Although this atomic design maximizes capacity, it poses severe challenges for routing complexity and memory access. To address these, OmniMoE adopts a system-algorithm co-design: (i) a Cartesian Product Router that decomposes the massive index space to reduce routing complexity from O(N) to O(sqrt(N)); and (ii) Expert-Centric Scheduling that inverts the execution order to turn scattered, memory-bound lookups into efficient dense matrix operations. Validated on seven benchmarks, OmniMoE (with 1.7B active parameters) achieves 50.9% zero-shot accuracy across seven benchmarks, outperforming coarse-grained (e.g., DeepSeekMoE) and fine-grained (e.g., PEER) baselines. Crucially, OmniMoE reduces inference latency from 73ms to 6.7ms (a 10.9-fold speedup) compared to PEER, demonstrating that massive-scale fine-grained MoE can be fast and accurate. Our code is open-sourced at https://github.com/flash-algo/omni-moe.

Abstract

Sparse Mixture-of-Experts

Explore similar work

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs