Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers

Dian Jiao; Jiaxin Duan; Shuai Zhao; Jiabing Leng; Yiran Zhang; Feng Huang

cs.LGAug 31, 2026

Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers

Authors: Dian Jiao, Jiaxin Duan, Shuai Zhao, Jiabing Leng, Yiran Zhang, Feng Huang

Organizations: China Electronics Cloud Technology Co., Ltd.

Abstract

Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely used Mixture-of-Experts (MoE) architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool. This Global Experts Sharing mechanism enables elastic control over token-level parameter activation and computational consumption (FLOPs). Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55% of parameters. Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget. CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.

Figures & tables

Figure 1: PPL of MoE models with random expert replacement and deliberate expert replacement based on CKA analysis.

Figure 2: Overview of Transformer architectures: MoE (upper left), dense (upper right), and our CS-MoE (lower). The dashed box indicates a virtual router.

Panel of Figure 3 — Figure 3: Comparison between CS-MoE and Dense models.

Panel of Figure 4 — Figure 4: Comparison among CS-MoE models with varying activations.

Panel of Figure 5 — Figure 5: Comparison between CS-MoE and Sparse MoE models with equal activations.

Figure 6: Efficiency analysis of CS-MoE models.

C-Eval (Multi-discipline Chinese Evaluation)
Steps	0.6B / -A0.9B		1.7B / -A4B		4B / -A8B
Steps	Dense	CS-MoE	Dense	CS-MoE	Dense	CS-MoE
2,500	28.2	31.5	30.1	31.7	29.3	29.8
7,500	35.1	39.3	36.4	40.1	35.6	41.0
12,500	37.5	41.9	38.6	42.3	40.7	43.2
AQuA (Multi-step Algebraic Reasoning)

Table 1: Downstream benchmark accuracy (%) on C-Eval and AQuA across training checkpoints.

Benchmark	8B Dense Baseline	8B-A8B CS-MoE (Ours)	Diff ( $\Delta$ )
BBH Suzgun et al. (2023)	0.2867	0.7507	+0.4640
CMMLU Li et al. (2024a)	0.7417	0.7942	+0.0525
ARC Clark et al. (2018)	0.6875	0.7300	+0.0425
GSM8K Cobbe et al. (2021)	0.9400	0.9450	+0.0050
C-Eval Huang et al. (2023)	0.6996	0.7038	+0.0042
IFEval Zhou et al. (2023)	0.4300	0.4005	-0.0295

Table 2: Downstream performance comparison at 8B scale after full-scale pre-training on 4.5 trillion tokens.

1. Independent Expert Allocation ( $N_{indep}$ )
Ablation Setting	PPL $\downarrow$	C-Eval $\uparrow$	AQuA $\uparrow$
$N_{indep}=0$ (Shared pool only)	10.32	41.8	39.5
$N_{indep}=1$ (Default)	10.08	43.2	40.5
$N_{indep}=2$	10.11	43.0	40.3
2. Router Architecture
Centrally Shared Router	10.29	41.9	39.6

Table 3: Ablation results of architectural components under matched 4B parameters and 4B activations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Substitution Type	Direction	+ $\Delta$ PPL (ESR=5%)	+ $\Delta$ PPL (ESR=20%)
Targeted (CKA)	Shallow $\to$ Deep	+1.22	+3.45
Targeted (CKA)	Deep $\to$ Shallow	+1.54	+3.98
Random Swap	Shallow $\leftrightarrow$ Deep	+4.20	+12.80
Random Control	Arbitrary Layer Swap	+4.15	+15.30

Appendix

Table 4: Validation perplexity increase (

+\Delta

PPL) under targeted CKA vs. random expert substitutions.

Model Type	Total Params	Activated Params	Layers ( $L$ )	$d_{ffn}$	$d_{exp}$	$N_{indep}$	Shared Pool ( $M$ )	Top- $K$ ∗
Dense	0.6B / 1.7B / 4B / 8B	Same	16 - 36	3,072 - 12,288	-	-	-	-
CS-MoE 0.6B	0.6B	0.6B / 0.9B / 1.7B	28	3,072	768	1	84	4 / 10 / 21
CS-MoE 1.7B	1.7B	1.7B / 4B	28	6,144	1,536	1	84	4 / 13
CS-MoE 4B	4B	4B / 8B	36	9,728	2,432	1	108	4 / 10
CS-MoE 8B	8B	4B	36	9,728	2,432	1	324	4
CS-MoE 8B	8B	8B	36	12,288	3,072	1	324	4

Appendix

Table 5: Detailed model hyperparameters. *: Top-

K

activated experts include independent experts of each layer.

Hyperparameters	Settings
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1\text{e-}8$ )
Peak Learning Rate	3e-4
Weight Decay	0.01
Batch Size	512 (effective token count $\approx$ 1M)
Warmup Steps	1,000
Total Steps	12,500 (Base) / 4.5T tokens (Scaled)

Appendix

Table 6: Unified Training Hyperparameters.

Model Scale	Dense Baseline	CS-MoE (Ours)	Reduction (%) $\downarrow$
0.6B Scale	13.62	13.48	1.03%
1.7B Scale	11.30	11.12	1.59%
4B Scale	10.45	10.08	3.54%
8B Scale	9.75	9.55	2.05%

Appendix

Table 7: Final pre-training perplexity corresponding to Figure 3 (CS-MoE vs. Dense across scales).

Model Scale	Activation Configuration	Final Perplexity (PPL)
0.6B Scale	0.6B-A0.6B	13.48
	0.6B-A0.9B	13.18
	0.6B-A1.7B	13.11
1.7B Scale	1.7B-A1.7B	11.12
1.7B Scale	1.7B-A4B	10.82
4B Scale	4B-A4B	10.08

Appendix

Table 8: Final pre-training perplexity corresponding to Figure 4 (CS-MoE with varying activation counts

K

Configuration	Standard MoE	CS-MoE (Ours)	Reduction (%) $\downarrow$
8B-A4B	9.70	9.65	0.52%
12B-A4B	9.40	9.19	2.23%

Appendix

Table 9: Final pre-training perplexity corresponding to Figure 5 (CS-MoE vs. Standard MoE under double-aligned total and active parameters).

Model	Pre-train PPL	C-Eval (%)	AQuA (%)
4B Dense Baseline	10.25 $\pm$ 0.04	40.8 $\pm$ 0.3	39.0 $\pm$ 0.5
4B CS-MoE (Ours)	10.06 $\pm$ 0.03	42.3 $\pm$ 0.2	40.5 $\pm$ 0.4

Appendix

Table 10: Mean and standard deviation over 3 independent training runs at 4B scale.

Model Configuration	Prefill (ms)	Decoding (ms/tok)
Dense 8B	12.5	14.2
Standard Sparse MoE 8B	14.1	16.5
CS-MoE 8B (Default)	13.8	15.8
CS-MoE 8B (with Cache Optimization)	13.1	14.9

Appendix

Table 11: Real-world latency profiling at 8B model scale on

8\times

NVIDIA H200 GPUs.

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

MoRE: Mixture of Reused Experts
Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Department of Computer Science, Cornell University
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Language Models Mixture of Experts
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
ExFusion: Efficient Transformer Training via Multi-Experts Fusion
Mar 30, 2026Jiacheng Ruan, Daize Dong, Xiaoye Qu +5Transformer Mixture of Experts
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, No 800, Dongchuan Road, Minhang District, Shanghai, China, 200240 · Shanghai Artificial Intelligence Laboratory, Shanghai, China, 200240 · Huazhong University of Science and Technology, Hubei, China, 430074 +3

Sep 16, 2026J/K move · Enter open · S save

MoRE: Mixture of Reused Experts

Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace, Christian Belardi+3

Department of Computer Science, Cornell University

Mixture-of-Experts Language Models Sparse Mixture-of-Experts

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures decouple model capacity from computational cost, yet incur high memory footprints as parameters grow linearly with the number of experts. Recurrent Transformers achieve parameter efficiency by reusing layer weights, but typically lack the capacity for competitive language modeling. We propose Mixture of Reused Experts (MoRE), a hybrid that shares expert pools across groups of adjacent layers. Each layer retains its own router but selects from a larger shared pool, expanding the diversity of routing combinations without additional parameters. To enable shared experts to distinguish between layers, we introduce lightweight learnable depth embeddings that condition each layer's input before routing. Experiments across three model scales (114M-1.15B parameters) show that MoRE consistently achieves lower perplexity and stronger downstream performance than standard MoEs and state-of-the-art weight-sharing architectures at matched compute and parameter budgets, with only minimal modifications to existing MoE implementations.

Abstract

Mixture-of-Experts Language Models

Explore similar work

MoRE: Mixture of Reused Experts

UniPool: Learning Expert-to-Layer Ownership from Brief Global Access

ExFusion: Efficient Transformer Training via Multi-Experts Fusion