Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute

Vu Quang Hoang; Nghia Hieu Nguyen

cs.LGOct 1, 2026

Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute

Authors: Vu Quang Hoang, Nghia Hieu Nguyen

Organizations: University of Information Technology · Vietnam National University, Ho Chi Minh City, Viet Nam

Abstract

Sparse Mixture-of-Experts (MoE) models combine learned routing with selection from a large expert pool. We isolate the contribution of dynamic expert combination using a dense analogue: $K$ SwiGLU experts, all active for every token and combined by a softmax gate, at fixed total FFN width. With no larger pool or discrete selection, the dense baseline is the $K=1$ case. Validation loss varies non-monotonically with $K$ : $K=2$ improves over the baseline by $0.0048$ , whereas $K=4$ and $K=6$ worsen it by $0.0053$ and $0.0197$ , respectively. Routing generally remains soft and expert usage balanced, except in the first layer of the $K=4$ model, where routing is nearly one-hot. Forcing this gate to uniform increases loss by $2.4$ nats on a diagnostic subset, indicating that its concentrated routing is functionally important. We further show that the architecture is exactly a dense SwiGLU with token-dependent, simplex-constrained group scaling and contains the dense baseline in its function class. These single-run results suggest a granularity sweet spot for always-active, softly gated FFNs at fixed width, with $K=2$ performing best among the configurations tested.

Figures & tables

$K$	$d_{e}$	Router params	Non-embedding params	Total params (tied embed./head)
1 (dense)	$4d_{\text{model}}=1536$	0	28,321,152	47,619,840
2	$2d_{\text{model}}=768$	768	28,330,368	47,629,056
4	$d_{\text{model}}=384$	1,536	28,339,584	47,638,272
6	$\tfrac{2}{3}d_{\text{model}}=256$	2,304	28,348,800	47,647,488

Table 1: Exact parameter counts. 12 layers,

d_{\text{model}}=384

, 6 heads, head dim 64, vocab 50,257, embeddings tied with the output head. The router adds at most 2,304 parameters total (0.005% of the model), which we treat as negligible for matching purposes.

FFN block at initialization	Output std	Variance / baseline
Dense SwiGLU ( $d_{\text{ffn}}=3d_{\text{model}}$ )	0.1056	1.00
Experts, scaled init	0.0111	0.011
Experts, fan-in init	0.0610	0.334

Table 2: Step-0 output scale of the FFN block (numerical simulation of the initialization scheme with i.i.d. Gaussian inputs). The fan-in variant’s residual ratio

\approx 1/K

matches the softmax-averaging prediction; the scaled variant starts roughly an order of magnitude quieter in standard deviation.

Figure 1: Validation loss over 20,000 steps for the dense baseline (

K{=}1

) and

K=2,4,6

. (a) Full training. (b) Zoom on steps 8,000–20,000, where all four curves have entered a slowly-decaying plateau regime.

Figure 2: Validation loss minus the dense baseline’s validation loss, at the same step, for

K=2,4,6

. All three start positive (worse than baseline).

K=2

crosses zero between steps 4000 and 4500 and stays negative afterward;

K=4

and

K=6

never cross zero and plateau at a positive gap that grows with

K

Configuration	$d_{e}$	Val. loss (step 20,000)	$\Delta$ Val vs. dense
Dense SwiGLU ( $K{=}1$ )	$4d_{\text{model}}$	3.8752	—
$K{=}2$	$2d_{\text{model}}$	3.8704	$-0.0048$
$K{=}4$	$d_{\text{model}}$	3.8805	$+0.0053$
$K{=}6$	$\tfrac{2}{3}d_{\text{model}}$	3.8949	$+0.0197$

Table 3: Validation loss at step 20,000.

\Delta

is expert model minus dense baseline, so negative means the expert model is better. One run per configuration; all use the same init and router variant; parameter counts in Table 1 .

$K$	Per-token entropy	Mean max weight (uniform)	Frac. $>0.9$	Usage entropy
2	0.719	0.762 (0.500)	0.189	0.948
4	0.668	0.622 (0.250)	0.097	0.901
6	0.713	0.517 (0.167)	0.015	0.941

Table 4: Router statistics at step 20,000, averaged over layers, for the models in Table 3 .

Layer	Mean pairwise cosine sim.	Max pairwise cosine sim.
0	0.414	0.650
1	0.161	0.249
2	0.085	0.187
3	0.049	0.156
4	0.044	0.143
5	0.040	0.125

Table 5: Pairwise cosine similarity between the

K{=}4

model’s expert outputs, by layer, on a held-out batch.

Configuration	Val. loss	Val. ppl.
Normal routing (trained gates)	3.7493	42.49
Layer 0 forced uniform	6.1216	455.59

Table 6: Uniform-gate ablation on layer 0 of the

K{=}4

model, evaluated on a smaller, fixed subset of the validation split (CPU-only, for speed; see text). Both rows use this identical subset, so the comparison between them is unaffected by the choice of subset; the absolute values are not directly comparable to Table 3 ’s full-validation-split figures.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 462 papers

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Feb 17, 2026Ziyu Zhao, Tong Zhu, Zhi Zhang +6Mixture-of-Experts Language Models Mixture-of-Experts Pruning
Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory +3
Rethinking Routing Exploration in Modern Mixture-of-Experts
Oct 15, 2026Bingcong Li, Hao Ma, Michael Muehlebach +2Mixture-of-Experts Models Expert Routing
ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1
Confidence-Adaptive SwiGLU for Mixture-of-Experts
May 30, 2026Shaohua Li, Xiuchao Sui, Xiaobing Sun +4Mixture-of-Experts Language Models Neural Network Activation Functions
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Shanghai University of Engineering Science, China

Feb 17, 2026J/K move · Enter open · S save

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

Ziyu Zhao, Tong Zhu, Zhi Zhang, Tiantian Fan+5

Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory+3

Mixture-of-Experts Language Models Mixture-of-Experts Pruning LLM Compression

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: dynamic structural pruning that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and downcycling approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.

Abstract

Mixture-of-Experts Models

Explore similar work

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

Rethinking Routing Exploration in Modern Mixture-of-Experts

Confidence-Adaptive SwiGLU for Mixture-of-Experts