ParGLU: Partial Gating Improves Language Modeling in MoE Transformers
Abstract
Multiplicative gating improves Transformer feed-forward networks (FFNs), but it reduces intermediate representation width under a fixed parameter budget. This trade-off is particularly relevant to Mixture-of-Experts (MoE) models, where individual experts have limited capacity. We introduce ParGLU, a partially gated FFN that combines SwiGLU and ReLU in two parallel branches. ParGLU matches the parameter count and principal matrix computation budget of conventional SwiGLU. We evaluate ParGLU in 30M, 75M, and 152M MoE language models using three random seeds per scale. ParGLU outperforms SwiGLU across all three seeds at 75M and 152M, reducing mean validation NLL by 0.0200 and 0.0249, respectively. Results at 30M are mixed. Controlled ablations show that increasing backbone width or expert width strengthens ParGLU's relative performance. Ablation studies further show that combining SwiGLU and ReLU improves performance over either architecture alone. Token-frequency analysis reveals larger improvements on low-frequency tokens. These findings show that fully gated FFNs are not always the most effective use of an MoE expert's parameter budget. Partial gating can improve language modeling in MoE Transformers.
Figures & tables
| Model | SwiGLU | ParGLU | Total Params | Active Params | |
|---|---|---|---|---|---|
| 30M | 256 | 128 | 64 / 96 | 30.16M | 15.12M |
| 75M | 384 | 256 | 128 / 192 | 75.48M | 30.35M |
| 152M | 384 | 608 | 304 / 456 | 152.12M | 44.95M |
| Model | SwiGLU NLL | ParGLU NLL | Seed 42 | Seed 43 | Seed 44 | Mean |
|---|---|---|---|---|---|---|
| 30M | 3.249961 | 3.253571 | ||||
| 75M | 3.050291 | 3.030250 | ||||
| 152M | 2.965396 | 2.940475 |
| Total Params | SwiGLU NLL | ParGLU NLL | |||
|---|---|---|---|---|---|
| 256 | 128 | 30.16M | 3.243776 | 3.253688 | |
| 256 | 256 | 48.74M | 3.160740 | 3.142031 | |
| 384 | 128 | 47.61M | 3.112823 | 3.113106 | |
| 384 | 256 | 75.48M | 3.064122 | 3.023994 |
| FFN Architecture | Gated Width | Non-Gated Width | Val NLL | |
|---|---|---|---|---|
| Pure SwiGLU | 256 | — | 3.064122 | 0 |
| Pure ReLU 2 | — | 384 | 3.048075 | |
| ParGLU 25/75 | 64 | 288 | 3.038758 | |
| ParGLU 50/50 | 128 | 192 | 3.023994 | |
| ParGLU 75/25 | 192 | 96 | ||
| SwiGLU + GELU 50/50 | 128 | 192 | 3.041322 |
| Model | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| 30M | ||||
| 75M | ||||
| 152M |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Training dataset | SmolLM |
| Vocabulary size | 32,768 |
| Context length | 4,096 |
| Optimizer | AdamW |
| Peak learning rate | |
| Weight decay | 0.1 |
| Model | Seed | SwiGLU NLL | ParGLU NLL |
|---|---|---|---|
| 30M | 42 | 3.243776 | 3.253688 |
| 30M | 43 | 3.247576 | 3.242496 |
| 30M | 44 | 3.258530 | 3.264528 |
| 75M | 42 | 3.064122 | 3.023994 |
| 75M | 43 | 3.049117 | 3.033163 |
| 75M | 44 | 3.037634 | 3.033593 |
| Group | Frequency Range | Validation Tokens |
|---|---|---|
| Q1 (Lowest) | – | 1,333,362 |
| Q2 | – | 1,343,753 |
| Q3 | – | 1,366,266 |
| Q4 (Highest) | 1,325,257 | |
| Total | — | 5,368,638 |
Transformer FFNs
Transformer feed-forward networks (FFNs) are the within-token MLP blocks whose activations, structure, efficiency, and role in model computation are studied.
See all 45 papers