Path-Constrained Mixture-of-Experts

Zijin Gu; Tatiana Likhomanenko; Vimal Thilak; Jason Ramapuram; Navdeep Jaitly

cs.LGMar 18, 2026

Path-Constrained Mixture-of-Experts

Authors: Zijin Gu, Tatiana Likhomanenko, Vimal Thilak, Jason Ramapuram, Navdeep Jaitly

Organizations: Apple · †Google

Abstract

Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts at each layer independently. We propose viewing MoE computation through the lens of expert paths -- the sequence of expert selections a token makes across all layers. This perspective reveals that, despite $N^L$ possible paths for $N$ experts across $L$ layers, tokens in practice cluster into a small fraction of paths that align with linguistic function, yet the vast majority of paths remain unexplored, representing a statistical inefficiency. This motivates architectures that constrain the effective path space to amplify this natural concentration. As one instantiation, we introduce \pathmoe{}, which shares router parameters across blocks of consecutive layers. Analysis confirms that \pathmoe{} amplifies the emergent path structure: it produces more concentrated path clusters, better cross-layer consistency, and greater robustness to routing perturbations. Experiments on 0.9B and 16B parameter \pathmoe{} models demonstrate consistent improvements on perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary losses. These results establish expert paths as a useful design axis for MoE architectures, complementary to existing work on independent routing mechanisms.

Figures & tables

Figure 1: Statistical inefficiency of independent routing.

Figure 2: Spectrum of routing constraints in MoE architectures. (a) Independent routing:each layer has its own router

r_{i}

, creating

N^{L}

possible paths for

N

experts and

L

layers. (b) Block-wise parameter sharing routing, dubbed PathMoE: layers within a block share router parameters. (c) Fully decision shared routing: all layers share one router and its decision.

Figure 3: Cumulative token coverage as a function of the number of paths.

Figure 4: Token specialization of representative expert paths. Each word cloud shows the most frequent tokens processed by a path specialized for that linguistic category.

Routing	ARC-E	BoolQ	HSwag	LAMBADA	OBQA	PIQA	SocIQA	WinoGr.	Avg.	Perplexity	Throughput	Memory
Routing	ARC-E	BoolQ	HSwag	LAMBADA	OBQA	PIQA	SocIQA	WinoGr.	Avg.	Perplexity	(k tok/s/GPU)	(GiB)
Independent Routing
Indep-MoE	44.57	56.45	45.99	46.19	29.80	66.87	38.84	51.54	47.53	12.91	55.21	66.94
Rand-MoE	40.49	61.10	37.95	38.83	28.20	64.47	38.89	50.99	45.12	16.14	75.17	47.49
X-MoE	43.27	60.55	46.26	45.22	32.20	67.63	39.61	52.80	48.44	13.21	56.37	67.06
Recurrent Routing

Table 1: Main results on Fineweb-100B with 0.9B total / 0.37B active MoE architecture. Throughput is reported per GPU and memory reports peak active GPU memory.

Figure 5: Training dynamics with and without load balancing losses (LBL).

	Task	Indep	Path	$\Delta$
Comm.	WinoGr.	56.12	57.93	+1.81
	PIQA	68.99	71.38	+2.39
	SocIQA	40.58	40.07	–0.51
	CSQA	43.00	48.73	+5.73
Lang.	LAMBADA	51.97	53.64	+1.67
	HSwag	63.22	63.32	+0.10

Table 2: 16B model results (16.2B total / 2.13B active) on DCLM-Pro.

Figure 6: Ablation studies on routing hyperparameters. (a) Effect of top-

k

routing. (b) Effect of block size. Dashed lines show individual benchmarks; solid black line shows the average.

Figure 7: Routing consistency, specialization, and robustness. (a) Path consistency versus window size. (b) Sustained engagement versus minimum consecutive layers. (c) Cumulative routing entropy across layers. (d) Perplexity degradation under expert permutation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Empirical Metric	PathB4-MoE	Indep-MoE
Routing entropy $H(\mathbf{E})$	21.14 bits	22.20 bits
Routing decisions correlation for consecutive layers	85.6%	62%
Unique paths observed	5,109,282	6,263,708
Effective path space $2^{H(\pi)}$	$2.31\times 10^{6}$	$4.82\times 10^{6}$

Appendix

Table 3: Detailed empirical routing entropy computation and expert path space statistics for Indep-MoE and PathB4-MoE (block size

B=4

) with

L=24

layers and

N=16

experts. All entropy measurements are in bits. We use evaluation data with 7.18M tokens.

Hyperparameter	0.9B Model	16B Model
Model Architecture
Total parameters	0.9B	16.2B
Active parameters	0.37B	2.13B
Model dimension	1280	2048
FFN hidden dimension	768	2112
Layers	24	28

Appendix

Table 4: Training hyperparameters for 0.9B and 16B models.

Routing	LBL	ARC-E	BoolQ	HSwag	LAMBADA	OBQA	PIQA	SocIQA	WinoGr.	Avg.
Indep-MoE	0.01	44.57	56.45	45.99	46.19	29.80	66.87	38.84	51.54	47.53
Indep-MoE	0	43.18	58.29	47.05	47.55	28.80	65.94	39.36	53.67	47.98
PathB4-MoE	0.01	44.15	57.06	46.07	46.61	32.60	66.49	40.23	51.93	48.14
PathB4-MoE	0	44.70	60.40	47.95	49.45	31.60	66.32	40.94	55.64	49.62

Appendix

Table 5: Performance comparison on Fineweb-100B dataset (0.9B total / 0.37B active).

Routing	ARC-E	BoolQ	HSwag	LAMBADA	OBQA	PIQA	SocIQA	WinoGr.	Avg.
Indep-MoE	55.18	56.27	48.19	40.73	35.80	66.70	39.46	55.96	49.79
LowRank-MoE	58.84	59.42	48.26	40.68	33.00	66.54	40.69	54.38	50.22
PathMoE	57.49	60.06	48.30	40.69	34.40	67.25	39.87	56.99	50.63
PathB8-MoE	58.16	58.20	48.26	40.35	34.40	66.65	40.33	54.38	50.09

Appendix

Table 6: Performance comparison on DCLM-Pro dataset (0.9B total / 0.37B active).

Routing	ARC-E	BoolQ	HSwag	LAMB.	OBQA	PIQA	SocIQA	WinoGr.	Avg.
X-MoE	43.27	60.55	46.26	45.22	32.20	67.63	39.61	52.80	48.44
PathB4X-MoE	44.07	61.07	47.29	48.30	30.80	66.43	39.97	52.96	48.86

Appendix

Table 7: Combining PathMoE with X-MoE on Fineweb-100B (0.9B total / 0.37B active). PathMoE improves X-MoE’s average accuracy (standard deviation is

\sim 0.28\%

# Paths	ARC-E	BoolQ	HSwag	LAMBADA	OBQA	PIQA	SocIQA	WinoGr.	Avg.
10	40.45	59.54	39.34	40.87	30.40	66.81	37.82	54.54	46.22
100	43.81	56.79	47.04	45.10	30.80	67.46	40.23	54.22	48.18
500	44.74	59.42	48.24	47.58	31.20	66.00	39.97	53.75	48.86
All	44.82	56.48	47.76	46.42	31.60	66.76	39.41	56.43	48.71

Appendix

Table 8: Full per-task results for expert path restriction. Models are trained with routing restricted to the top-

K

most frequent paths identified at 5k steps (1.25% of training).

Category	Type	Examples
person_names	Lexical	Andrea, Richard, Oprah, Mary, John, Elizabeth
titles_roles	Lexical	secretary, minister, commander, winner, professor, CEO
speech_verbs	Lexical	said, explained, told, asked, claimed, announced
adverbs_discourse	Lexical	especially, actually, particularly, however, therefore
adverbs_manner	Lexical	quickly, carefully, successfully, directly, properly
adverbs_time	Lexical	now, today, recently, always, sometimes, currently

Appendix

Table 9: Token categories used for path specialization analysis. Each category is defined by curated word lists or morphological patterns.

Explore similar work

PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
Sep 1, 2026Ziyan Gan, Fangxin Liu, Chenyang Guan +10Mixture-of-Experts Pruning Mixture-of-Experts Inference
1Shanghai Jiao Tong University, China · 2Alibaba Group, China
Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Language Models Expert Routing
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
Jun 16, 2026Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
AGH University of Krakow, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Poland · Jagiellonian University, Poland +1

Sep 1, 2026J/K move · Enter open · S save

PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition

Ziyan Gan, Fangxin Liu, Chenyang Guan, Junjie Wang+9

1Shanghai Jiao Tong University, China · 2Alibaba Group, China

Mixture-of-Experts Pruning Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However, modern MoE inference remains heavily constrained by the rigid, whole-expert abstraction. Existing frameworks manage, schedule, or prune experts as atomic execution units, which fixes the optimization boundary too early and leaves fine-grained intra-expert computational redundancy underexplored. In this work, we present PCoMoE, a path-compositional execution framework that shifts MoE inference from coarse-grained expert selection to fine-grained path composition. PCoMoE incorporates a path-level formulation of expert computation, a compatibility-aware layer-wise pruning strategy to suppress low-value path combinations, and a hardware-friendly execution engine to exploit reusable sub-expert structures under strictly bounded overheads. Experimental results demonstrate that PCoMoE achieves up to a 1.31x end-to-end inference speedup while enhancing model accuracy by 10%. The code is available at https://github.com/gzyyy0/PCoMoE

Abstract

Explore similar work

PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition

Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs